TEACHING
MOD 3.2 – Deep learning and AI: an Introduction
By allowing breakthroughs that were hitherto unthinkable in a growing number of fields, e.g, computer vision, speech recognition, games, deep learning has revolutionized the field of artificial intelligence which has become the main pillar of our society. In this course, we aim to introduce the concepts, techniques and basic tools of deep learning and will cover in particular the following aspects:
- Introduction of machine learning and deep learning,
- Classification / regression and the descent of the gradient
- Data graph & backpropagation
- Training deep neural networks
- Convolutional Neural Networks (CNN)
- Deep reinforcement learning
- Q-learning, Policy gradient, Actor-Critic
Keywords
Deep learning, classification, regression, data graphs and backward propagation, CNNs, reinforcement learning; Q-learning, policy gradient, critical actor
MSO 3.4 – Deep Structured Learning
Deep learning has revolutionized an increasing number of domains, e.g., computer vision, natural language processing, games, etc. Structured learning is machine learning which aims to output data, e.g., sequences, matrix, graphs, which have components under some dependencies, e.g., words in a sentence. In this course, we aim to introduce fundamental concepts, theories and advanced techniques in deep structured learning, covering in particular sequence to sequence learning and Generative Adversarial Network (GAN). A number of practical works will be scheduled, including for instance image generation, image to text generation, text-to-image generation, style transfer, etc..
Keywords
Structured learning, recursive networks, LSTM, Attention-based models, Transformer, Bert, GAN, cGAN
MSO 3.5 – Computer Vision
Computer vision is to endow computers with human-like vision capabilities. This course aims to introduce fundamental methods and cutting-edge techniques in computer vision. It covers topics such as image filtering, template matching, object detection, semantic segmentation, visual object instance segmentation, object tracking, camera optics, epipolar geometry, stereovision.
Keywords
pixel and image filtering, template matching, CNN, U-Net, object detection, instance segmentation, object tracking, camera optics, epipolar geometry, stereovision
ELC C4 – Image sensing and processing
The aim of this course is to introduce basic concepts and techniques on image acquisition and image processing sensors. It covers the foundations and addresses the principles of image formation, image filtering, image transformation, frequency analysis and Fourier transform, edge detection, extraction of points of interest (Harris, Laplacien, SIFT) and their matching, segment detection and Hough transform. The applications are numerous, such as for example medicine, quality control, artificial vision, satellite imagery, etc.
Keywords
pixels, spatial and frequency image filtering, 2D Fourier transform, image edge detection, Harris, Laplacian and SIFT local points of interest, segment detection and Hough transform
Project: Object Grasping with a UR3 robot using Deep Learning (visual pushing and grasping)
Level (year) : PaR 2018 – 2019 (Finished)
Name of the students : Lucas Neto Nakadaira, João Vitor Sousa Floriano
Video :
Code and Report : https://github.com/jsousafl/par142-VPG-adaptation
The goal of this project is to use a robotic arm (UR3 from Universal Robot) to grasp and sort objects in heap using a Deep Learning method called Visual Pushing and Grasping (VPG). This technique allows the robot to grasp objects that are initially very difficult to access properly (no grasping possibilities because too close to other objects) by moving the heap in a first step.
The first phase will have the student being familiar with the VPG code developed by Princeton University et available on github (https://github.com/andyzeng/visual-pushing-grasping) This will require to study and learn how to use tools such as PyTorch (Deep Learning), V-Rep (Simulator) and algorithms needed to control the UR3 robot.
In a second step, the students will adapt the existing code to the specific environment available at the laboratory at ECL. In particular, modifications will be needed to 1) control a smaller UR3 robot instead of the UR5 from Princeton, 2) replace the code extracting images from the RGB-D camera (Intel RealSense D415 for Princeton, Kinect for ECL), 3) check and adapt the calibration procedure linked to both of these aspects, and 4) test the learning robustness by using object available at ECL.
Project: Panda robotic arm as partner to play Chess
Level (year) : PaR 2019 – 2020
Name of the students : Clément Vinot, Victor Bayona
Video :
Code and Report :
The goal of this project is to use a robotic arm (Panda, from Franka Emika) to play Chess with a human partner.
In a first step, students will participate in the setting up and installation of the robot inside the ECL laboratory before being familiarised with the code allowing to control the robot and play chess, using existing code from a student project at Imperial College London (available on github: https://github.com/nebbles/DE3-ROB1-CHESS). The project has also extensive documentation on https://de3-rob1-chess.readthedocs.io/en/latest/ and is mainly written in Python. The initial goal will be to reproduce and adapt the code to the specific setup at the ECL, before improving performance or the quality of the interaction.
Depending on the student’s interests and opportunities, these improvements can be for instance the addition of Natural Speech Processing and Production during oral communication between the human and the robot; the use of a better AI system to play Chess; or the addition of more games/tasks that the robot can play/do.
This project will require to study computer sciences tools such as ROS (Robotics Middleware), OpenCV (Computer Vision) and Sunfish (AI for Chess), in addition to being confronted to typical Robotics scientifics and technical problematics: integration and communication between sensors and robotics parts, calibration, control and movements.
Project: Object Grasping with a UR3 robot using Deep Learning (visual pushing and grasping)
Level (year) : PaR 2019 – 2020
Name of the students : Igor Leandro Viera, Caio Da Costa De Barros Pimentel Luke
Video :
Code and Report :
The goal of this project is to use a robotic arm (UR3 from Universal Robot) to grasp and sort objects in heap using a Deep Learning method called Visual Pushing and Grasping (VPG). This technique allows the robot to grasp objects that are initially very difficult to access properly (no grasping possibilities because too close to other objects) by moving the heap in a first step.
The first phase will have the student being familiar with the VPG code developed by Princeton University et available on github (https://github.com/andyzeng/visual-pushing-grasping) This will require to study and learn how to use tools such as PyTorch (Deep Learning), V-Rep (Simulator) and algorithms needed to control the UR3 robot.
In a second step, the students will adapt the existing code to the specific environment available at the laboratory at ECL. In particular, modifications will be needed to 1) control a smaller UR3 robot instead of the UR5 from Princeton, 2) replace the code extracting images from the RGB-D camera (Intel RealSense D415 for Princeton, Kinect for ECL), 3) check and adapt the calibration procedure linked to both of these aspects, and 4) test the learning robustness by using object available at ECL.