The execution of a task is composed of a sequence of individual actions, however a comprehensive task description is hardly a linear sequence of actions. Think of an assembly task: one can either bring all components to the assembly area and then stick the components together to form the result or one can stick together some components into a part-result, and then bring all part-results to the final assembly area and merge the parts together. An action inside the task sequence can have multiple pre-required actions to be completed, which leads to the modelling idea for a task of a DAG (directed acyclic graph).
In this project, a collection of human daily activities should be modelled based on multiple visual demonstrations into a DAG, which should enable activity recognition. Additionally, it would be a nice feature to have, if the system observes the execution of a task from the middle of the task (so not the beginning) and could still correctly classify the task.

Skills:

• good C++ knowledge and experience
• good algorithms and data structures knowledge, in particular relating to graph theory
• a solid mathematical foundation

Available modules:

• 3d Visual Perception: human skeleton joints as well as object positions & poses
• Contact-segmentation based on hand position and object bounding boxes

Tasks:

• Create action descriptions that are visually assessable
• Create a list of daily activities containing the actions as building blocks
• Generate a sequence of actions performed in a visual demonstration
• Merge sequences from multiple demonstrations with different sequences into a DAG
• Perform classification of the activity based on a (possibly new/unseen) action sequence
• Create a metric for similarity between sequences (depending on e.g. sequence length; interacted objects; etc.) which will be used for classification
• Create a strategy for classification that would enable recognizing the action from "the middle" of it's execution

Visual object pose (position and orientation) recognition is still an unsolved task. Certain approaches create models which work e.g. assuming well-textured objects, assuming a special shape of an object, or a custom background simplifying the object model creation. The goal of this project is to create object models using data coming from an RGBD sensor that will enable object 6D pose recognition.
Feature points from a camera image work only with well-textured objects but is a time-efficient method. Matching 3d point-clouds of objects works with no occlusions, has good results if the object is non-symmetric, but is a time-consuming method. The goal of the project is to create an object representation combining the advantages of both methods to enable visual 6D object pose recognition.

Skills:

• good Python/C++ knowledge and experience
• a very good mathematical foundation

Available modules:

• RGBD sensor data processing
• Image feature extraction

Tasks:

• Create module to scan an object and fuse it into a point cloud representation of the object from RGBD data
• Literature review on 3d object representation and rotation determination
• Optional: Investigate approaches to represent object symmetries
• Jointly (with myself) create and discuss object modelling strategies
• Implement and validate approach on household objects

This project aims to model (i.e. represent and recognize) human gestures visually. Gestures (facial expressions included) are both static and dynamic, requiring time-dependent analysis to be performed. Given 3D (color and depth) information of a person, the goal is to output at any time what gesture(s) is the person performing.
While not explicitly discouraged, deep learning methods are welcome to be investigated, however, large training data will not be available from us ( maybe there are existing datasets that would allow to create and train a model; this will have to be determined during the literature research). Ideally, the created model would understand (represent) the gesture from very few demonstrations (e.g. 1 to 5) and from previous experience analyzing gestures.

Skills:

• good Python/C++ knowledge and experience
• a solid mathematical foundation (especially in algebra and geometry)

Available modules:

• RGBD sensor data processing
• Human Pose Estimation (incl. hand + fingers)
• Facial Feature Extractor

Tasks:

• Create list of gestures (static & dynamic; incl. facial information or not) to be modelled by the system
• Generate gesture data & collect the resulting perception data (skeleton keypoints & facial features; available also in 3d)
• Create classifier for gestures (including dynamic analysis: time-evolving gestures)

For human action recognition and understanding, segmentation plays an important role. This project will investigate methods of human action segmentation solely from human skeleton movements, extending previous research done focusing only on the right arm. The human body is capable of performing simultaneous actions with different body parts or even one action with multiple body parts. We want to be able to model these cases and enable action recognition leading into activity recognition based on human-performed motions.
Using a body-part joint-position energy function, a segmentation of the demonstrated full-body action is performed, resulting in motion primitives of each body part. The demonstration will be a sequence of 3d skeleton joint-positions. Based on the learned motion primitives, an action model will be created, later used for action recognition, parametrization of the motion primitives, creating a personalized motion profile which can be used to recognize a person without facial-feature information.

Skills:

• good C++ knowledge and experience
• a solid mathematical foundation (algebra, analysis and (3d) geometry)

Available modules:

• Generation of a sequence of skeleton 3d joint positions from video
• Previous work performed for right-arm actions only (to be extended to full-body actions)

Tasks:

• Investigate motion segmentation approaches based on skeleton joint positions
• Defined a list of motion primitives for each body part (equivalent to joints and joint angles of a robot)
• Perform segmentation of a demonstrated motion
• Describe the demonstrated motion using the previously defined motion primitives
• Create a motion profile for each person -> perform classification of the person based on the recognized motion profile
• Create a graph representation of action sequences -> perform online activity recognition based on the motion primitives
• Visualize the actions in a live/simulated video (preferably in vrep, Unity, Unreal; ok on std-out)

TODO: complete project description
Create a simple language model describing the english grammar.
Create a list of daily tasks that a smart-system or an elderly care robot would need to execute as a human aid.
Create a text parser => classify intended robot command
Produce textual responses (which will be transformed to speech using an existing API) pertaining to each command or upon failure or upon not understanding, etc.

TODO: project description

TODO: project description

TODO: project description

TODO: project description

For human action recognition and understanding, segmentation plays an important role. This project should investigate methods of human action segmentation based on human interaction with the environment.
Given a sequence of graphs representing the (geometric-)relations between the objects in the environment, the result is a sequence segmentation of the graphs based on the identified (learned) actions in the sequence.

Skills:

• C++ or Python knowledge & experience

Available modules:

• Teaser++: 3D relative object pose estimation from object meshes
• RelationGraph generation from 3D poses of objects

Tasks:

• Integrate 3D object pose into the RelationGraph-generation process
• Integrate the human (represented as a bounding-box; position, contact relations) into the RelationGraph
• Apply learning method to learn actions based on changes within the RelationGraph
• Visualize the actions in a live/simulated video (preferably in Unity or Unreal; ok on std-out)

TODO: project description
Based on the object shape and the gripper kinematics and shape, determine grasp points (incl. symmetrical representation). Compactly represent list of grasps + create human-friendly visualization/simulation of the grasp points in vrep, Unity or Unreal.