It is rare that we humans have full knowledge of the states of the object in our environment. Consider a cup on the desk of your office colleague. One does not know its content. If the cup is on the desk with its opening downward, then we can infer that its content is not a liquid. If the cup's opening is pointing upward and the colleague is drinking from it, then we can infer that the cup was not empty; the current fill status is however still unknown (could be empty, could still contain something).
In this project, an implementation of the inference process of unknown entity properties based on static configurations (entity location) and successful skill execution sequences is to be done.

Skills:

• good C++ knowledge and experience
• good algorithms and data structures knowledge
• knowledge of formal verification methods
• a solid mathematical foundation

Available modules:

• Entity property definitions with corresponding value ranges
• Location determination procedure of entities in an environment
• Skill definitions with effects on entity properties and prerequisites on entity properties for a successful skill execution

Tasks:

• Model natural world effects on objects (such as gravity or heat) and develop inference rules for entity properties based on these effects
• Create flow of functions that act on the unknown properties of entities based on skill execution sequences and develop inference rules

A task is a desired state of the environment. The environment is comprised of physical entities: objects and agents. Each physical entity has properties (Objects have a color; Containers have a list of contained instances; LiquidContainers most likely contain liquid object instances). The collection of property values of objects and agents is the state of the environment.
A task defines the desired state but not necessarily how to reach that desired state. A plan has to be created that takes into account the abilities of the executing agent and that chooses suitable skills for the agent to execute that transform the current environment state into the desired one defined by the task.
In this project, an implementation a planner that creates a task plan is to be designed.

Skills:

• good C++ knowledge and experience
• good algorithms and data structures knowledge
• knowledge of classical planners and optimization methods
• a solid mathematical foundation

Available modules:

• Action definitions modelling changes on entity properties
• Skill to action associations
• Agent ability definitions that are composed to form skill executions

Tasks:

• Model the differences between the current state and the desired state.
• Extract the actions that perform these changes.
• For each action, select a suitable skill that implements that action in the environment and that can be executed by the executing agent in the environment.
• Optimize skill selection based on heuristics, optimize for distance travelled, energy, time, or other cost factors.
• For a robotic agent/executor, the output of the plan should be a sequence of abilities (motion primitives) or a behavior tree that the robot can execute.

The execution of a task is composed of a sequence of individual actions, however a comprehensive task description is hardly a linear sequence of actions. Think of an assembly task: one can either bring all components to the assembly area and then stick the components together to form the result or one can stick together some components into a part-result, and then bring all part-results to the final assembly area and merge the parts together. An action inside the task sequence can have multiple pre-required actions to be completed, which leads to the modelling idea for a task of a DAG (directed acyclic graph).
In this project, a collection of human daily tasks should be modelled based on multiple visual demonstrations into a DAG, which should enable tasks recognition. Additionally, it would be a nice feature to have, if the system observes the execution of a task from the middle of the task (so not the beginning) and could still correctly classify the task.

Skills:

• good C++ knowledge and experience
• good algorithms and data structures knowledge, in particular relating to graph theory
• a solid mathematical foundation

Available modules:

• 3d Visual Perception: human skeleton joints as well as object positions & poses
• Contact-segmentation based on hand position and object bounding boxes

Tasks:

• Create action descriptions that are visually assessable
• Create a list of daily activities containing the actions as building blocks
• Generate a sequence of actions performed in a visual demonstration
• Merge sequences from multiple demonstrations with different sequences into a DAG
• Perform classification of the task based on a (possibly new/unseen) action sequence
• Create a metric for similarity between sequences (depending on e.g. sequence length; interacted objects; etc.) which will be used for classification
• Create a strategy for classification that would enable recognizing the action from "the middle" of its execution