Hand-object pose estimation is a challenging task due to multiple factors like occlusion, and ambiguity in pose recovery.  To overcome this issue, multi-view camera systems are used.

Using 2D keypoint detectors for hands and objects like  Yolov8-pose and mmpose we can uplift the 2D detections to 3D. However, the detections usually are usually noisy, and some keypoints may be missing.  

We want to utilize deep learning methods for smoothing, inpainting, and uplifting these detections to 3D in order to estimate the pose of the corresponding hands and objects.

The task is formulated as follows:

Given a sequence of noisy 2D key points for human hands and an object captured from calibrated camera views. Using a deep learning model, estimate a smooth trajectory of the hand and object poses.

• Python
• Knowledge about Deep Learning
• Knowledge about Pytorch
• Previous Knowledge about 3D data processing is a plus.

A HiWi position is available for the Lab Course Human Activity Understanding. 

The position offers 6 h/week contract.

The lab involves:

•  Practical Sessions where the students collect data from a color/depth sensor setup.
•  Notebook Sessions where the students are introduced to a jupyter notebook with brief theoretical content and homework.
• Project Sessions, where the students are working on their own projects.  

The main tasks of this position involve the following:

• Helping students with data collection in Practical and Project Sessions.
• Assisting during the notebook sessions with regard to the contents of the notebooks and homework. 

• Knowledge about ROS.
• Knowledge about python.
• Basic Knowledge in Deep Learning

Recently, there has been increasing success in the generation of human motion and object grasp. On the other hand, an increasing number of datasets capture human hands' interaction with surrounding objects in addition to action labels. 

One advantage of Diffusion models is that they can easily conditioned on different types of input like text embeddings and control parameters.  

Your task will include exploring the existing models and implementing the hand-object interaction diffusion model.

Datasets We could use: 

•  https://hoi4d.github.io/
•  https://taeinkwon.com/projects/h2o/
•  https://dex-ycb.github.io/

Motion Generation Models

• https://guytevet.github.io/mdm-page/
• https://goal.is.tue.mpg.de/

• Experience and interest in deep learning research.
• Knowledge and experience with Pytorch.

In this project the task is to fit a parametric hand mesh model and a set of rigid objects to a sequence of multi-view RGB-D cameras. Existing models for hand key-point detection and 6DoF pose estimation for rigid objects models have significantly evolved in recent years. Our goal is to utilize such models to estimate the hand and object poses.

Related Work

1. https://dex-ycb.github.io/
2. https://www.tugraz.at/institute/icg/research/team-lepetit/research-projects/hand-object-3d-pose-annotation/
3. https://github.com/hassony2/obman
4. https://github.com/ylabbe/cosypose

• Knowledge in computer vision.
• Experience about segmentation models (i.e. Detectron2)
• Experience with deep learning frameworks PyTorch or TensorFlow(2.x).
• Experience with Pytorch3D is a plus.

The goal of this project is to use interactive segmentation methods to collect data for instance segmentation models and then analyze the impact of the instance segmentation modality on the performance of action detection networks.