Robust Hand-Object Pose estimation from Multi-view 2D Keypoints
Hand-object pose estimation is a challenging task due to multiple factors like occlusion, and ambiguity in pose recovery. To overcome this issue, multi-view camera systems are used.
Using 2D keypoint detectors for hands and objects like Yolov8-pose and mmpose we can uplift the 2D detections to 3D. However, the detections usually are usually noisy, and some keypoints may be missing.
We want to utilize deep learning methods for smoothing, inpainting, and uplifting these detections to 3D in order to estimate the pose of the corresponding hands and objects.
The task is formulated as follows:
Given a sequence of noisy 2D key points for human hands and an object captured from calibrated camera views. Using a deep learning model, estimate a smooth trajectory of the hand and object poses.
- Knowledge about Deep Learning
- Knowledge about Pytorch
- Previous Knowledge about 3D data processing is a plus.
HiWI Position Porject Lab Human Activity Understanding
A HiWi position is available for the Lab Course Human Activity Understanding.
The position offers 6 h/week contract.
The lab involves:
- Practical Sessions where the students collect data from a color/depth sensor setup.
- Notebook Sessions where the students are introduced to a jupyter notebook with brief theoretical content and homework.
- Project Sessions, where the students are working on their own projects.
The main tasks of this position involve the following:
- Helping students with data collection in Practical and Project Sessions.
- Assisting during the notebook sessions with regard to the contents of the notebooks and homework.
- Knowledge about ROS.
- Knowledge about python.
- Basic Knowledge in Deep Learning
Generative Hands Object Interactions Using Diffusion Models
deep-learning, diffusion, stable-diffusion, action, smpl-x,mano, hand-object-interaction
Recently, there has been increasing success in the generation of human motion and object grasp. On the other hand, an increasing number of datasets capture human hands' interaction with surrounding objects in addition to action labels.
One advantage of Diffusion models is that they can easily conditioned on different types of input like text embeddings and control parameters.
Your task will include exploring the existing models and implementing the hand-object interaction diffusion model.
Datasets We could use:
Motion Generation Models
- Experience and interest in deep learning research.
- Knowledge and experience with Pytorch.