The goal of this work is to develop a framework for fitting a Neural Radiance Field (NeRF) for hand-object interaction sequences while parametrizing the hand and object poses. 

The task could be described as follows:

Given a sequence of calibrated and synchronized multi-view RGB videos for two hands interacting with an object, along with the ground truth poses. We need to fit a NeRF network to the interaction sequence where the poses are fed as part of the input parameters.

The goal is to be able to synthesize novel poses that are similar to the input by altering the input pose parameters.

Basic Requirements

• Basic Knowledge of deep learning
• Python, PyTorch

Nice to Have:

• Knowledge about 3D computer vision or computer graphics.
• Familiarity with 3D deep learning libraries (Pytorch3D or Kaolin)

A HiWi position is available for the Lab Course Human Activity Understanding. 

The position offers 6 h/week contract.

The lab involves:

•  Practical Sessions where the students collect data from a color/depth sensor setup.
•  Notebook Sessions where the students are introduced to a jupyter notebook with brief theoretical content and homework.
• Project Sessions, where the students are working on their own projects.  

The main tasks of this position involve the following:

• Helping students with data collection in Practical and Project Sessions.
• Assisting during the notebook sessions with regard to the contents of the notebooks and homework. 

• Knowledge about ROS.
• Knowledge about python.
• Basic Knowledge in Deep Learning

Electrical Impedance Tomography (EIT) is an imaging technique that estimates the impedance of human body tissues by passing an alternating current through pairs of electrodes and measuring the voltage and current among other pairs of electrodes.

The inverse problem aims to reconstruct a cross-section tomographic image of the body part given the measurements.

EIT Wearable devices were applied successfully to the area of hand gesture classification and resulted in high-accuracy machine learning models [1][2].

The goal of the project is to research and test possible calibration approaches for sufficient and reproducible measurement results of EIT (similar to [3]), as well as go into hand gesture classification based on the measured impedance values of a human forearm [1][2].

We provide the wearable EIT band that is being developed in collaboration with Enari GmbH and the necessary computer vision building blocks for dataset collection.

The attached figure shows a pipeline of the image reconstruction

[1] Zhang, Yang & Harrison, Chris. (2015). Tomo: Wearable, Low-Cost Electrical Impedance Tomography for Hand Gesture Recognition. 167-173. 10.1145/2807442.2807480.

[2] D. Jiang, Y. Wu and A. Demosthenous, "Hand Gesture Recognition Using Three-Dimensional Electrical Impedance Tomography," in IEEE Transactions on Circuits and Systems II: Express Briefs, vol. 67, no. 9, pp. 1554-1558, Sept. 2020, doi: 10.1109/TCSII.2020.3006430.

[3] Zhang, Y., Xiao, R., & Harrison, C. (2016). Advancing Hand Gesture Recognition with High Resolution Electrical Impedance Tomography. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (pp. 843–850). Association for Computing Machinery.

• Knowledge with python
• Knowledge in Digital Signal Processing or Deep Learning

Hand-object pose estimation is a challenging task due to multiple factors like occlusion, and ambiguity in pose recovery.  To overcome this issue, multi-view camera systems are used.

Using 2D keypoint detectors for hands and objects like  Yolov8-pose and mmpose we can uplift the 2D detections to 3D. However, the detections usually are usually noisy, and some keypoints may be missing.  

We want to utilize deep learning methods for smoothing, inpainting, and uplifting these detections to 3D in order to estimate the pose of the corresponding hands and objects.

The task is formulated as follows:

Given a sequence of noisy 2D key points for human hands and an object captured from calibrated camera views. Using a deep learning model, estimate a smooth trajectory of the hand and object poses.

• Python
• Knowledge about Deep Learning
• Knowledge about Pytorch
• Previous Knowledge about 3D data processing is a plus.

In this project the task is to fit a parametric hand mesh model and a set of rigid objects to a sequence of multi-view RGB-D cameras. Existing models for hand key-point detection and 6DoF pose estimation for rigid objects models have significantly evolved in recent years. Our goal is to utilize such models to estimate the hand and object poses.

Related Work

1. https://dex-ycb.github.io/
2. https://www.tugraz.at/institute/icg/research/team-lepetit/research-projects/hand-object-3d-pose-annotation/
3. https://github.com/hassony2/obman
4. https://github.com/ylabbe/cosypose

• Knowledge in computer vision.
• Experience about segmentation models (i.e. Detectron2)
• Experience with deep learning frameworks PyTorch or TensorFlow(2.x).
• Experience with Pytorch3D is a plus.

The goal of this project is to use interactive segmentation methods to collect data for instance segmentation models and then analyze the impact of the instance segmentation modality on the performance of action detection networks.

Recently, there has been increasing success in the generation of human motion and object grasp. On the other hand, an increasing number of datasets capture human hands' interaction with surrounding objects in addition to action labels. 

One advantage of Diffusion models is that they can easily conditioned on different types of input like text embeddings and control parameters.  

Your task will include exploring the existing models and implementing the hand-object interaction diffusion model.

Datasets We could use: 

•  https://hoi4d.github.io/
•  https://taeinkwon.com/projects/h2o/
•  https://dex-ycb.github.io/

Motion Generation Models

• https://guytevet.github.io/mdm-page/
• https://goal.is.tue.mpg.de/

• Experience and interest in deep learning research.
• Knowledge and experience with Pytorch.