Student projects and final year projects at the Chair of Media Technology

Fuse AoA information with multipath delay componens for robust indoor localization.

We identify semantic information about human and object in the video by object detection and human detection and later classify the interaction information by a Transformer model. A public dataset will be used and we will collect the dataset ourselves (we provide the sensors and code to collect the dataset).

• Good programming skills in Python, Pytorch
• Knowledge of computer vision
• Basic knowledge of handling sequential data

Obtain signal information to determine different human activities tasks. Use nexmon firmware package.

https://ieeexplore.ieee.org/document/8307394

Python

AI/ML

Raspberry pi

The student will optimize through simulations the network atchitecture in order to obtain a robust positioning  estimate in a indoor set-up.

matlab

AI/ML

Identify and track multiple light sources in a  video stream.

The student must record the video multiple objects to the tracked in the video stream.

Python

Signal/image processing

The student must use a radar sytem to obtain vital signs of a patient.

The system must be embedded in a hospital bed.

Vital signs such as breathing rate and HR will be targeted; Others applications must be discussed.

Python

Radar

IQ signal processing

The student must set up a transmission scheme based on MIMO technology and OFDMA.

Take samples in different points inside an indoor environment for further processing of the signal characteristics to obtain an estimated position of the mobile node.

Python

WIreless communications with focus in channel state information and OFDMA MIMO systems

Knowledge in USRP not required but is a plus

The generation of a Digital Twin through 3D reconstruction is and will be a major topic for AR applications, 6G communication, and robotics.

Following [1], this thesis also follows the approach of using prior CAD database information for the 3D reconstruction. This thesis may use self-attention-based networks, confidence-part-based shape retrieval [2], and generative networks to partially reconstruct/ partially retrieve objects, thus achieving higher accuracy.

References

[1] Bokhovkin, Alexey and Angela Dai. “Neural Part Priors: Learning to Optimize Part-Based Object Completion in RGB-D Scans.” ArXiv  abs/2203.09375 (2022): n. pag. https://arxiv.org/abs/2203.09375

[2] Beyer, Tim and Angela Dai. “Weakly-Supervised End-to-End CAD Retrieval to Scan Objects.” ArXiv abs/2203.12873 (2022): n. pag. https://arxiv.org/abs/2203.12873

[3] Tang, Jiaxiang, Xiaokang Chen, Jingbo Wang and Gang Zeng. “Point Scene Understanding via Disentangled Instance Mesh Reconstruction.” ArXiv abs/2203.16832 (2022): n. pag. https://arxiv.org/abs/2203.16832

• Experience with Git
• Python (Pytorch)
• Knowledge in working with 3D Point Clouds and Meshes (preferable)

3D object detection has been a major task for point cloud-based 3D reconstruction of indoor environments. Current research has focused on having a low inference time for 3D object detection. While this is preferable, a lot of cases do not profit from this. Especially considering the use of a pre-defined static Digital Twin for AR and robotics application, thus this decreases the incentive for low inference time at the cost of accuracy.

As such this thesis will follow the approach of [1] (in this work only based on point cloud data) to generate proposals of layout and objects in a scene through for example [2]/[3] and use some form of optimization algorithm (reinforcement learning, genetic algorithm) to optimize to the correct solution.

Further, for more geometrical-reasonable results the use of a relationship graph neural network, as in [4], would be applied in the pipeline.

References

[1] Hampali, Shreyas, et al. “Monte Carlo Scene Search for 3D Scene Understanding.” 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)  (2021): 13799-13808. https://arxiv.org/abs/2103.07969#:~:text=We explore how a general, from noisy RGB-D scans.

[2] Chen, Xiaoxue, Hao Zhao, Guyue Zhou, and Ya-Qin Zhang. “PQ-Transformer: Jointly Parsing 3D Objects and Layouts From Point Clouds.” IEEE Robotics and Automation Letters  7 (2022): 2519-2526. https://arxiv.org/abs/2109.05566

[3] Qi, C., Or Litany, Kaiming He and Leonidas J. Guibas. “Deep Hough Voting for 3D Object Detection in Point Clouds.” 2019 IEEE/CVF International Conference on Computer Vision (ICCV)  (2019): 9276-9285. https://arxiv.org/abs/1904.09664

[4] Avetisyan, Armen, Tatiana Khanova, Christopher Bongsoo Choy, Denver Dash, Angela Dai and Matthias Nießner. “SceneCAD: Predicting Object Alignments and Layouts in RGB-D Scans.” ArXiv  abs/2003.12622 (2020): n. pag. https://arxiv.org/abs/2003.12622

• Python (Pytorch)
• Experience with Git
• Knowledge in working with 3D Point Clouds (preferable)
• Knowledge about optimization methods (preferable)

Virtual conference systems provide an alternative to physical meetings that have significantly grown in importance over the last years. However, larger events require the audience to be muted to avoid an accumulation of background noise and distorted audio. While this is sufficient for unidirectional meetings, many types of meetings strongly rely on the feedback of their audience, such as in performing arts.
In this project, we want to extend Zoom sessions with a simple Chatbot that collects the audience participation of each user using a straightforward button interface. Then, the system renders the overall audience feedback based on the feedback state collected from each user. The project combines signal and audio processing with the chance to gain practical experience with app development and SDKs.

References

• Create a Chatbot App
• Build a Chatbot

• Good knowledge in Nodejs/JavaScript

• Experience with Git

• Experience with Zoom SDK would be a plus

Autonomous grasping and manipulation are complicated tasks which require precise planning and a high level of scene understanding. Although robot autonomy is evolving since decades, there is still need for improvement, especially for operating in unstructured environments like households. Human demonstration can improve the autonomous robot abilities further to increase the task success in different scenarios. In this thesis we will work on user interaction methods for describing a robotic task via modifying the viewed scene.

Useful background:

- 3D Human-computer interfaces

- Game Design

- Digital signal processing

Required abilities:

- Experience with Unity and ROS

- Motivation to yield a good work

Autonomous grasping and manipulation are complicated tasks which require precise planning and a high level of scene understanding. Although robot autonomy is evolving since decades, there is still need for improvement, especially for operating in unstructured environments like households. Human demonstration can improve the autonomous robot abilities further to increase the task success in different scenarios. In this thesis we will work on learning from human demonstration for improving the robot autonomy.

Required background:

- Digital signal processing

- Computer vision

- Neural networks and other ML algorithms

Required abilities:

- Experience with Python or C++

- Experience with Tensorflow or PyTorch

- Motivation to yield a good thesis

Robotic teleoperation is often used to accomplish complex tasks remotely with human-in-the-loop. In cases, where the task requires very precise manipulation, virtual fixtures can be used to restrict and guide the motion of the end effector of the robot while the person teleoperates. In this thesis, we will analyze the geometry of virtual fixtures depending on the scene and task. We will use reinforcement learning to estimate ideal virtual fixture model parameters. At the end of the thesis, the performance can be evaluated with user experiments.

Useful background:

- Machine learning (Reinforcement Learning)

- Robotic simulation

Requirements:

- Experience with Python & Deep learning frameworks (PyTorch / Tensorflow...)

- Experience with a RL framework

- Motivation to yield a good outcome

Teleoperation Systems

Bilateral teleoperation with haptic feedback provides its users with a new dimension of immersion in virtual or remote environments. This technology enables a great variety of applications in robotics and virtual reality, such as remote surgery and industrial digital twin [1]. Figure 1 shows a generalized human-in-the-loop teleoperation system with kinesthetic feedback, where the operator commands a remote/virtual robot to explore the environment and experiences the interactive force feedback through the haptic interface. 

Teleoperation systems face many challenges caused by unpredictable environment changes, time-delayed feedback, limited network capacity, etc. [2]. These issues inevitably distort the force feedback signal, degrading the transparency and stability of the system. In the past decades, many control algorithms and hardware architectures were developed to tackle these problems in the past decades [3].

Time Domain Passivity Method (TDPA)

TDPA is a passivity-based control scheme that ensures the stability of teleoperation systems in the presence of communication delays [4] (See Figure 2.). It abstracts two-port networks from the haptic system and observes the energy flow between the networks. Passivity condition is maintained by dissipating extra energy generated by non-passive networks. Original TDAP suffers from position drift and feedback force jump [5], and one reason for the position drift is that the energy, which is generated by the delayed communication, is dissipated in the task space of the robots.

Jacobian Null-Space for Redundancy Robot

Many robot mechanisms have redundant degrees of freedom (rDOFs), which means that they have a larger number of joints than the number of dimensions of their task or configuration space. The null space of the Jacobian null space stands for the redundant dimensions which can be exploited to dissipate extra energy by damping the null space motion without affecting the task space [5].

Your Task and Target

In this work, we target at improving the performance of TDPA by considering dissipating energy generated by time delay and other factors in the Jacobian null-space of the kinesthetically redundant robots. With the help of the Jacobian null-space method, we can avoid dissipating energy in the task space, so as to alleviate position drift and force distortion while keeping the system passive. For more information, previous work can be referred to as [7-9].

In this master's internship, your work will include

1.      1. surveying the related algorithms

2.      2. constructing the simulation environment

3.      3. experimenting with the state-of-the-art Jacobian null-space TDPA method.

4.      4. analyzing system passivity in Cartesian task space, joint space, null space, etc.

Requirements

All requirements are recommended but not mandatory. However, you will need extra effort to catch up if you are unfamiliar with the following topics:

1.    3. Basic knowledge about robotics and control theory is favorable.

2.    2. Experience with robotics simulation software and platforms is favorable.

3.    1. C++, Matlab, and Python would be the primary working language. Basic knowledge about one or more of them is highly recommended.

This is a working student position at the Human Activity Understanding Group at the Chair of Media Technology.

The student will support various initiatives in game-engine based 3D simulation. We work with both Unity3D as well as Unreal Engine. Some videos of our simulator in action can be seen at: https://www.ce.cit.tum.de/lmt/forschung/themen/human-activity-understanding/

In particular, the student will extend and maintain the existing human activity simulators, and also work on interfacing VR hardware such as:

• Head-mounted displays like Oculus or Vive, Microsoft Hololens
• Motion capture system (OptiTrack)
• VR gloves
• Custom hardware setups

Also see: https://portal.mytum.de/schwarzesbrett/hiwi_stellen/NewsArticle_20220407_071536

Strong interest and experience in game development and 3D simulation. Experience with one of ideally both Unity3D and Unreal Engine, OR experience with Blender and interest in learning Unity3D/Unreal.

Please send in your CV and describe your experience at the time of application.

In shared control teleoperation, the robot intelligence and human input can be blended together to achieve improved task performance and reduce the human workload. In this topic, we would like to investigate how we can combine human input and robot intelligence effectively to achieve at the end full robot autonomy. We will employ robot learning from demonstration approaches, where we provide task demonstrations using teleoperation. 

We aim to test the developed algorithms in simulation and using Franka Emika robot arm.

Requirements:

Basic experience in C/C++

ROS is a plus

High motivation to learn and conduct research

Tasks:

• Help students with the basics of MATLAB (e.g. matrix operations, filtering, image processing, runtime errors)
• Correct some of the homework problems 
• Understand the DSP coursework material

We offer:

• Payment according to the working hours and academic qualification
• The workload is approximately 6 hours per week from May 2022 to August 2022
• Technische Universität München especially welcomes applications from female applicants

Application:

• Please send your application with a CV and transcript per e-mail to basak.guelecyuez@tum.de 
• Students who have taken DSP course preferred.

In this project, we investigate embodied (active) object detection in real-world environments. The goal is to actively adapt the path of an autonomous agent to increase the performance of an object detector.

• Object detection
• Pytorch
• Knowledge of graph optimization

Scene graphs were first proposed [1] as a data structure that describes the object instances in a scene and the relationships between these objects. The nodes in the scene graph represent the detected target objects, whereas the edges denote the detected pairwise relationships. A complete scene graph can represent the detailed semantics of a dataset of scenes.

Scene graph generation (SGG) is a visual detection task for building structured scene graphs. In this work, we would like to compare the three traditional SGG models: Neural Motifs [2], Graph R-CNN [3], and IMP [4]. We will train and evaluate the models using the Visual Genome dataset and other commonly used datasets. In addition, our dataset will be annotated and then utilized.

[1]: J. Johnson, et al. "Image retrieval using scene graphs."

[2]: R. Zellers, et al. "Neural motifs: Scene graph parsing with global context."

[3]: J. Yang, et al. "Graph r-cnn for scene graph generation."

[4]: D. Xu, et al. "Scene graph generation by iterative message passing."

• Good Programming Skills (Python, GUI design)
• Knowledge about Ubuntu/Linux/Pytorch
• Knowledge about Computer vision/Neural network
• Motivation to learn and conduct research

In this work, we would like to design a generative model for robot learning from demonstration. We will consider a table cleaning task, determine varying task conditions, and apply a generative model such that the robot can generalize to novel unseen conditions. We aim to use both simulation and Franka arm for testing our algorithms. 

Requirements:

Experience in C/C++

ROS is a plus

High motivation to learn and conduct research

When teaching robots remotely via teleoperation, the communication (between the human demonstrator and the remote robot learner) imposes challenges. Network delay, packet loss, and data compression might cause completely or partially degraded demonstration qualities. In this thesis, we would like the make the best out of varying quality demonstrations provided via teleoperation with haptic feedback. We will use different teleoperation setups to test the developed robot learning approaches:

Requirements:

Experience in C/C++

ROS is a plus

High motivation to learn and conduct research

Autonomous grasping and manipulation are complicated tasks which require precise planning and a high level of scene understanding. Although robot autonomy is evolving since decades, there is still need for improvement, especially for operating in unstructured environments like households. Human demonstration can improve the autonomous robot abilities further to increase the task success in different scenarios. In this thesis we will work on learning from human demonstration for improving the robot autonomy.

Required background:

- Digital signal processing

- Computer vision

- Neural networks and other ML algorithms

Required abilities:

- Experience with Python or C++

- Experience with Tensorflow or PyTorch

- Motivation to yield a good thesis

In this project, we aim to capture low-latency RGB data from a CSI camera in sync with an inertial measurement unit(IMU) and use the TPU unit for AI processing, especially for real-time segmentation applications. The first step of the project is to modify the provided embedded Linux kernel and the embedded board with an ARM Cortex processor to communicate with the TPU module. In the next step of the project, the synchronization of the inertial measurement data must be addressed with the frames of the RGB data from the camera. We use Edge TPU ML accelerator ASIC designed by Google that provides high-performance ML inferencing for TensorFlow Lite models.

Good programming skills in C++.

Strong understanding of embedded Linux programming.

Knowledge about Debian Kernel and UBoot.

Knowledge about ARM Cortex Processors.