Student projects and final year projects at the Chair of Media Technology

In haptics, we want to display vibrations to users to convey specific feelings, like touching a specific surface material. With classical methods, we would then compress a vibration, transmit it to the user, and display it using an actuator. With heavy compression, important features can get lost, such that the resulting signal does not accurately represent the original one.

In this project, you will develop a classification system applied on spectrograms of vibrotactile signals.

The goal is to be able to classify material types and to analyze the impact of compression on the classification results.

Related paper:

[1] M. Strese, L. Brudermueller, J. Kirsch, und E. Steinbach, „Haptic Material Analysis and Classification Inspired by Human Exploratory Procedures“, IEEE Trans. Haptics, Bd. 13, Nr. 2, S. 404–424, Apr. 2020, doi: 10.1109/TOH.2019.2952118.

Prerequisites

Motivation

Chemical laboratories rely on a range of automated machines for tasks like liquid handling or centrifugation. Yet many repetitive procedures are still performed by human chemists, particularly if dexterous manipulation or adaptation to changing conditions are needed. Fully automating these tasks through traditional programming is impractical, since lab setups or execution protocols can vary significantly over time or between sites.

Learning from demonstration offers a compelling alternative. Instead of explicitly programming every motion and decision, an expert demonstrates a task and the robot learns a generalizable policy from a set of demonstrations. Combined with multimodal sensing, where data from cameras, tactile sensors and the robot's own proprioception are fused to understand the scene and guide execution, this enables a system that can adapt to variations and recover from disturbances.

The approach is not limited to laboratory tasks and can generalize to applications in manufacturing, assembly, etc.

System

The setup consists of a 4-DOF SCARA robot arm equipped with a parallel gripper, camera(s), tactile sensors and laboratory equipment. A demonstration setup already exists and is capable of showcasing the robot's manipulation capabilities in a laboratory environment.

Research Project

In this project, you will build on the existing demonstration setup and advance it towards robust learning from demonstration. This includes improving the physical setup, defining meaningful laboratory tasks to be learned and developing a multimodal sensing and learning pipeline. You will work with state-of-the-art models and algorithms in robot learning, adapt/tune them to the SCARA platform and evaluate how reliably new tasks can be acquired from a limited number of expert demonstrations. The work combines elements of robotics, machine learning, sensor fusion, and mechatronics.

Goals

Improve the existing demonstration setup and define a set of representative laboratory tasks for learning.

Improve and further develop a multimodal sensing pipeline that fuses data from cameras, robot proprioception and (possibly) tactile sensors to inform task execution.

Implement and tune learning-from-demonstration algorithms that allow the robot to acquire new tasks from expert demonstrations.

Evaluate robustness: the learned policies should handle (small) variations in conditions and recover from disturbances during execution.

Prerequisites

This work falls within the scope of indoor localization, more precisely the fingerprinting-based indoor localization. 

Your task will be to investigate the potential and outcomes of opting for a multi-layer reference map during the "Offline phase", which represents a potential improvement idea that has never been adopted or tested in current state-of-the-art fingerprinting schemes.

The aim here is to achieve a better trade-off between both performance and costs, yielding a better localization method. 

Prerequisites

The Chair of Media Technology at the Technical University of Munich (TUM) is offering a Master's/Bachelor's Thesis (or guided research) opportunity in the context of model-based deep learning for indoor localization, combining classical Kalman filtering theory with modern neural state estimation.

Indoor localization using WiFi RSSI and inertial sensors (IMU) is a sequential state-estimation problem naturally suited to Kalman filtering, where the IMU drives the motion model and WiFi RSSI provides the correction. However, the relationship between RSSI and position is highly nonlinear and building-specific, motivating a hybrid model- and data-driven approach. KalmanNet replaces the analytically-derived Kalman gain with a GRU-based recurrent network learned from data, retaining the interpretability and low-data efficiency of the classical filter while implicitly learning complex, unmodeled dynamics.

The thesis investigates KalmanNet on the IMUWiFine dataset (220 per-AP RSSI values, 9-channel IMU, fine-grained ground-truth trajectories across a multi-floor building) along a complexity ladder: from a fully active system with complete per-access-point observability, through intermediate degradation stages (AP reduction, sectorization), to a fully passive distributed antenna system (DAS), where only a single aggregated RSSI value is available per time step.

Selection and Reproduction of Baseline

As KalmanNet has not previously been applied to this dataset, the first part of the thesis involves reproducing a KalmanNet baseline and validating it against classical filters (EKF, UKF) and the existing LSTM-based end-to-end baseline. The implementation should support:

• Per-AP RSSI and per-antenna observation vectors

• IMU-driven state-evolution modeling

• GRU-based Kalman gain estimation, trained end-to-end

Degradation Analysis and Passive DAS Reformulation

Building upon the baseline above, the second part focuses on systematically degrading observability and identifying what must change for KalmanNet to remain viable as the system moves toward a passive

DAS. This involves:

• Reducing and sectorizing the AP/antenna observation set

• Investigating the non-invertibility and dimensionality collapse of aggregated RSSI observations

• Synthesizing aggregated RSSI from existing per-AP data to emulate the passive DAS setting

• Increasing reliance on IMU-based state evolution to compensate for degraded observability

Prerequisites

Large-scale diffusion models and, similarly, flow models are strong natural image priors [1,2]. This capability is enabled by both architectural advancements, such as Transformers [3], and internet-scale training.

This capability of the diffusion models enabled perceptual image restoration where non-generative or small-scale models failed to produce realistic restorations [4]. However, these methods rely on external feature extractors to condition the diffusion backbones on the degraded inputs. We showed that this is unnecessary, as the backbone itself exhibits a degree of robustness to degradation [5]. However, this requires processing both the degraded input and the diffused image simultaneously, resulting in high computational complexity.

This project will focus on improving the performance of this family of methods.

[1] Rombach, Robin, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. "High-resolution image synthesis with latent diffusion models." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684-10695. 2022.

[2] Liu, Xingchao, Chengyue Gong, and Qiang Liu. "Flow straight and fast: Learning to generate and transfer data with rectified flow." arXiv preprint arXiv:2209.03003 (2022).

[3] Peebles, William, and Saining Xie. "Scalable diffusion models with transformers." In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4195-4205. 2023.

[4] Li, Xin, Yulin Ren, Xin Jin, Cuiling Lan, Xingrui Wang, Wenjun Zeng, Xinchao Wang, and Zhibo Chen. "Diffusion models for image restoration and enhancement: A comprehensive survey." International Journal of Computer Vision 133, no. 11 (2025): 8078-8108.

[5] Eteke, Cem, Alexander Griessel, Wolfgang Kellerer, and Eckehard Steinbach. "BIR-Adapter: A parameter-efficient diffusion adapter for blind image restoration." Pattern Recognition (2026): 113824.

Prerequisites

Understanding human hand and object interaction is fundamental for meaningfully interpreting human action and behavior.

With the advent of deep learning and RGB-D sensors, pose estimation of isolated hands or objects has made significant progress.

However, despite a strong link to real applications such as augmented and virtual reality, joint reconstruction of hand and object has received relatively less attention.

This task focuses on accurately reconstructing hand-object interactions in three-dimensional space, given a single RGB image.

Prerequisites

In today's digital era, image and video content dominate online traffic, accounting for the majority of global data transmission [1]. Efficient compression is therefore essential for delivering high-quality content under limited bandwidth and storage constraints. Recently, Deep Neural Networks (DNNs) have emerged as a powerful alternative to traditional compression methods, leveraging nonlinear representations to improve compression efficiency and visual quality [1].

However, beyond compression performance, practical deployment requires careful consideration of computational cost and energy consumption, especially on resource-constrained devices such as smartphones. Importantly, neither the number of parameters nor the number of operations in a DNN directly reflects its true energy consumption. Models with fewer parameters or operations may still consume more energy due to hardware-specific factors such as memory access patterns [2].

This project aims to systematically measure and model the runtime and energy consumption of DNNs across heterogeneous devices, including desktop GPUs, laptops, and/or smartphones. The objectives include refining and extending an existing energy measurement setup to ensure reliability and reproducibility, analyzing the relationship between model structure, runtime, and energy consumption, and designing predictive models for runtime and energy estimation across devices.

Existing approaches in the literature include device-specific [3] and device-adaptive [4] runtime estimation methods. Students are encouraged to explore and extend these approaches and to propose novel runtime and energy modeling techniques for the generalization across devices.

[1] JS Gomes, M. Grellert, FLL Ramos and S. Bampi, "End-to-End Neural Video Compression: A Review," IEEE Open Journal of Circuits and Systems, vol. 6, pp. 120-134, 2025.

[2] X. Yang, J. Kwon, Y. Li and Y. Chen, "Designing Energy-Efficient Convolutional Neural Networks using Energy-Aware Pruning," CVPR, 2017.

[3] LL Zhang, S. Han, J. Wei, N. Zheng, T. Cao, Y. Yang and Y. Liu, "NN-Meter: Towards Accurate Latency Prediction of Deep-Learning Model Inference on Diverse Edge Devices," MobiSys, pp. 81-93, 2021.

[4] C. Feng, LL Zhang, Y. Liu, J. Xu, C. Zhang, Z. Wang, T. Cao, M. Yang and H. Tan, "LitePred: Transferable and Scalable Latency Prediction for Hardware-Aware Neural Architecture Search," NSDI, 2024.

Prerequisites

Generative AI models (e.g., LLMs, diffusion models) are increasingly used to create multimodal content (text + images) for applications like interactive storytelling, educational tools, and digital media. However, ensuring that generated content is safe, unbiased, and age-appropriate remains a critical challenge. Manual moderation is unscalable, and existing automated filters often lack contextual understanding or multimodal reasoning.

 This thesis explores the development of automated pipelines to evaluate and refine AI-generated content, with a focus on:

• Real-time safety assessment of text-image pairs.
• Automatic re-prompting to guide models toward compliant outputs.
• Adaptability to diverse use cases (e.g., children’s toys, educational platforms).

Objectives

1. Multimodal Safety Evaluation:

   • Investigate state-of-the-art metrics (e.g., CLIP-based similarity, toxicity scores, emotional valence) to detect unsafe or age-inappropriate content in text-image pairs.
   • Develop a lightweight, interpretable scoring system combining:
     • Text analysis (e.g., perspective API, custom fine-tuned classifiers).
     • Image analysis (e.g., NSFW detectors, aesthetic/emotional classifiers).
     • Cross-modal alignment (e.g., does the image match the text’s intent and safety constraints?).

2. Re-Prompting Strategies:

   • Design adaptive prompting techniques to iteratively refine outputs (e.g., using reinforcement learning or constrained decoding).
   • Explore few-shot learning to generalize safety rules across domains (e.g., fairy tales vs. scientific explanations).

3. Benchmarking and Evaluation:

   • Curate a multimodal dataset of edge cases (e.g., subtle biases, ambiguous contexts).
   • Compare against human annotations and existing tools (e.g., Google’s Perspective API, LAION filters).
   • Optimize for latency and computational efficiency (critical for embedded/edge devices).