Automating Stop-Motion Animation using Computer Vision and AI
Beschreibung
Can computer vision and generative AI eliminate the tedious manual work involved in producing stop-motion animation?
Stop-motion animation is created by physically manipulating objects and capturing them frame by frame. While this provides artists with direct control over the scene, the production process is highly labor-intensive. For example, animators often need to carefully avoid appearing in the captured image or manually remove hands, supports, rigs, and other unwanted objects during post-processing.
Recent advances in computer vision and generative AI provide an opportunity to automate parts of this workflow.
The goal of this master's thesis is to develop an AI-assisted stop-motion production pipeline that automatically identifies and corrects artifacts introduced during the physical manipulation of a scene.
Your Task
You will develop a prototype that combines computer vision, image processing, and generative AI to automate selected steps of stop-motion production.
A central use case will be automatic hand removal:
- The animator physically moves a character or object.
- An image is captured, potentially while the animator's hand is still visible.
- The system automatically detects and segments the hand.
- The occluded area is reconstructed using information from other frames and/or a generative inpainting model.
- The corrected frame is automatically added to the stop-motion sequence.
This would allow an animator to concentrate on physically creating the motion rather than repeatedly moving their hands out of the camera view.
Several approaches could be investigated:
- Single-frame generative inpainting: Use an AI model to reconstruct the masked region based on the surrounding image.
- Multi-frame reconstruction: Search neighboring frames for observations of the region currently hidden by the hand.
- Background modelling: Build a representation of the static scene and use it to reconstruct occluded regions.
- Hybrid approach: Combine information from previous frames with generative inpainting when no reliable visual observation is available.
- The last approach is particularly interesting because stop-motion provides information that ordinary image inpainting does not: many images of almost the same physical scene are available.
Possible Research Questions
- How reliably can hands and manipulation tools be detected and segmented in stop-motion images?
- Can AI remove a hand even when it partially occludes the animated object?
- Does using information from neighboring frames improve reconstruction compared with single-image inpainting?
- How can temporal consistency be maintained between AI-corrected frames?
- Can the system distinguish intentional object movement from unwanted changes to the scene?
- How much manual post-processing can an automated pipeline eliminate?
- Can the approach enable a different stop-motion workflow in which the animator does not have to leave the camera view between frames?
Voraussetzungen
- Strong programming skills, preferably in Python
- Interest in Computer Vision and Generative AI
- Experience with PyTorch, OpenCV, or similar frameworks is beneficial
- Interest in image/video processing, animation, or computational photography
- Motivation to develop and experimentally evaluate a working prototype
Betreuer:
Improving IMU-Based Motion Capture through Visual Tracking and Sensor Fusion
Beschreibung
Can visual tracking from a VR headset compensate for drift and inaccuracies in wearable motion-capture systems?
Inertial motion-capture systems such as the Rokoko Smartsuit and Smartgloves enable full-body and hand motion capture without requiring a dedicated optical motion-capture studio. However, IMU-based tracking can suffer from accumulating drift, positional errors, and inaccuracies in the reconstructed motion, particularly during longer recording sessions.
The goal of this thesis is to develop a hybrid motion-capture approach that combines the motion data from a Rokoko system with visual marker tracking using the cameras of a Meta Quest 3.
Visual markers attached to the Smartsuit and Smartgloves provide external spatial observations that can be used to correct or constrain the IMU-based motion estimate. Sensor fusion between the two modalities could thereby produce more accurate, stable, and drift-resistant motion capture.
During recording, the Quest 3 should operate in passthrough mode, allowing the performer to see the real environment naturally while motion and marker data are captured.
Your Task
In this thesis, you will design and implement a prototype for visual-inertial sensor fusion for motion capture.
The work will include:
- Recording body and hand motion using a Rokoko Smartsuit and Smartgloves
- Attaching suitable visual markers to selected body and hand locations
- Detecting and tracking those markers using the Meta Quest 3 cameras
- Establishing the spatial relationship between the Rokoko, Quest, and marker coordinate systems
- Synchronizing the visual tracking measurements with the Rokoko motion data
- Developing a sensor-fusion method that uses visual observations to correct or constrain IMU-based motion capture
- Integrating the system with Quest 3 passthrough so that users can see and interact with their physical surroundings during a recording
- Experimentally evaluating the improvement in motion-capture quality
A particularly interesting question is whether the visual system needs to reconstruct the complete pose or whether occasional visual position/orientation constraints are already sufficient to eliminate most IMU drift.
Possible Research Questions
- How accurately can visual markers attached to a mocap suit be tracked by a head-mounted XR device?
- Can visual marker observations compensate for accumulating positional and rotational drift in IMU-based motion capture?
- Where should markers be placed on the body to provide the largest improvement with the fewest markers?
- How should IMU and visual measurements be fused when markers are only intermittently visible?
- Can visual constraints improve the capture of hands and arms, where small errors can produce significant pose inaccuracies?
- How robust is the approach during fast movements, self-occlusion, and longer recording sessions?
- How does the proposed hybrid system compare with the original Rokoko motion capture in terms of accuracy and drift over time?
Voraussetzungen
- Strong programming skills in C++, C#, and/or Python
- Interest in Virtual Reality, motion capture, or computer vision
- Basic knowledge of 3D transformations, coordinate systems, and linear algebra
- Experience with Unity, OpenXR, OpenCV, or Meta Quest development is beneficial
- Interest in sensor fusion, tracking, or robotics is a plus
Betreuer:
Interactive Multimodal Story Generation Grounded in 3D
Beschreibung
Can multimodal story generation (para text + para image) be improved by grounding it in 3D?
Recent advances in generative AI enable the creation of impressive images and videos from textual prompts. However, creating a coherent visual story across multiple shots remains challenging. Characters may change appearance, objects can move unexpectedly, and spatial relationships may be lost between shots.
The recent StoryBlender: Inter-Shot Consistent and Editable 3D Storyboard with Spatial-temporal Dynamics approach addresses this problem by generating structured 3D storyboards rather than independent 2D images. Its pipeline combines semantic-spatial grounding, persistent assets in a common 3D space, spatial-temporal dynamics, and multi-agent planning with verification.
In this thesis, you will investigate how these ideas can be extended toward interactive multimodal story generation.
Your Task
The goal is to design and prototype a system that translates a multimodal story description into a sequence of coherent 3D scenes or shots.
Depending on the focus of the thesis, the system could accept combinations of:
- Natural-language descriptions or scripts
- Reference images for characters, objects, or environments
- Sketches or spatial layouts
- Existing 3D models or scenes
- Potentially speech or other interaction modalities
The system should interpret these inputs, construct an internal representation of the story, and generate a sequence of consistent and editable 3D shots.
Possible Research Directions
The thesis could investigate one or more of the following:
- Agentic generation: Can specialized AI agents take responsibility for tasks such as script interpretation, scene composition, asset selection, camera placement, animation, and verification
- Interactive editing: How can users modify generated results using natural language, for example: “Move the camera closer to the character.”, “Keep the same room, but make the next shot an over-the-shoulder shot.”, “Replace the robot with this reference model in all shots.”
A key advantage of a 3D representation is that cameras and scene assets can subsequently be changed directly rather than requiring the entire image to be regenerated, which is also one of the motivations behind StoryBlender.
Voraussetzungen
- Strong programming skills, preferably in Python
- Experience or interest in Generative AI, LLMs, or Vision-Language Models
- Interest in computer graphics, 3D scene generation, animation, or visual storytelling
- Familiarity with tools such as PyTorch, Hugging Face, Blender, Unity, or similar frameworks is beneficial
- Motivation to implement and experimentally evaluate a research prototype
Betreuer:
Automating Stop-Motion Animation with Virtual Reality
Beschreibung
How can Virtual Reality make the creation of stop-motion animations faster, more intuitive, and more accessible?
Stop-motion animation is a creative but labor-intensive process: objects must be positioned manually, photographed frame by frame, and repeatedly adjusted to achieve the desired motion. Virtual Reality offers an opportunity to rethink this workflow by allowing users to create and manipulate animations directly in an immersive 3D environment.
Your Task
In this master’s thesis, you will design and develop a VR application for Meta Quest that supports the creation of stop-motion animations.
The thesis will involve:
- Designing an intuitive VR interaction concept for creating stop-motion animations
- Developing a VR prototype for Meta Quest
- Choosing and evaluating an appropriate development approach, for example:
- Unity / OpenXR for a native VR application, or
- Immersive Web SDK (IWSDK) for a WebXR and browser-based 3D application
- Implementing functionality for positioning and manipulating objects, defining animation frames, previewing motion, and supporting repetitive animation tasks
- Exploring approaches for partially automating the stop-motion workflow
- Conducting a user study to evaluate the proposed approach
- Analyzing usability, efficiency, learnability, and overall user experience
Possible Research Questions
- Can VR reduce the effort required to create stop-motion animations?
- Which parts of the stop-motion workflow can be effectively automated?
- How accurately and efficiently can users define object motion in VR?
- How does a VR-based workflow compare with conventional stop-motion animation in terms of task completion time, usability, and user experience?
- What are the advantages and limitations of a native VR implementation versus a WebXR-based approach for this type of application?
Voraussetzungen
- Experience or strong interest in Virtual Reality / Extended Reality
- Programming experience and an interest in Unity/C# or web-based 3D technologies
- Interest in WebXR and browser-based immersive applications is a plus
- Interest in Human-Computer Interaction, 3D interaction, or animation
- Motivation to develop and evaluate a working research prototype
Betreuer:
Multi-level Fingerprinting-based Indoor Localization Scheme
Indoor Localization, Multipath, Fingerprinting
Multi-layer reference map implementation for fingerprinting-based indoor localization.
Beschreibung
This work falls within the scope of indoor localization, more precisely the fingerprinting-based indoor localization.
Your task will be to investigate the potential and outcomes of opting for a multi-layer reference map during the "Offline phase", which represents a potential improvement idea that has never been adopted or tested in current state-of-the-art fingerprinting schemes.
The aim here is to achieve a better trade-off between both performance and costs, yielding a better localization method.
Voraussetzungen
Required:
- Python and/or Matlab
- Basic knowledge in signal processing and wireless communication
- Analytical thinking and creativity
Kontakt
To get more info/details and initiate contact:
Majdi.abdmoulah@tum.de
(Please attach your CV and transcript)
Betreuer:
KalmanNet-Based Indoor Localization: From Active Sensing to Passive Distributed Antenna Systems
Kalman filter, extended Kalman filter (EKF), unscented Kalman filter (UKF), KalmanNet, Kalman gain, GRU (gated recurrent unit), model-based deep learning, hybri
This thesis applies KalmanNet (a hybrid model- and a data-driven Kalman filter) that replaces the analytically derived Kalman gain with a GRU-based recurrent network — to indoor localization using the IMUWiFine dataset, which combines 220 per-AP WiFi RSSI values with 9-channel IMU data across a multi-floor building. The work follows a complexity ladder from a fully active sensing setup with complete per-access-point observability, through intermediate degradation stages such as AP reduction and sectorization, to a fully passive distributed antenna system (DAS) where only a single aggregated RSSI value is available per time step, investigating how a learned Kalman gain can compensate as observability collapses.
Beschreibung
The Chair of Media Technology at the Technical University of Munich (TUM) is offering a Master's/Bachelor's Thesis (or guided research) opportunity in the context of model-based deep learning for indoor localization, combining classical Kalman filtering theory with modern neural state estimation.
Indoor localization using WiFi RSSI and inertial sensors (IMU) is a sequential state-estimation problem naturally suited to Kalman filtering, where the IMU drives the motion model and WiFi RSSI provides the correction. However, the relationship between RSSI and position is highly nonlinear and building-specific, motivating a hybrid model- and data-driven approach. KalmanNet replaces the analytically-derived Kalman gain with a GRU-based recurrent network learned from data, retaining the interpretability and low-data efficiency of the classical filter while implicitly learning complex, unmodeled dynamics.
The thesis investigates KalmanNet on the IMUWiFine dataset (220 per-AP RSSI values, 9-channel IMU, fine-grained ground-truth trajectories across a multi-floor building) along a complexity ladder: from a fully active system with complete per-access-point observability, through intermediate degradation stages (AP reduction, sectorization), to a fully passive distributed antenna system (DAS), where only a single aggregated RSSI value is available per time step.
Selection and Reproduction of Baseline
As KalmanNet has not previously been applied to this dataset, the first part of the thesis involves reproducing a KalmanNet baseline and validating it against classical filters (EKF, UKF) and the existing LSTM-based end-to-end baseline. The implementation should support:
• Per-AP RSSI and per-antenna observation vectors
• IMU-driven state-evolution modeling
• GRU-based Kalman gain estimation, trained end-to-end
Degradation Analysis and Passive DAS Reformulation
Building upon the baseline above, the second part focuses on systematically degrading observability and identifying what must change for KalmanNet to remain viable as the system moves toward a passive
DAS. This involves:
• Reducing and sectorizing the AP/antenna observation set
• Investigating the non-invertibility and dimensionality collapse of aggregated RSSI observations
• Synthesizing aggregated RSSI from existing per-AP data to emulate the passive DAS setting
• Increasing reliance on IMU-based state evolution to compensate for degraded observability
Voraussetzungen
• Background in electrical engineering, computer science, or a related field
• Interest in signal processing, sensor fusion, and machine learning
• Programming experience (e.g., Python, PyTorch)
• Motivation to work independently on a technically challenging and research-oriented topic
Kontakt
To get more info/details and initiate contact:
Majdi.abdmoulah@tum.de
(Please attach your CV and transcript)
Betreuer:
Efficient Methods for Diffusion-Based Image Restoration
Beschreibung
Large-scale diffusion models and, similarly, flow models are strong natural image priors [1,2]. This capability is enabled by both architectural advancements, such as Transformers [3], and internet-scale training.
This capability of the diffusion models enabled perceptual image restoration where non-generative or small-scale models failed to produce realistic restorations [4]. However, these methods rely on external feature extractors to condition the diffusion backbones on the degraded inputs. We showed that this is unnecessary, as the backbone itself exhibits a degree of robustness to degradation [5]. However, this requires processing both the degraded input and the diffused image simultaneously, resulting in high computational complexity.
This project will focus on improving the performance of this family of methods.
[1] Rombach, Robin, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. "High-resolution image synthesis with latent diffusion models." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684-10695. 2022.
[2] Liu, Xingchao, Chengyue Gong, and Qiang Liu. "Flow straight and fast: Learning to generate and transfer data with rectified flow." arXiv preprint arXiv:2209.03003 (2022).
[3] Peebles, William, and Saining Xie. "Scalable diffusion models with transformers." In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4195-4205. 2023.
[4] Li, Xin, Yulin Ren, Xin Jin, Cuiling Lan, Xingrui Wang, Wenjun Zeng, Xinchao Wang, and Zhibo Chen. "Diffusion models for image restoration and enhancement: A comprehensive survey." International Journal of Computer Vision 133, no. 11 (2025): 8078-8108.
[5] Eteke, Cem, Alexander Griessel, Wolfgang Kellerer, and Eckehard Steinbach. "BIR-Adapter: A parameter-efficient diffusion adapter for blind image restoration." Pattern Recognition (2026): 113824.
Voraussetzungen
This is a research-focused project. The students are expected to be eager to do research.
Knowledge of Diffusion Models.
Experience with Computer Vision.
PyTorch.
Kontakt
cem.eteke@tum.de
Betreuer:
Measurement and Estimation of Energy Consumption in Neural Image and Video Compression
Beschreibung
In today's digital era, image and video content dominate online traffic, accounting for the majority of global data transmission [1]. Efficient compression is therefore essential for delivering high-quality content under limited bandwidth and storage constraints. Recently, Deep Neural Networks (DNNs) have emerged as a powerful alternative to traditional compression methods, leveraging nonlinear representations to improve compression efficiency and visual quality [1].
However, beyond compression performance, practical deployment requires careful consideration of computational cost and energy consumption, especially on resource-constrained devices such as smartphones. Importantly, neither the number of parameters nor the number of operations in a DNN directly reflects its true energy consumption. Models with fewer parameters or operations may still consume more energy due to hardware-specific factors such as memory access patterns [2].
This project aims to systematically measure and model the runtime and energy consumption of DNNs across heterogeneous devices, including desktop GPUs, laptops, and/or smartphones. The objectives include refining and extending an existing energy measurement setup to ensure reliability and reproducibility, analyzing the relationship between model structure, runtime, and energy consumption, and designing predictive models for runtime and energy estimation across devices.
Existing approaches in the literature include device-specific [3] and device-adaptive [4] runtime estimation methods. Students are encouraged to explore and extend these approaches and to propose novel runtime and energy modeling techniques for the generalization across devices.
[1] JS Gomes, M. Grellert, FLL Ramos and S. Bampi, "End-to-End Neural Video Compression: A Review," IEEE Open Journal of Circuits and Systems, vol. 6, pp. 120-134, 2025.
[2] X. Yang, J. Kwon, Y. Li and Y. Chen, "Designing Energy-Efficient Convolutional Neural Networks using Energy-Aware Pruning," CVPR, 2017.
[3] LL Zhang, S. Han, J. Wei, N. Zheng, T. Cao, Y. Yang and Y. Liu, "NN-Meter: Towards Accurate Latency Prediction of Deep-Learning Model Inference on Diverse Edge Devices," MobiSys, pp. 81-93, 2021.
[4] C. Feng, LL Zhang, Y. Liu, J. Xu, C. Zhang, Z. Wang, T. Cao, M. Yang and H. Tan, "LitePred: Transferable and Scalable Latency Prediction for Hardware-Aware Neural Architecture Search," NSDI, 2024.
Voraussetzungen
Strong coding skills in Python and ML libraries, background in machine learning, motivation for research and experimentation, experience with mobile or embedded platforms would be a plus
Kontakt
serdar.caglar@tum.de