Volumetric Video
Volumetric Video Studio (3rd Gen)
Since February 2023, our new volumetric capture studio is in operation in our new premises at Lanolinfabrik.
3D Human Body Reconstruction for eXtended Reality (XR)
New applications in media production especially for eXtended Reality (XR) require realistic 3D models of humans, which move and behave naturally. We developed a concept for 3D Human Body Reconstruction based on a 360° multi-view capture system for recording dynamic objects and the related processing pipeline to compute realistic moving 3D models of humans.
Volumetric Video Capture
We investigated several multi-view capture systems in the past years towards the goal to create a fully functional 360 volumetric capture studio.
Volumetric Video Workflow
Creating dynamic 3D models from humans is a challenging task. We developed a complete and automatic volumetric video workflow, which transforms several video streams into a coherent dynamically changing 3D model. The challenge is to reduce the amount of data by several orders of magnitude, while keeping the geometrical detail and texture quality.
Volumetric Video Production
Since 2016, when we started with the first capture systems for full 3D human body capture, several XR experiences have been produced in order to show the progress and capability of this new technology. The different productions cover a large range of genres demonstrating the capability of this new data format.
Augmented, Virtual and Mixed Reality
Dynamic Exposure Visualization of Air Quality Data with Augmented Reality
We are developing new ways to visualize environmental data such as air pollution or traffic for Augmented Reality applications. The objective is to provide the user with complex spatio-temporal information in a comprehensive way. We proudly present the world's first AR app that visualizes real-time environmental 3D sensor data, the Dynamic Exposure Visualization App (DEVA).
Interactive Visualization for Cultural Heritage
Digitization of cultural heritage artefacts is key to making them accessible anywhere at any time either for education or research. We investigate novel ways how to visualize and interact with such kind of data.
Interactive 3D Book Explorer
Touchless interaction in combination with 3D visualization offers the right technology for interactive exploration of three-dimensional data. A robust demonstrator was developed to show the potential of this technology.
Volumetric Data Visualization in Medicine
Complex three-dimensional medical data requires new approaches for visualization to better understand and interpret. Based on our expertise in 3D visualization, we developed an intuitive application for rendering of volumetric data.
Virtual Communication and Collaboration
Remote Assistance in Construction and Industry
Within the national funded German research project DigitalTWIN (BMWI) we have developed a mobile remote assistance system, which is able to connect a worker on the construction site with a remote expert, for example the architect in the office.
Immersive Video Communication
We are experts with many years of experience in the field of immersive video communication, working on concepts, systems and algorithm development for single and multi-user 2D/3D videocommunication scenarios supporting eye contact and correct gesture representation as well as 3D rendering on autostereoscopic displays.
Virtual Eye Contact
We developed a real-time multi-view video processing module for video conferencing systems offering direct eye contact between the local and remote conferees.
Hybrid Meetings and Collaboration
Due to our extensive experience in immersive video communication, we will further research and develop new approaches for future hybrid collaboration and communication technologies and services.
Deep Learning for Immersive Media and 3D Scene Processing
Foreground/Background Separation
Deep learning can be successfully applied to foreground/background separation tasks. In the context of our volumetric video pipeline, we investigate new approaches for object segmentation in order to increase robustness and quality of the segmentation.
Neural Representations and Rendering
We are investigating how analysis-by-synthesis based on neural geometry representations and neural rendering can be used in real-world multi-view reconstruction scenarios in the context of human body reconstruction and dynamic scenes.
Multi-View Stereo for High-Accuracy Human Body Reconstruction
We investigate and optimize deep learning-based multi-view stereo networks Vis-MVSNet for depth map estimation in the context of professional volumetric video reconstruction.
Previous Research Topics
Pose and Gesture Analysis
We research and develop fast and reliable algorithms for detection and tracking of whole body movements as well as movements of specific body parts. Our solutions focus on body, head, eye, hand and finger tracking as well as posture, movement and gait analysis. Our algorithms are based on non-intrusive, marker less sensors which provide 2D and/or 3D video signals, e.g. colour, depth-from stereo, time of flight or structured light. These algorithms are designed to work with a variety of cameras from low cost, mobile to high-end devices finding a best compromise between performance and price, from offline behaviour analysis to real-time human-computer interaction.
Behaviour Analysis for Human-Computer Interaction
Based on our detection and tracking algorithms as well as additional novel input sources such as voice control, we create novel human computer interaction modules. This involves calibration, fusion and interpretation of various data in a specific context. These modules are used in various applications such as behaviour analysis for ambient assisted living solutions, interaction with applications and media content as well as novel video communication applications.