3D Human Body Reconstruction for eXtended Reality (XR)

Future media productions such as walk-in movies, interactive TV shows or virtual reality experiences require innovative computer vision technologies. In contrast to classical animation of virtual characters, the objective is to create complete and highly realistic 3D reconstructions of moving persons. To advance these vision technologies, the Fraunhofer Heinrich Hertz Institute HHI has combined computer vision and computer graphics to represent real persons through realistic and natural moving 3D models - a so-called Volumetric Video. These models are ready for integration in virtual or photo-realistic 3D scenes.

Technical Background

The system allows to capture a real person or object and integrate it directly into a computer-generated virtual world. The real person is represented by a dynamic three-dimensional model which can then be observed from arbitrary viewpoints in the virtual scene. In this way, the user is able to virtually walk around this person and watch it from different perspectives. Facial expressions as well as moving clothes are reconstructed at high geometrical detail and texture quality. A maximum of naturalness of the visual impression is achieved.

Technically, the system consists of several camera-based, highly precise modules for depth estimation which are arranged in a flexible way around the person. They capture its textured 3D shape simultaneously from various directions and angles. The data will be fused and transformed to a consistent, natural and dynamic 3D representation of the person. Post-processing modules adapt the data structures and formats and allow for direct integration in standardized post-production workflows and virtual reality players for VR glasses.


  • Integration of real persons in computer generated scenes
  • Realistic and lifelike impression
  • High quality and naturally moving dynamic 3D models
  • Flexible and modular multi-camera system for customized setups
  • Fully equipped lighting and capture system for 360° full body reconstruction
  • Real-time preview on set
  • Efficient cloud processing of high-quality texture and geometry
  • Hardware solution for scalable and local high-performance cloud server systems
  • Integrated processing of data for virtual and augmented reality (VR/AR) glasses and systems
  • Fast integration in professional post production workflows
  • Recording up to 50 frames per second
  • Possible extension to complete real-time workflow