New media applications such as interactive TV shows or presentations for viewing with virtual reality glasses require innovative computer vision technologies. The objective is the 3D reconstruction of persons, objects and environments. To advance these vision technologies, the Fraunhofer Heinrich Hertz Institute HHI combined computer graphics and image-based rendering to represent real persons in real-time through realistic and natural moving 3D models. These models are ready for integration in virtual or real scenes. Possible image adaptations include semantic changes such as changes in the direction of a model's gaze, eye contact or the animation and correction of movements.
The system allows to capture a real person or object and integrate it directly into a computer-generated virtual world. The real person is represented as a three-dimensional model which can then be observed from arbitrary viewpoints in the virtual scene. In this way, the user is able to virtually walk around this person. A maximum of naturalness of the visual impression is achieved.
Technically, the system consists of several camera-based high precision depth estimation modules which are arranged in a flexible way around the person. They capture it simultaneously from various directions and angles. The data will be fused and transformed to a consistent, natural and dynamic 3D representation of the person. Post-processing modules adapt the data structures and formats and allow for direct integration in standardized post-production workflows and virtual reality applications required for VR glasses.
- Integration of real persons in computer generated scenes
- Realistic and lifelike impression
- High quality and naturally moving dynamic 3D models
- Flexible and modular multi-camera system for customized setups
- Real-time preview on set
- Possible extension to complete real-time workflow
- Integrated processing of data for virtual and augmented reality (VR/AR) glasses and systems
- Fast integration in professional post production workflows
- Recording up to 50 frames per second