February 8, 2022
Decai Chen, research associate in the “Immersive Media and Communication” group of the “Vision and Imaging Technologies” department at the Fraunhofer Heinrich Hertz Institute (HHI), was awarded the Best Paper Award at the 2021 International Conference on 3D Immersion (IC3D). Together with his co-authors, he was able to prevail against three other nominated papers. The award-winning paper "Accurate Human Body Reconstruction for Volumetric Video" focuses on the use of deep-learning-based multi-view stereo networks to achieve high-fidelity human body reconstruction in volumetric video production.
In light of rapid development and high demand for augmented reality (AR) and virtual reality (VR) applications, volumetric video is becoming increasingly important as it bridges the gap between the real and virtual world. Also known as free-viewpoint video, volumetric video provides an immersive user experience in realistic scenes from arbitrary unconstrained viewpoints. It can therefore be applied in a wide range of areas, not only in classic communications and media, but also in therapy and rehabilitation. However, in order to exploit the full potential of this technology, it is essential to ensure high-quality reconstruction and representation of human agents.
While some volumetric video systems show great potential for high quality 3D reconstruction, their sophisticated setups with active depth cameras are very costly and therefore not very desirable. Decai Chen's team proposes a different approach in their paper "Accurate Human Body Reconstruction for Volumetric Video": Using an adapted volumetric reconstruction pipeline, they aim to achieve accurate general 3D reconstruction of elements - specifically human bodies - at a fine geometry level using only RGB cameras. For this purpose, the researchers optimized and adapted a state-of-the-art multiview stereo network (MVS) to leverage deep-learning-based feature representation.
To align the neural network information with visual data of studio images and realistic human bodies, the team also created a real 3D dataset of human bodies captured in a studio environment. In addition to that, they propose a novel post-processing approach for filtering and fusing depth maps to obtain high-quality point clouds for mesh reconstruction. In contrast to the standard filtering and fusion approach, which only removes points that are either geometrically inconsistent or have low photometric confidence levels, Decai Chen and his team go one step further, additionally filtering out points that fall within the background mask of any camera and lie outside the visual cone of a given number of cameras. This allows for accurate depth map estimation under the existing setup of multiview cameras in a recording studio.
Extensive experiments showed that the presented method achieves results with significantly better quality than existing approaches. The team conducted their research within the framework of the KIVI project funded by Investitionsbank Berlin (IBB) and the Horizon 2020 project INVICTUS .
The Best Paper Award was presented to the authors during the IC3D virtual conference program. The International Conference on 3D Immersion (IC3D) is an annual conference integrated within Stereopsia EUROPE, the World Immersion Forum, and technologically co-sponsored by the Signal Processing Society of the IEEE.