In the European FP7 research project MUSCADE the Fraunhofer Heinrich Hertz Institute HHI develops calibration and assistance tools for multi-camera setups that enable high-quality 3D production for next-generation 3DTV with no need for special glasses.
Technical Background
For efficient production and transmission of 3DTV content suitable for glasses-free viewing, we need to generate virtual views on the receiver side. The number of views can then be adapted to the specific autostereoscopic display in use. Depth image-based rendering (DIBR) is one such technology which generates disparity maps containing depth information of the scene using sophisticated depth estimation techniques.
Only a limited number of real camera views with associated disparity maps are needed to generate new virtual camera views. Thus only a limited number of HD video streams with associated disparity maps will be transmitted to the receiver side – two views in the case of MVD2 (Multi-Video + Depth) and four views in the case of MVD4. Researchers at Fraunhofer HHI have developed a real-time encoder for MVC (Multi-View Video Coding) which compresses the four video streams by exploiting redundancies within the four cameras views. This saves bandwidth and allows for transmission of all four video streams with associated disparity maps using existing broadcasting infrastructures such as satellite networks.
The MUSCADE project aims to support a wide range of 3D displays with backwards compatibility to legacy stereoscopic displays. Thus two of the transmitted views coincide with a stereoscopic video pair which supports stereoscopic displays without view interpolation. The multi-camera rig developed by the MUSCADE project meets all these requirements.
Two cameras inside a mirror-box capture the two views needed for a stereoscopic display while two additional satellite cameras allow for wide baseline applications. Even so, MVD4 content acquisition still remains a daunting challenge. The multi-camera rig needs to be calibrated to a high degree of accuracy to ensure that all cameras are positioned on a common baseline. Such a geometric constraint is necessary for efficient camera recording and subsequent estimation of disparity maps.
Against this background, Fraunhofer HHI is developing a multi-camera version of its Stereoscopic Analyzer assistance system (STAN) which will ensure proper calibration of the multi-camera rig and allow for real-time registration of captured camera signals. All remaining vertical disparities are eliminated during the registration process which means that the video streams can now be used for accurate estimation of dense disparity maps. As corresponding pixels lie on the same line in the corresponding images, this speeds up the search for pixel-dense point correspondences.
Furthermore, this geometric setup also simplifies the view generation process – pixels are shifted horizontally only to generate virtual camera views. This technique thus allows for parallelized processing. In combination with the technologies developed by the partners in the MUSCADE consortium, an efficient means of generating content for the next generation of 3D displays is now on track.