Fraunhofer Institute for Telecommunications,
Heinrich-Hertz-Institut, HHI
Video Communication and ApplicationsNewsTechnologies and SolutionsResearch GroupsProjectsPublicationsPeople and Contact

Advanced View Synthesis

For 3D video solutions, generic depth-enhanced video formats, such as MVD and associated coding methods have been developed, which support different types of 3D displays, like stereoscopic glasses-based or autostereoscopic glasses-free displays with different number of views. In order to generate the required number of views at individual positions for different displays, we develop improved view synthesis algorithms, which are able to reduce coding artifacts.

Figure 1: Overview of reliability-based view synthesis (for MVD format)
[“Ballet” Images are from Interactive Visual Media Group of Microsoft Research].

The processing diagram of our extended view synthesis algorithm is shown in Figure 1 for the case of a multi-view video plus depth format (MVD). With some variations, this algorithm is also used applicable to other 3D formats, like layered depth video (LDV). The algorithm contains the general building blocks, used in depth image-based rendering:

  • Layer extraction: Prior to projection, different layers are classified with respect to their reliability in rendering: a reliable main layer, an unreliable foreground boundary layer and an unreliable background boundary layer.
  • Layer projection: First, the reliable main layer is projected from a neighboring original position to the requested intermediate position. The layers from both views are blended, using a viewing parameter to allow smooth navigation along the viewing range. Afterwards, unreliable information is only projected to fill missing information
  • Hole filling and filtering: Disoccluded information along foreground objects is filled from adjacent background information, small holes are filled from neighboring pixels, Finally, unnaturally looking foreground edges are smoothed at the very edge by applying low-pass filtering

Publications

[1] K. Müller, K. Dix, P. Merkle, T. Wiegand, “Temporal Residual Data Sub-Sampling in LDV Representation Format”, Proc. IEEE 3DTV Conference, Tampere, Finland, June 2010.

[2] K. Müller, A. Smolic, K. Dix, P. Merkle, and T. Wiegand, “Coding and Intermediate View Synthesis of Multi-View Video plus Depth”, Proc. IEEE International Conference on Image Processing (ICIP'09), Cairo, Egypt, pp. 741-744, Nov. 2009.

[3] K. Müller, A. Smolic, K. Dix, P. Kauff, and T. Wiegand, “Reliability-based Generation and View Synthesis in Layered Depth Video”, Proc., IEEE International Workshop on Multimedia Signal Processing (MMSP2008), Cairns, Australia, pp. 34-39, Oct. 2008.

[4] A. Smolic, K. Müller, K. Dix, P. Merkle, P. Kauff, and T. Wiegand, “Intermediate View Interpolation based on Multi-View Video plus Depth for Advanced 3D Video Systems”, Proc. IEEE International Conference on Image Processing (ICIP'08), San Diego, CA, USA, pp. 2448-2451, Oct. 2008.

[5] K. Müller, A. Smolic, K. Dix, P. Merkle, P. Kauff, and T. Wiegand, “View Synthesis for Advanced 3D Video Systems”, EURASIP Journal on Image and Video Processing, Special Issue on 3D Image and Video Processing, vol. 2008, Article ID 438148, 11 pages, 2008. doi:10.1155/2008/438148.