Real-time Stereo-to-Multiview Conversion

IP core and FPGA reference implementation for depth estimation

Glasses-free 3D will be the future of 3D home entertainment. To display stereoscopic 3D content on multiview displays a 3D format conversion is required. Developed at Fraunhofer HHI, the real-time stereo-to-multiview conversion engine allows playback of any stereoscopic 3D video content on most existing autostereoscopic displays. Costly offline conversion is no longer needed and personal 3D viewing preferences can be adjusted on-the-fly.

Depth estimation – the heart of stereo-to-multiview conversion – is now available as a pure hardwired IP core suited for FPGA and ASIC implementation. For evaluation purposes a reference FPGA implementation can be provided.

Technical Background

To generate content for autostereoscopic multiview displays Fraunhofer HHI’s stereo-to-multiview conversion creates additional virtual camera perspectives from available stereo input. The generation of additional views is based on depth maps estimated from the initial stereoscopic images. Depth maps represent pixel-by-pixel information about the depth of the scene.

With this depth information, additional virtual camera perspectives can be generated by applying depth image- based rendering (DIBR). Thus it is possible to generate an arbitrary number of views and adapt the number and position of these virtual views to the specific properties of existing and future glasses-free 3D displays. To provide the optimal 3D viewing experience, the depth representation can be adapted individually – including the level of perceived depth by adjusting the inter-axial distance as well as the position of scene objects in relation to the screen plane by adjusting horizontal image translation.

Based on the already available software reference implementation a bit accurate hardware IP core for the depth estimation process has been implemented which is suited for the integration into consumer devices. The IP core is based on a dedicated hardware architecture, which does not need to incorporate any additional processing power form other resources, e.g. processor cores. The IP core features standardized video interfaces for video input and output for easy System on Chip (SoC) integration. For validation purposes a 1080p30 real-time capable FPGA reference implementation for Altera Stratix V evaluation boards is available.

Specifications

  • Based on long-lasting experience in stereo-tomultiview conversion by Fraunhofer HHI
  • Hardware IP Core available
  • Pure hardwired implementation, no processor core needed
  • Suited for various FPGA architectures and ASIC technologies
  • 1080p30 real-time capable FPGA reference implementation on Altera Stratix V featuring HDMI video I/O
  • Low latency processing (3 frames)
  • Low system clock requirements (160 MHz for 1080p30)
  • Available as VHDL or technology specific IP core including C-reference models and test benches for verification