Berlin-based research project “XRIA” is developing an XR application for the lifelike depiction of top musicians

The research group Immersive Media & Communication group within the Vision & Imaging Technologies (VIT) department at Fraunhofer HHI is cooperating with two Berlin-based companies for the research project XRIA: the creative studio High Road Stories, serving as the project coordinator, as well as Berlin Phil Media GmbH. XRIA aims to develop a new digital application format enabling users to experience and actively interact with pop stars and top musicians in an eXtended Reality (XR) Experience.

Innovative technologies bring the XR Masterclass and XR concert experience to life

Building on the success of the Berliner Philharmoniker’s streaming platform Digital Concert Hall, volumetric video and spatial audio serve as the project’s technological foundation. These have so far been of limited use in standalone XR headsets due to the high data complexity.

Against this backdrop, the project partners will implement two specific use cases: an interactive learning application for music students in the form of an XR masterclass, and a multi-user XR concert experience for performances. The project is highly innovative in several areas because of the different content-related and technical challenges—the project partners are pooling their respective expertise in key research areas.

Targeted research focuses as drivers of innovation

In close collaboration, the project partners are jointly exploring and realizing new forms of interaction with musicians, the virtual environment, and the music itself, with the goal of creating deeply immersive and participatory experiences.

Fraunhofer HHI sets standards in the lifelike 3D capture and processing of real people using volumetric video. Musicians and the finest details of movement and instruments are portrayed through novel Gaussian splatting methods in combination with extremely high-resolution cameras. To achieve this, the team of Fraunhofer HHI is developing a novel 3D recording system, AI-based reconstruction methods, and a scalable streaming format for modern head-mounted displays that enables high quality with reduced bandwidth requirements.

Creative lead High Road Stories is responsible for developing the user experience and visual design of both use cases, supported by the vast expertise in music education and communication of Berlin Phil Media GmbH.

Berlin Phil Media GmbH, in collaboration with Henrik Oppermann (schallgeber), oversees the capture and playback of spatial audio and will set a new benchmark in the recording, processing, and playback of spatial audio with six degrees of freedom in XR. This allows users to move freely within the virtual space and perceive changes in sound based on their position.

The consortium is committed to bringing the research use cases into commercial practice.

Funding for the project, which runs until September 2027, is provided by Investitionsbank Berlin, with financing from the European Regional Development Fund (ERDF) and the State of Berlin. Learn more here.