November 17, 2021
The Fraunhofer Heinrich Hertz Institute (HHI) has started the new project VoluProf (Volumetric Mixed Reality Professor for Omnipresent and User-Optimized Teaching). It was recently launched at the “3IT - Innovation Center for Immersive Imaging Technologies ”. The project will use mixed reality (MR) technologies to make online teaching more direct, interactive, and user-centered. For this purpose, the researchers develop photo- and audio-realistic avatars of real lecturers. The project started in September 2021 and will run until August 2024. The Federal Ministry of Education and Research (BMBF) is funding the project with 1.98 million euros.
In many respects current online teaching is still inferior to classroom instruction. The main reason for this is that traditional online teaching uses static video and audio scripts to convey information. The potential of these methods is limited because learners can rarely interact with the instructors and cannot give feedback.
VoluProf aims to change this by bringing together the best of the real and virtual worlds: Combining the interactivity of face-to-face instruction with the flexibility and inclusion of online teaching. To achieve this, the project is developing avatars of lecturers with whom learners can interact by using a MR headset.
Three Fraunhofer HHI research groups are involved in the project. The “Computer Vision and Graphics” group of the “Vision and Imaging Technologies” department is responsible for the creation of photorealistic virtual humans and their animation based on volumetric video data. The group’s team will develop AI methods for the creation of the avatars of the lecturers, focusing on context-based synthesis of natural movements, gestures and facial animation. This way, the virtual lecturers will be as realistic as possible.
The “Interactive and Cognitive Systems” group of the same department is responsible for perceptual aspects and focuses on the assessment and prediction of the perceived quality of image and immersion. They will conduct usage studies, model the relevant perceptual aspects, and integrate the created models into the VoluProf system.
The "Multimedia Communications" group of the "Video Communication and Applications" department will focus on the efficient transmission of the volumetric avatar to the learners using modern telecommunication networks. Particularly important here is to achieve a minimal interaction latency to guarantee an optimal user experience. Furthermore, the researchers will develop rate adaptation technologies that quickly adapt to different network conditions and thus ensure a smooth user experience.
The project team also investigates the ethical, legal and social implications of the developed platform and continuously integrates them into the technical development.
In addition to Fraunhofer HHI, Deutsche Telekom AG, Volucap GmbH, Aristech GmbH, the University of Rostock and the German Research Center for Artificial Intelligence (DFKI) are involved in the project.