June 8, 2021

Kick-off for VR project NeuroHum

The VR research project "NeuroHum" (Neurocognitively-guided modeling of virtual humans for enhanced realism in immersive media) started with its first project phase in early May. In the project, the "Vision and Imaging Technologies" department of the Fraunhofer Heinrich Hertz Institute (HHI), together with the "Mind-Body-Emotion" and "Neural Interactions and Dynamics" groups of the Department of Neurology at the Max Planck Institute (MPI) for Human Cognitive and Brain Sciences, is researching the perception of virtual humans and feeding the findings back into modeling and synthesis. The project is scheduled to run for four years.

Today, virtually depicted humans are an essential part of numerous application areas, for example movies, games or human-machine interfaces supported by computer graphics (CG). This is not only the case for traditional on-screen representations, but also for immersive interactive environments such as virtual (VR), augmented (AR), or mixed reality (MR). Creating realistic, believable, and emotive digital representations of people is still one of the most challenging tasks in CG, especially for real-time and interactive use cases. However, the mentioned features are essential for a variety of applications with socio-emotional components, such as trainings or therapies that involve other (virtual) humans or digital companions (e.g., for elderly people).

Current models of artificial humans are either hand-crafted or based on statistical assumptions. They do not systematically and objectively incorporate neurocognitive effects that virtual humans have on the user. More recently, we saw the introduction of photorealistic avatars reconstructed from real world footage or synthesized using novel generative deep learning approaches (neural animation, deep fakes). The key challenge in representing virtual humans is to avoid the "Uncanny Valley effect". This phenomenon describes that highly naturalistic virtual humans, which are still recognizable as artificial, can evoke uncomfortable feelings in viewers.

The goal of the NeuroHum collaborative project is to improve the perceptual and emotional realism of virtual humans in immersive media through neurocognitively driven computer graphics. In an iterative analysis-synthesis loop, the researchers will extract neurocognitive representations in naturalistic experiments from comprehensive assessments of sensation, user behavior, and physiology, map them to technical parameters, and integrate them into the generative process for virtual humans.

The "Computer Vision and Graphics" and "Interactive and Cognitive Systems" groups of Fraunhofer HHI's VIT department bring many years of experience in 3D video analysis and processing, MR and VR scenario development, and machine learning. The department represents a leading research unit in two areas closely related to this project: First, the capture, modeling, and animation of humans via hybrid representations that combine high-quality real-world data for realism with CG elements for animation. A complete volumetric capture studio is available for this purpose. It is capable of creating high-quality photorealistic 3D models of dynamic objects and people, which can be converted into hybrid representations for animatable photorealistic avatars. The second field covers the analysis of various data obtained by electroencephalography (EEG). EEG is a method used in medical diagnostics and neurological research to measure the brain's summed electrical activity by recording voltage fluctuations on the surface of the head. For the project, Fraunhofer HHI researchers identify psychophysiological quality measures from EEG recordings. Furthermore, they develop robust machine learning algorithms for the analysis of EEG signals and for image-processing visual quality estimation.

NeuroHum is funded within the framework of the cooperation program between the Max Planck Society and the Fraunhofer Society. Among other things, this cooperation program aims to expand networking within the scientific community. Both societies consider cooperation to be of particularly high importance and focus on outstanding results that attract the attention of the scientific community as well as the media and the general public.