Multimodal AR-assisted ENT-Surgery

Duration: September 2018 - August 2021


Intraoperative imaging is currently performed using white light illumination and surgical decisions are taken without additional information of further visual assisting tools resulting in disadvantages as not all available information have been exploited. The project MultiARC targets closing this information gap and reducing surgical duration as well as complications and revision surgeries.

Goals and Approaches

The focus of MultiARC is to combine hyperspectral imaging, 3D-measurement and a true-to-scale augmentation of clinical images for ear, nose and throat (ENT)-surgery in microscopy and endoscopy. To achieve this goal, we develop an intraoperative system for a collaborative and immersive decision-making chain. This system includes a novel illumination unit with suitable evaluation algorithms for tissue differentiation and a stereo-image-based measurement solution. The intraoperative images are updated according to the captured information and augmented with the additional information (tissue type and measurements). To achieve a true-to-scale 3D visualization, a robust calibration setting of stereo-zoom optics will be developed. The evaluation and a transfer into other application areas is done in the AR/VR-Living-Lab.

Innovation and Outlook

The MultiARC system allows intuitive and interactive handling of AR-data for multilateral telepresence. The multimodal data is visualized in the binocular of the microscope, on AR/VR-glasses and 3D-displays. The augmentation of virtual prostheses allows an exact decision-making of the implant sizes and will reduces revisions.

Fraunhofer HHI is responsible for the development of the novel hyperspectral tissue differentiation algorithms as well as the stereo-image-based measurement solution for microscopic and endoscopic applications.


Project Consortium

  • pripares GmbH
  • ARRI Medical
  • Fraunhofer HHI
  • Ubimax GmbH
  • Charité – Universitätsmedizin Berlin, Klink für HNO
  • Schölly Fiberoptic GmbH (associated partner)