Interactive Media - Human Factors Publications - Pastoor, Conomis: Mixed Reality Displays
3D Video-Communication
Siegmund Pastoor and Christos Conomis, In Schreer, O., Kauff, P., Sikora, T. (Hrsg.), 2005. Wiley Verlag
14.1 Introduction
The term "Mixed Reality" (MR) was coined by Paul Milgram about a decade ago. It defines a system where both real and virtual (computer-generated) things appear to coexist in the same space. The potential value of MR systems is increasingly recognized and appreciated in a wide variety of fields, including simulation, medicine, architecture, driving, maintenance, industrial design and entertainment (Tamura, 2002). A great potential lies in the capability of not just mimicking or enhancing properties of the real world, but of exceeding the physical laws governing reality, such as the possibility of stepping back and forth in time, performing an "undo" function, and rendering hidden things visible.
The automotive industry is one of the early adapters of MR technologies. Emerging applications include assembly and disassembly of vehicles in production, maintenance and repair tasks, electrical troubleshooting, in-situ visualization of calculated stress distribution and deformation in a crash test (Friedrich and Wohlgemuth, 2002), as well as exterior and interior design. For example, Ohshima et al. (2003) applied MR technologies in order to evaluate design concepts using a partly physical and partly virtual car mock-up. The real components of the mock-up included the driver's seat, parts of the dashboard, the steering wheel and some basic devices for controlling the audio, air conditioner and navigation systems. The doors, ceiling, pillars, as well as the entire instrument panel with meters and displays were virtually overlaid onto the driver's field of view. The real control devices provided haptic feed-back when touched, while their actual appearance was easily changeable using graphical overlays superimposed into vision. Hence, different designs could be visualised, interactively altered and evaluated from a usability point of view. In another application a new car model was visualised almost entirely in the digital domain (Tamura, 2002). Interested customers could walk around the virtual car model and even get into it by "opening" a door. A real seat was prepared to confirm the sight from the driver's position through the virtual windows.
Figure 14.1 illustrates how an advanced MR headset can be used in a car repair task. The transparent mirror in front of the eye superimposes computer-generated information upon the technician's field of view. Instructions for assembling parts of the engine, including the tightening torques of the various screws, are available without having to leave the vehicle. It is even possible to directly communicate with the spare parts inventory in order to see whether a required replacement part is in stock.
Apart from the automotive sector, MR systems are increasingly used in other industrial applications particularly in order to give assistance and feedback in real-time processes. For example, Aiteanu et al. (2003) developed a welding system where the traditional protection helmet was replaced by a special helmet-mounted display. The display provides a better view of the working area; if required, an online agent suggests corrections of the welding gun's position, by analysing the electrical welding parameters, and points on welding errors. Pettersen et al. (2003) developed an easy to use MR system for interactively programming waypoints and specified actions of an industrial robot. During the programming sequence, the system presents visual feedback of results for the operator, allowing him to inspect the process result before the robot has performed the actual task.
Various applications of MR technologies are emerging in the areas of city planning and architecture (se e.g. Kato et al. (2003) and Project ARTHUR (2003)) since it is much easier to setup and modify virtual models instead of physical ones. Usually, the virtual models of buildings and facilities are complemented by simple real objects representing the virtual ones. The real objects operate as placeholders; they can easily and intuitively be grasped and moved on the planning desk, hence, providing a kind of tangible interface. When looking through the MR display, the user sees the virtual buildings at the locations of the placeholders.
In a similar way, antique ruins and monuments can be reconstructed virtually by computer software and visually superimposed onto the real location. Hence, visitors of archaeological sites can see temples, statues and buildings of old cities virtually placed on their original fundaments in their original surroundings (ARCHEOGUIDE, 2002).
Regarding applications for entertainment, MR technologies allow creating novel video games, where players can exercise their ingenuity in both real and virtual spaces - making full use of their hands and feet as interactive devices, instead of being limited to game pads, steering wheels, pedals, and the like (Tamura, 2002; Stapleton 2003). Broll et al. (2004) developed the Mixed Reality Stage, an interactive pre-production environment which can be employed to rapidly plan, arrange and visualise scenes for music shows and theatre presentations. Virtual models and characters are projected into a real, but down-scaled model of the stage with tracked generic modules representing actors and decoration. Lighting conditions, moveable stage components and actors' paths can be planned and modified intuitively.
The overlay of virtual images on the real world is increasingly used to enhance computer supported collaboration between people. Kato et al. (2002) developed a MR conferencing application where the remote collaborators are presented on virtual monitors which can be freely positioned in space. Users can view and interact with virtual objects using a shared virtual whiteboard. Further applications in the context of video communications are discussed in detail in a special chapter of this book (see Chapter 5).