Video-based Tracking

We believe that in the near future traditional computer input devices like the keyboard and the mouse are going to be supplemented or replaced by more natural interfaces. For many applications, vision-based trackers are suitable as interaction devices. We have developed a family of non-intrusive, markerless computer vision based modules for human computer interaction in indoor environments. The modules are designed to work with a variety of cameras ranging from low cost to high end devices allowing to find a best compromise between performance and price. All trackers may communicate via established protocols (e.g. Polhemus protocol on serial line) as well as via specially designed DCOM interfaces.

Pointing with the fingertip is an easy and natural way to interact with virtual objects shown on a 3D display. Our hand tracker measures the 3D position of the user's fingertip at a rate of 50 Hz. Besides the pointing actions, some simple gestures signalling stop, start, left and right are recognized. In order to increase the tracking range, the basic pair of stereo cameras can be extended by additional cameras. Without using any markers the video-based object tracker computes the position and orientation of known, geometrically defined objects. These data are used to overlay 3D graphical objects and to annotate information about the scene on live video. The object tracker can also be used as a flexible interacting device to manipulate virtual objects with real tools.

Using sensor fusion techniques, the various tracking modules may be combined to set-up novel interaction tools allowing smart applications. For example, a new and very natural pointing tool is created by combining the head (pupil) tracker with the hand (finger tip) tracker; with such a tool the user can easily control the position of a marker on the screen by pointing in the desired direction. Tracking the users gaze vector allows a speech recognition engines to react more reliable and context sensitive.

These tracking modules are developed in the mixed3D project, which is funded by the Federal Republic of Germany, Ministry of Education and Research under Grant No. 01 BD 250.