Tracking for Projector-Camera Systems

Projector-camera systems are a useful tool to perform projection mapping and to present information directly matched with the local environment. This is beneficial in different contexts for example to allow robots to project their intentions spatially correct into working environments, to enhance medical procedures by projecting additional information, or to improve interaction with objects with spatial user interfaces e.g. in industrial visual inspection scenarios.

Projection Distortion-based Object Tracking

In dynamic scenes, object tracking maintains the illusion of an enhanced target object if the physical and virtual objects are well aligned. However, traditional trackers based on texture or contour information are often distracted by the projected content and tend to fail. We have developed a model-based tracking strategy, which directly takes advantage from the projected content for pose estimation in a projector-camera system.

An iterative pose estimation algorithm captures and exploits visible distortions caused by object movements. In a closed-loop, the corrected pose allows the update of the projection for the subsequent frame. 

Synthetic frames simulating the projection on the model are rendered and an optical flow-based method minimizes the difference between edges of the rendered and the camera image. Since the thresholds automatically adapt to the synthetic image, a complicated radiometric calibration can be avoided.


The pixel-wise linear optimization is designed to be easily implemented on the GPU. Our approach can be combined with a regular contour-based tracker and is transferable to other problems, like the estimation of the extrinsic pose between projector and camera.

The following video shows how the 3d pose of the moving object is estimated from the distortion of the projection, which is then corrected to match the object surface again.


Markerless Closed-Look Projection Plane Tracking

The recent trend towards miniaturization of mobile projectors is allowing new forms of information presentation and interaction. Projectors can easily be moved freely in space either by humans or by mobile robots. Our methods are also able to dynamically track the orientation and position of the projection plane only by analysing the distortion of the projection by itself, independent of the presented content. This enables distortion-free projection with a fixed metric size for moving projector-camera systems.


AR Framework

The algorithms have been integrated in an augmented reality (AR) software implementing the closed-loop projection-based object tracking. The software allows to create and configure AR scenes, performs pose estimation in real-time and provides a websocket interface to communicate with other sensors in a multimodal system.



This topic is funded in the project EASY-COHMO.


N. Gard, A. Hilsmann and P. Eisert,Projection Distortion-based Object Tracking in Shader Lamp Scenarios, IEEE Transactions on Visualization and Computer Graphics, vol. 25, no.11, pp. 3105-3113, Nov 2019.

N. Gard and P. Eisert,
Markerless Closed-Loop Projection Plane Tracking for Mobile Projector-Camera Systems, IEEE International Conference on Image Processing (ICIP), Athens, Greece, October 2018.