Behaviour, Intention and Mental State

A system that assists someone has to know what that person is doing and, more often
than not, what they are about to do. We decode human movement into sequences of
symbolic tokens that map back onto a set of action patterns, which makes recognition
usable for touchless interfaces in medical technology and in ambient systems.

Intention itself is not observable. We use latent-variable models in which the
observable action is the manifest variable of a hidden intentional process, and infer
intention as a symbolic set of patterns.
Intentions are multi-faceted, they change over time, and the same action can be driven
by several of them at once, so our models take the individual user and their context
as arguments rather than as noise, which is what makes them transfer across
environments and applications.

Emotion we approach in the same way, applying different psychological models depending
on the context of use and drawing on neurophysiological data — EEG, gaze,
pupillometry, skin conductance, heart rate — to reach states that are not visible in
the face at all. Our test settings are chosen because they are unforgiving: operating
rooms, de-escalation training for public security forces, motor rehabilitation.

B. Nierula, K. Tomotaki-Dawoud, M. Akguel, M.-T. Lafci, D. Przewozny, A. Hilsmann, P. Eisert, and S. Bosse. ‘Occlusion-Robust Multimodal Emotion Recognition in VR via Fusion of Facial Images and EMG’. in: Proceedings of the Workshop on Shaping Human-AI-powered Experiences in XR (SHAPEXR 2026) at the 31st ACM Conference on Intelligent User Interfaces (IUI 2026)

K. Katsarou, G. Zountsas, K. Tomotaki-Dawoud, A. Ehrenhoefer, P. Chojecki, D. Przewozny, D. Runde, I. M. Sauer, A. Mouakher, and S. Bosse. ‘Event-Level Detection of Surgical Instrument Handovers in Videos with Interpretable Vision Model’. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026, pp. 9184–9193

K. Tomotaki-Dawoud, B. Nierula, F. T. Siewe, T. Koch, D. J. Meyer, A. Bock, M. Heinze, D. Knuth, D. Martin, J. Schander, A. Hilsmann, P. Eisert, and S. Bosse. ‘Multi-View Gesture Recognition in Conflict Situations’. In: 2024 International Symposium on Multimedia (ISM). 2024 International Symposium on Multimedia (ISM). Dec. 2024, pp. 267–268