Multimodal Behavior Processing Group

The research group Multimodal Behavior Processing, headed by Jun.-Prof. Dr. Hanna Drimalla, is dedicated to the automatic analysis of social interaction signals (e.g., facial expression, gaze behavior, voice etc.) using machine learning as well as speech and image processing. Three aspects are the focus of our research: the detection of positive and negative affect, the measurement of stress, and the analysis of social interaction patterns. All three have in common that they are multimodal and time-dependent phenomena. To address this complexity, we collect innovative training data and develop novel analysis methods.

Automatic affect recognition: Previous solutions for affect recognition are based on non-representative and unrealistic data sets. The first step towards better emotion recognition is therefore the collection of a balanced video data set in a situation as natural as possible using standardized test procedures. With this data material we want to develop algorithms of machine learning for affect recognition. Since the data material is video data, innovative approaches to the integration of different modalities (voice, facial expression, gaze behavior) can be used.

Computer-based stress measurement: The measurement of stress has so far focused mainly on self-report or individual parameters of the physiological response. In different stress paradigms we want to record the non-verbal behavior of test persons together with physiological markers. In addition, we developed a Digital Stress Test ( to collect a large data corpus on stress reactivity. Based on these multimodal data, we develop algorithms for automatic stress detection and validate them in natural environments.

Analysis of social embedding: To assess the social integration of a person, clinicians and researchers often use questionnaires. In order to capture the social integration of a person more sensitively and objectively, we want to develop an automatic analysis of online social interaction data. In a large online study, we compare the sensitivity of this approach to classical questionnaires. Furthermore, we identify characteristic and helpful interaction patterns using machine learning to predict the social embedding and resilience of a respondent.

Open Positions & Ongoing Studies

  • If you would like to join our group, you can assign to our job-mailing-list: We announce all open positions, thesis, and internships via this list.
  • If you want to support our research as a participant, you can assign to our study-mailing-list: We announce all online and offline experiments and studies via this list.