Social Interaction Analysis

Human social interaction relies on non-verbal signals such as facial expressions, voice, gaze behavior, head movements, and physiological responses. To study these signals under standardized yet realistic conditions, we developed two paradigms: the Berlin Emotion Recognition Test (BERT), a computer-based task for assessing emotion recognition, and the Simulated Interaction Task (SIT), a standardized social interaction paradigm for recording non-verbal behavior.

Using data from these paradigms, we develop multimodal machine learning methods to analyze social interaction behavior and affect-related signals. Our work focuses on integrating different modalities, including facial expression, voice, gaze, movement, and physiology, and on identifying interpretable behavioral patterns associated with differences in social communication. This includes research in conditions such as autism spectrum disorder, social anxiety disorder, attention deficit hyperactivity disorder (ADHD), and major depressive sisorder.

Beyond automated analysis, we also develop tools to make multimodal behavioral data more interpretable and clinically useful, for example through interactive visualization and explainable AI approaches.

From 01.12.2020 – 30.11.2024, this research was funded by the "Empathische Künstliche Intelligenz" (EKI) grant, FKZ 01IS20046.

Development

To develop a sensitive task, the pictures were extracted from video clips in which professional actors expressed the target emotions. The actors had been instructed with emotional scripts (e.g. imagine you receive an unexpected present) to perform the facial expressions, starting with a neutral expression. This led to a more naturalistic material. From each video clip, frames of three different intensities were extracted. These pictures of facial emotion expressions built the item pool for the BERT. This pool was reduced to the most sensitive items in a pre-study at an open house public event in Berlin, Germany, where large scientific institutions open their doors for the general public. In this pre-study with a sample of opportunity, 46 participants were asked to recognize the emotion of each of the items. Each picture was presented with the six basic emotions as possible answers. Based on their responses, for each video clip, we selected the picture, which discriminated best between low- and high-scoring participants. Additionally, we identified for each item the most difficult distractor out of the five incorrect emotion labels. In a follow-up online-study [3] with 436 participants, the selected pictures and distractors were tested and further improved with respect to reliability and discriminatory power by choosing the best eight items per emotion and most-difficult distractor.

Reference Values and Norming

Initial comparative values are already available to help interpret BERT results:

In an initial study [4], neurotypical participants gave an average of 79% correct responses, while participants with autism spectrum condition (ASC) gave 73% correct responses. Mean reaction times were 2832 ms for the neurotypical group and 4100 ms for the ASC group. However, these figures should be interpreted with caution, as BERT was combined with an additional imitation task in this study, which generally reduced performance in emotion recognition.

A more recent study [5] with a larger sample and without an additional imitation task currently provides a better first point of reference: mean recognition performance was 87.82% for the neurotypical group and 80.85% for the ASC group; mean reaction times were 2505 ms for the neurotypical group and 2963 ms for the ASC group.

However, these values do not yet represent validated normative values from a large reference sample and should therefore only be used as preliminary guidance.

A larger norming study, in which anonymized data from practices and institutions may be included, is currently being planned.

If you are interested in contributing to the norming study, please feel free to contact us at hanna.drimalla@uni-bielefeld.de