Interacting with our environment evokes complex responses across behavioral, subjective, and physiological levels. Researchers often face the challenge of achieving a good balance between naturalistic and unconstrained behavior while ensuring precise and insightful biological readouts. Virtual reality (VR) enables naturalistic (i.e., dynamic and interactive) experiments with simultaneous recording of multimodal psychophysiological and behavioral measures, such as electrophysiology, autonomic measures, eye and body movements. These data can be used to identify characteristic patterns in clinical and healthy samples.

In this symposium, we present the work of five researchers, who use interactive VR paradigms beyond “button presses” to study the psychophysiology of socio-affective processes.

Sabrina Gado will kick-off the symposium with findings from a recent meta-analysis and an experimental study on the effects of social anxiety on exploration behavior and autonomic responsiveness.

Leon Kroczek will then present how emotional expressions and interpersonal distance influence socio-affective processing in a naturalistic approach task which combines mobile EEG and VR.

Michael Gaebler will present a set of studies on the behavior and psychophysiology of perceiving virtual humans in 3D and immersive VR.

Marius Rubo will present eye-tracking and behavioral data from dyadic interactions in social VR.

Finally, Marta Andreatta will conclude the symposium with a clinical study testing in VR the (possible) beneficial effect of exposure therapy on pathological learning mechanisms in anxiety patients.

In summary, this symposium showcases and highlights VR's potential to exploit rich behavioral patterns allowing for a nuanced understanding of the physiological underpinnings of (mal)adaptive socio-affective processes.

Virtual reality (VR) has emerged as a promising tool for investigating social behavior. By tracking gaze, movements, and physiological responses, VR facilitates a fine-grained analysis of different aspects of social approach-avoidance tendencies. Adaptive social functioning critically relies on effectively balancing these behaviors with exaggerated social avoidance often being associated with mental disorders such as social anxiety. Here, we present insights from a meta-analysis and two experimental studies exploring which context factors modulate the behavioral and autonomic responses of socially anxious individuals.

The meta-analysis comprises 188 studies investigating the effects of trait social anxiety on several outcome variables. It showed that the naturalism of a study setting is a relevant modulating factor. For the 21 included VR studies, we observed an association of social anxiety with heightened subjective distress, increased skin conductance, and decreased visual attention on social stimuli.

With two experimental studies (both N=48), we focused on how people adapt after social learning experiences. We found that previous social encounters with friendly or unfriendly virtual agents significantly influenced participants’ exploration behaviors, including interpersonal distance and spontaneous fixations, as well as their subjective evaluations of these agents. Importantly, trait social anxiety was associated with higher fear ratings and increased interpersonal distance. Although autonomic responses indicated the valence of social encounters, no long-term physiological adaptations persisted during the social approach-avoidance test.

This research demonstrates the potential of VR to investigate (mal)adaptive social behavior in conditions resembling real-live encounters.

Persons display numerous social and affective cues when engaging in face-to-face social interactions, including facial emotional expressions and interpersonal distance. Interactive partners can use these cues to process another person’s intentions or to assess threat in social encounters. However, it remains an open question whether different social cues are processed independent from each other or interactively, i.e. whether emotional information conveyed by proximity can impact processing of facial expressions. To investigate this question, the present study (N = 46) combined a naturalistic approach task in Virtual Reality with mobile EEG measurement. In the study, participants walked towards virtual agents who would react to the approach either at close (1 m) or far (2.5 m) distances by displaying either angry, neutral, or happy facial expressions. ERPs elicited by the agents reaction as well as ratings of valence and arousal were assessed as dependent variables. Results show that close compared to far distances were experienced as less pleasant, more arousing, and resulted in enhanced LPP components. Furthermore, angry expressions were experienced as less pleasant than happy expressions, while emotional expressions in general were more arousing, and elicited a more negative EPN component than neutral expressions. Importantly, however, neither the ratings nor the ERPs indicated an interaction between interpersonal distance and facial emotional expressions (also supported by Bayesian analysis). Overall, our findings suggest that socio-affective information in interpersonal distance and facial emotional expressions is processed independent from each other and that both cues contribute to the evaluation of face-to-face interactions.

Immersive Virtual Reality (iVR) affords the interaction with computer-generated environments, of which virtual humans have become an essential part. Yet, creating realistic and emotive digital representations of humans is still challenging, for example because of the “uncanny valley” effect, which suggests that highly photorealistic virtual humans that can still be recognized as artificial are unappealing. In social interactions, faces are a particularly important source of information, for example to identify persons and their emotions. While face perception typically happens in 3D, it is often studied with 2D stimuli.

I will present results from three sets of behavioural (large-scale online and lab-based) and EEG studies, in which we used 3D-reconstructed or computer-generated faces with different identities, emotional expressions, and stylization levels under 2D and 3D viewing conditions also using iVR technology to investigate (1) the influence of depth information on the identification and psychophysiology of face perception as well as (2) behavioural and physiological indices of realism, particularly zooming into the critical zone of the “uncanny valley”.

The studies are part of two projects, NEUROHUM and 3DIL, whose aims are to integrate neurocognitive indices of realism into the generation of virtual humans and to use realistic 3D faces in iVR to increase eyewitness identification accuracy, respectively.

Overall, neurocognitive measures of realistic virtual humans can not only advance neuroscience but also computer graphics (e.g., immersive media) and forensic psychology (e.g., lineup procedures).

Virtual Reality (VR) allows users not only to interact with computerized environments and computer agents in isolation, but also to interact with others in shared worlds. Tracking speech behavior, eye gaze and facial expressions in dyadic interactions in VR, we observed that (1) interaction behavior closely corresponded to patterns known from face-to-face interactions, (2) anonymized communication was not associated with social disinhibition, (3) social anxiety levels had similar effects on interaction behavior as known from face-to-face interactions, (4) participants' tendencies to gaze towards a partner's eye region was psychometrically distinguishable from their tendency to gaze towards eyes of people depicted in images and (5) participants could be identified with high accuracy based on their eye gaze distribution while interacting. While the use of Social VR in psychological research is still in its infancy, the technique may enrich both experimentation and data collection in several fields. I briefly present open-source tools which serve to allow other researchers to reproduce the in-house social VR setup used in my research.

Anxiety disorders are the most prevalent mental disorders and around 30-40% of the patients show relapses of the symptoms after therapy. These patients are characterized by an incapacity to identify safety, responding anxiously in numerous situations. In this study, we investigated whether exposition therapy might improve safety learning. Eighty-nine patients and 37 healthy controls underwent a classical context conditioning in virtual reality (VR). During an acquisition phase, a loud and desperate female scream (unconditioned stimulus, US) was unpredictably presented in one virtual office (threatening context, CTX+) but never in the other office (safety context, CTX-). During an extinction phase, which followed, participants revisited the two contexts, and the US was never delivered. Twenty-eight patients were then re-tested after an exposition therapy. Compared to healthy controls, patients demonstrated stronger physiological defensive responses in the safe context (CTX-) but reported higher subjective anxiety in CTX+. Moreover, patients did not reduce their conditioned anxiety and kept showing stronger defensive verbal as well as physiological responses during extinction. After exposition therapy, subjective anxiety was significantly reduced at the end of the extinction phase but not patients’ physiological responses. In summary, extinction of the conditioned anxiety was ameliorated in patients after exposure therapy, but only on the verbal and not on the physiological level. Both the stronger defensive responses in a safe context and the dissociation between the two levels of responses might be implicated in the relapses of the symptoms.