Abstract

Physiological sensing often complements studies of human behavior in virtual reality (VR) to detect users’ affective and cognitive states. Some psychological states, such as fear and frustration, can be particularly hard to differentiate from a physiological perspective as they are close in the arousal and valence emotional space. Moreover, it is largely unclear how users’ physiological reactions are expressed in response to transient psychological states such as fear, frustration, and insight—especially since these are rich indicators for characterizing users’ responses to dynamic systems but are hard to capture in highly interactive settings. We conducted a study (N=24) to analyze participants’ pulmonary, electrodermal, cardiac, and pupillary responses to moments of fear, frustration, and insight in immersive settings. Participants interacted in five VR environments, throughout which we measured their physiological reactions and analyzed the patterns. We also measured subjective fear and frustration using questionnaires. We found differences between fear and frustration pupillary, respiratory, and electrodermal responses, as well as between the pupillary changes that followed fear in a horror game and those that followed fear in a vertigo experiment. We present the relationships between fear levels, frustration levels, and their physiological responses. To detect these affective events and states, we introduce user-independent binary classification models that achieved an average micro F1 score of 71% for detecting fear in a horror game, 75% for fear of vertigo, 76% for frustration, and 75% for insight, showing the promise for detecting these states from passive and objective signals.