Automatic Analysis of Voice Emotions in Think Aloud Usability Evaluation: A Case of Online Shopping

Emotions elicited from interacting with technologies are fundamental components of user experience (UX). Two general approaches to measuring people’s emotional experiences exist: memory-based and moment-based. Whereas memory-based approaches are susceptible to the peak-end effect, moment-based approaches appear to better reflect the actual experiences of emotions. In this thesis, it is proposed that assessment of emotions of think aloud verbalisations is a moment-based approach for measuring emotional experiences. To evaluate the effectiveness of this approach, two independent user studies were conducted, respectively with 46 and 35 participants, in the domain of online shopping. In both studies, three assessment methods were applied for measuring emotional experiences: Self-reports, Vocal expressions (manual analysis), and Vocal Expressions (automatic analysis). Study 1 adopted machine learning (ML) approaches to analyse participants’ vocal expressions elicited during thinking aloud based on the discrete and dimensional models of emotions. While vocal analysis of verbal data was used as the momentbased approach, self-report questionnaires served as a means for the memory-based assessment. Results suggested that retrospective evaluations of emotions were significantly correlated with the most frequently elicited emotion (modal emotion) during the interaction. Study 2 expanded the extent of analysis of vocal expressions by taking a different set of ML techniques. Emotions were modelled by dimensional descriptors through binary pleasure (negative/positive), binary arousal (low/high) and ternary dominance (low/neutral/high). Results showed 66%, 70%, 41% recognition accuracy for Pleasure, Arousal and Dominance dimensions respectively. Moreover, elicited facial expressions were analysed to derive classification models to predict subjective self-reports. The main contribution of this thesis is the proposition and validation of an approach for automatic assessment of emotional experiences evoked during the think aloud protocol. Future work will investigate methods for improving the accuracy of automatic analysis, integration of multiple modalities (verbal and nonverbal) and different ML techniques in an actual automatic assessment tool.