Capturing the Complexity of Laughter | Acquisition, Annotation and Analysis of Laughter Data in Social Signal Processing
Michel Jansen is a PhD student in the department Human Media Interaction. (Co)Promotors are prof.dr. D.K.J. Heylen and dr. K.P Truong from the faculty of Electrical Engineering, Mathematics and Computer Science, University of Twente.
This thesis investigates laughter as a complex and richly layered social signal, emphasizing its important role in human social interactions within the multidisciplinary domain of Social Signal Processing (SSP). Structured around three thematic sections, the thesis first addresses the necessity for robust, multimodal laughter data collection and annotation methods. We provide an overview of existing laughter databases and pinpoint key challenges researchers face due to the absence of standardized procedures for laughter segmentation and labelling. To address these gaps, a first version of a laughter segmentation and label framework is proposed, systematically defining laughter units and sub-units to facilitate more accurate annotation and analysis.
Within the second section the human annotation process of laughter roles and functions is further explored. We further explore how contextual information and modality affect annotators' perceptions and judgments. This thesis identifies significant variability and generally a low consensus among annotators when categorizing the social roles of laughter using several annotation schemes inspired by prior literature. This variability challenges some of the current assumptions underpinning laughter annotation research. A follow-up study extends this exploration by allowing lay annotators to categorize laughter without much structured guidance. Findings from this study illustrate a rich diversity in lay perceptions, yet notable overlap is observed with established annotation schemes, suggesting that non-experts intuitively categorize laughter in similar ways as researchers have done in prior research.
In the third thematic area, the thesis investigates specific contextual influences on laughter production, notably familiarity and perceived humour. For this purpose, two multimodal laughter databases were specifically recorded and annotated by the author. The first database focusses on familiarity, exploring differences in laughter production between interlocutors who are familiar or unfamiliar with each other. This database is presented along with an analysis of acoustic and temporal features of the laugh instances in conversations between familiar and unfamiliar interlocutors. The second database, investigates laughter in relation to perceived humour, examining how acoustic and temporal characteristics of laughter vary according to humorous contexts. The acoustic analysis described in these chapters reveals distinct differences in laughter properties such as duration, frequency, synchronization, and vocal characteristics. These databases serve as resources for future laughter-related research in SSP, offering structured data collections and insights on acoustic and temporal laugh features that correlate with the mentioned social constructs.
Across these studies, a consistent thread emerges: laughter, despite being intuitively understood at a basic level, poses complex challenges in scientific segmentation, annotation, and analysis due to its inherently social and contextual nature. The field should focus on improving standards in laughter segmentation and labelling, clearer annotation guidelines, and improved understanding of annotator laugh perception and its possible biases. To address these issues, a structured collaboration might help to address some of these challenges. In addition, it might be valuable to explore different segmentation and annotation approaches such as iterative annotation or self-annotation to improve reliability and validity.
Finally, enhancing our understanding and ability to accurately interpret and model laughter signals can have substantial implications. Beyond theoretical contributions to SSP and related fields such as psychology and sociology, advances in laughter research could significantly inform practical applications. Potential areas include more empathetic social robots in healthcare, advanced interactive digital training tools, and emotionally intelligent conversational agents capable of interpreting and responding to nuanced social cues like laughter. By addressing foundational challenges in laughter data collection, segmentation and annotation, and analysis, this thesis significantly contributes to the broader pursuit of socially intelligent technologies capable of genuinely understanding and engaging with human social behaviours.
