socially intelligent robots that understand and respond to human touch
Touch is an important nonverbal form of interpersonal interaction which is used to communicate emotions and other social messages. As interactions with social robots are likely to become more common in the near future these robots should also be able to engage in tactile interaction with humans. Therefore, the aim of the research presented in this dissertation is to work towards socially intelligent robots that can understand and respond to human touch. To become a socially intelligent actor a robot must be able to sense, classify and interpret human touch and respond to this in an appropriate manner. To this end we present work that addresses different parts of this interaction cycle.
After the introduction in Part1 of the dissertation, we have taken a data-driven approach in Part2. We have focused on the sense and classify steps of the interaction cycle to automatically recognize social touch gestures such as pat, stroke and tickle from pressure sensor data.
In Chapter 2 we present CoST: Corpus of Social Touch, a dataset containing 7805 captures of 14 different social touch gestures. All touch gestures were performed in three variants: gentle, normal and rough on a pressure sensitive mannequin arm. Recognition of these 14 gesture classes using various classifiers yielded accuracies of up to 60%; moreover, gentle gestures proved to be harder to classify than normal and rough gestures. We further investigated how different classifiers, interpersonal differences, gesture confusions and gesture variants affected the recognition accuracy.
In Chapter 3 we describe the outcome of a machine learning challenge on touch gesture recognition. This challenge was extended to the research community working on multimodal interaction with the goal of sparking interest in the touch modality and to promote exploration of the use of data processing techniques from other more mature modalities for touch recognition. Two datasets were made available containing labeled pressure sensor data of social touch gestures: the CoST dataset presented in Chapter 2 and the ‘Human-Animal Affective Robot Touch’ (HAART) gesture set. The most important outcomes of the challenges were: (1) transferring techniques from other modalities, such as image processing, speech, and human action recognition provided valuable feature sets; (2) gesture classification confusions were similar despite the various data processing methods that were used.
In Part3 of the dissertation we present three studies on the use of social touch in interaction with robot pets. We have mainly focused on the interpret and respond steps of the interaction cycle to identify which touch gestures a robot pet should understand, how touch can be interpreted within a social context and in which ways a robot can respond to human touch.
In Chapter 4 we present a study of which the aim was to gain more insight into the factors that are relevant to interpret the meaning of touch within a social context. We elicited touch behaviors by letting participants interact with a robot pet companion in different affective scenarios. In a contextualized lab setting, participants acted as if they were coming home in different emotional states (i.e., stressed, depressed, relaxed and excited) without being given specific instructions on the kinds of behaviors that they should display. Based on video footage of the interactions and interviews we explored the use of touch behaviors, the expressed social messages and the expected robot pet responses. Results show that emotional state influenced the social messages that were communicated to the robot pet as well as the expected responses. Furthermore, it was found that multimodal cues were used to communicate with the robot pet, that is, participants often talked to the robot pet while touching it and making eye contact. Additionally, the findings of this study indicate that the categorization of touch behaviors into discrete touch gesture categories based on dictionary definitions is not a suitable approach to capture the complex nature of touch behaviors in less controlled settings.
In Chapter 5 we describe a study in which we evaluated the expressive potential of breathing behaviors for 1-DOF zoomorphic robots. We investigated the extent to which researcher-designed emotional breathing behaviors could communicate four different affective states. Additionally, we were interested in the influence of robot form on the interpretation of these breathing behaviors. For this reason two distinct robot forms were compared: a rigid wood-based form resembling a rib cage called ‘RibBit’ and a flexible, plastic-based form resembling a ball of fur called ‘FlexiBit’. In the study, participants rated for each robot how well the different breathing behaviors reflected each of four affective states: stressed, depressed, relaxed and excited. The results show that both robot forms were able to express high and low arousal states through breathing behavior, whereas valence could not be expressed reliably. Low arousal states could be communicated by low frequency breathing behavior and higher frequency breathing conveyed high arousal. In contrast, context might play a more important role in the interpretation of different levels of valence. Unexpectedly, robot form did not influence the perception of the behavior that was expressed. These findings can help to inform future design of affective behavior for robot pet companions.
In Chapter 6 we present a study in which we explored in what ways people with dementia could benefit from interaction with a robot pet companion with more advanced touch recognition capabilities and which touch gestures would be important in their interaction with such a robot. In addition, we explored which other target groups might benefit from robot pets with more advanced interaction capabilities. We administered a questionnaire and conducted interviews with two groups of health care providers who all worked in a geriatric psychiatry department. One group had experience with robotic seal Paro while the other group had no experience with the use of robot pets. The results show that health care providers perceived Paro as an effective intervention to improve the well-being of people with dementia. Furthermore, the care providers indicated that people with dementia (would) use mostly positive forms of touch and speech to interact with Paro. Paro's auditory responses were criticized because they can overstimulate the patients. Additionally, the care providers argued that social interactions with Paro are currently limited and therefore the robot does not meet the needs of a broader audience such as healthy elderly people that still live in their own homes. The development of robot pets with more advanced social capabilities such as touch and speech recognition might result in more intelligent interactions which could help to better adapt to the needs of people with dementia and could make interactions more interesting for a broader audience. Moreover, the robot's response modalities and its appearance should match the needs of to the target group.
To conclude, the contributions of this dissertation are the following. We have made a touch gesture dataset available to the research community and have presented benchmark results. Furthermore, we have sparked interest into the new field of social touch recognition by organizing a machine learning challenge and have pinpointed directions for further research. Also, we have exposed potential difficulties for the recognition of social touch in more naturalistic settings. Moreover, the findings presented in this dissertation can help to inform the design of a behavioral model for robot pet companions that can understand and respond to human touch. Additionally, we have focused on the requirements for tactile interaction with robot pets for health care applications.
