CTIT University of Twente
Research Business & Innovation About CTIT Research Calls Looking for a job? Intranet


Sustained Emotionally coloured Machine-human Interaction using Nonverbal Expression

Project Manager: Dr. Dirk K.J Heylen

Faculty of Electrical Engineering, Mathematics and Computer Science (EEMCS)

Tel.: +31-53-4893745

Email: d.k.j.heylen@utwente.nl

Project website:


The aim of the SEMAINE project is to build a Sensitive Artificial Listener – a multimodal

dialogue system with the social interaction skills needed for a sustained conversation with a

human user. The system will emphasise “soft” communication skills, i.e. non-verbal, social

and emotional perception, interaction and behaviour capabilities. The Sensitive Artificial

Listener paradigm involves only very limited verbal capabilities, but has been shown to be

suited for prolonged human-machine interaction. In this paradigm, we will build a real-time,

robust interactive system perceiving a human user's facial expression, gaze, and voice, and

engaging with the user through an Embodied Conversational Agent's body, face and voice.

The agent will exhibit audiovisual listener feedback in real time while the user is speaking,

and will take the user's feedback into account while the agent is speaking. The agent will

pursue different dialogue strategies depending on the user's state; it will learn to interpret the user's non-verbal behaviour and adapt its own behaviour accordingly.

Data to train system components will be collected initially using a Wizard-of-Oz setup, later

on using the autonomous system at increasing levels of maturity. Some of the data will be

released to the research community. Consortium partners are involved in several standardisation initiatives related to building socially competent interactive agents; these standards will be co-shaped by work in SEMAINE.

Project duration: 1 January 2008 – 31 December 2010

Project budget: 3.6 M-€ / 2.75 M-€ funding

Number of person/years: 30 fte

Project Coordinator: DFKI

Participants: DFKI, Queens University Belfast, Imperial College London, UT, University of Paris8, TU München

Project budget CTIT: 715 k-€ / 563 k-€ funding

Number of person/years CTIT: 5.9 fte

Involved groups: Human Media Interaction (HMI)