24/7 support and assistance from friends of Siri and Alexa

Imagine that as you get older, you have a team of experts around you who continuously and expertly look after your health and well-being. With the Council of Coaches programme, UT scientists wants to make this a reality. Minor detail: these coaches are not people, but virtual agents.

This project is ground-breaking because of its ‘multifaceted’ approach, says UT scientist Dirk Heylen. ‘There are already all kinds of systems aimed at stimulating behavioural change in users’, he explains.  ‘But until now, they are all single-agent systems, they only have a single voice. Think, for example, of pedometers, or systems like Siri and Alexa. What makes Council of Coaches innovative is that it brings together a very diverse team of agents. Each one has its own area of expertise and is concerned with a different aspect of your health and well-being. For example, one may be a dietician, the other a diabetes expert, another a peer. With the help of sensors and other instruments, they measure and monitor your behaviour, collect data, analyse it all, and use their findings to talk to you and give advice. Each agent can talk to the user independently, but they can also coordinate with each other, for example, to determine what the priority should be. For users, the great advantage is that they will have a team of experts on hand 24/7. No healthcare system can compete with that.’ 

The Council of Coaches programme is subsidized by Horizon 2020, Europe’s leading funding programme for research and innovation. Council of Coaches is carried out by a consortium of seven European partners, with the University of Twente and Roessingh Research & Development as lead partners.

Different ways of convincing

Another of Council of Coaches’ innovative features is that the system can converse with users in a variety of ways. ‘The goal is behavioural change or improvement, and healthcare providers can use a range of persuasive methods to help a patient to realize change. While one user may respond better to rational arguments and facts, or clear, direct instructions, such as ‘Do this’ or ‘Do that’, another other user may be more susceptible to emotional persuasion: ‘You will feel better if…’ And yet another may perform better when offered rewards: ‘If you do this now, then this evening you can…’ Our goal with Council of Coaches is not just to make sure the coaches know and can apply all these different strategies, but that they will learn from the individual user what works best for him or her, and when. This way the user will really benefit, while also remaining motivated to keep using the system.’

‘The Council of Coaches system is an example of scientific research designed to help a specific group of people reach a specific goal. In this case, the objective is to help elderly people maintain their health and wellbeing. The project requires in-depth research, while the intended outcome is very practical. I like this purpose-driven approach. At the same time, we are moving in a highly innovative direction: facilitating multi-party conversations between virtual agents. The technical side is challenging, but so is the multidisciplinary aspect, as we get to collaborate not just with engineers, but also with linguists, psychologists and ethicists. That is what makes this work fascinating’.

Technical challenges

Several features make the Council of Coaches technically highly advanced. Heylen: ‘The aspect of sensing, or monitoring and measuring behaviour, is a challenge in itself, on which many people on our team are working. The data we collect must be very accurate, or the system cannot achieve its goal. Many sensing technologies offer a lot of room for improvement in this respect. Combining data from various measurements is also a challenge: we work with different sets of data that, taken on their own, are not always fully reliable, and can even contradict each other – yet we do require them to deliver a coherent message. Another question is: how do you input all the information in order for the system to work? All that content – for all the different agents in COUCH – has to be written and entered. That in itself is a huge job.’ Heylen notes that more and more companies are appearing on the scene to take on this painstaking work. ‘A good thing, too, because it requires an immense amount of capacity.’

Psychological aspects

The UT’s typical combination of technological challenges and social aspects in its research is quite evident in the Council of Coaches project. Heylen: ‘In a society that is becoming increasingly technological, it is important to know how digital solutions are received. We’re giving a lot of attention to this aspect, and relying heavily on the insights of psychologists. Think, for example, of the question as to how we can ensure that this kind of system appeals to the target group, and manages to hold their attention for a longer period: typically a psychological challenge.’

Providing digital tools and infrastructure

The ultimate goal of the Council of Coaches research project is not to put a ready-to-use system on the market, but to lay the groundwork for it. Heylen: ‘Our challenge is to figure out how a group of virtual experts can work together, and to then provide the digital tools and infrastructure for it. Ideally, we will succeed in creating the underlying software, which other parties can then use to develop a working system. There is no other system like this in the world. That is what makes it scientifically interesting: can it be done, and if so, how? Together with the consortium that is working on it, the University of Twente is in a good position to succeed, as we’ve been carrying out pioneering research in the field of dialogue systems and virtual agents for twenty years now. Within Council of Coaches, we have already tested various prototypes and systems, and so far, everything indicates that the system we envisage is possible.’

Prof.dr. Dirk Heylen
prof.dr. D.K.J. Heylen (Dirk)
Professor of Socially Intelligent Computing at the University of Twente | Scientific Director of the 3TU Centre on Humans and Technology | Secretary of the Association for the Advancement of Affective Computing | Programme Director Human Computer Interaction and Design | Co-director of the Centre for Monitoring and Coaching

Studied Linguistics, Computer Science and Computational Linguistics at the University of Antwerp, Belgium

Fields of interest: embodied conversational agents, social signal processing, ambient intelligence, affective computing spoken dialogue systems, human computer interaction, back channels, brain-computer interfaces, interactive playgrounds, virtual and augmented reality