In the coBOTnity project we have developed technology to build interactive playful environments for creative storytelling. For further details, please contact Alejandro Catala. We briefly make an overview of the main components next.
We have developed the concept and implementation of surfacebot. It is intended as a movable character that is embodied in a tablet display attached to a robotic platform. It is capable of rendering facial expressions, speaking short utterances, moving, and displaying visual assets.
A variable number of surfacebots can be dynamically added in a play activity, where all devices are technically connected through a ROS overlay network via WiFi. The system supports the connection of control tablets dynamically as well, which can be used to either enact the story behaviour or create content.
Beside controlling, there a number of editors to create user assets that can be included in activities. These support the inclusion of Text-to-Speech, recorded audio, visual assessts based on images from the internet or drawn on the tablet itself. The sequencing of assets as compound behaviors is possible in the backend.
The system prototype is expanded with smart navigation features to prevent children spending cognitive resources in driving bots. Children can give single destinations or paths for the bots to follow. When giving paths, users can include modifiers at any point/landmarks in the planned path. These modifiers execute behaviours when the robots reach the landmarks. For example, existing facial or voice behaviours of the character as well as movement behaviours such as setting rotations, stop-and-go, and dancing can be scheduled as modifiers. Besides the general collision avoidance based on RVO2 library, some social behaviours such as following specific characters, running away from another bot and keeping a distance, or facing at the audience after reaching destinations are implemented features.
Additionally, there is a fatigue model implemented for the surfacebot that controls the gain and depletion of energy. If children tend to do continuous driving rather than storytelling, this feature forces them to play with story elements as the surfacebot will have to stop to recover energy.
The figure depicts how the architecture has been realized into different implemented components.
Special credits: Roberto Campisi, Luce Sandfort, Wouter Timmermans, Wouter van Veelen (CS bachelor students)
Enhanced storytelling setting
We have explored how to mediate interaction in the storytelling setting via a tablet app. The features in the enhanced setting support children in developing storytelling skills, helping them to structure the creation of stories and reflect on the links between the different events. The app facilitates the transition from free play towards more reflection in the child's storytelling process.
Special credit: Silke ter Stal (HMI master student)
Social-emotional development support
Enhancing children’s social awareness and their responses on social issues is also important. Through stories, they can safely experience social situations. The platform has been used in experiments to explore how social-emotional development can be affected by the format of the storytelling table and the level of assistance/responsiveness of the system. Storytelling tables enhanced by technology can be useful to reinforce emotional understanding and story event recall.
Special credits: Iris Visser, Hannie Gijlers
Creative dancing Tablets
Following user-centered methods, we have designed, implemented and evaluated a dancing robotic tablet prototype for co-creative human-robot interaction. Two types of autonomous robot behavior were considered as creativity support and evaluated in a user study. While imitation behavior was perceived as more intelligent; the generation behavior that attempted to challenge users and be different to the users’ input led to a greater variety of gestures. Observations support the idea that users were inspired to some extent by the robot’s input.
Special credits: Federico Fabiano, Hannah Pelikan, Jelle Pingen, Judith Zissoldt (HMI master students)
Enabling Emotion-aware Surfacebots
The middleware can be quikly integrated with other components used in robotic or HRI systems. For example, we have enabled the agents to be emotion-aware to implement further applications in which the characters can react emotionally based on the arousal of children. This uses the emotion recognizer based on speech developed by Jaebok Kim in the EU-funded project SQUIRREL.
Find out more about Jaebok's work here.
Some of the software code is available at gitlab.
Core coBOTnity Apps:
Surfacebot Smart Navigation middleware and apps:
Further helper software:
Alejandro Catala, Mariët Theune, and Dirk Heylen. 2017. My Tablet Is Moving Around, Can I Touch It?. In Proceedings of the Eleventh International Conference on Tangible, Embedded, and Embodied Interaction (TEI '17). ACM, New York, NY, USA, 495-502. DOI: https://doi.org/10.1145/3024969.3025082
Alejandro Catala, Mariët Theune, Hannie Gijlers, and Dirk Heylen. 2017. Storytelling as a Creative Activity in the Classroom. In Proceedings of the 2017 ACM SIGCHI Conference on Creativity and Cognition (C&C '17). ACM, New York, NY, USA, 237-242. DOI: https://doi.org/10.1145/3059454.3078857
Alejandro Catala, Mariët Theune, Dennis Reidsma, Silke ter Stal, Dirk Heylen. Exploring Children’s Use of a Remotely Controlled Surfacebot Character for Storytelling. In: 9th International Conference on Intelligent Technologies for Interactive Entertainment, INTETAIN 2017, 10 pages (2017)
Alejandro Catala, Mariët Theune, Cristina Sylla, Pedro Ribeiro. 2017. Bringing Together Interactive Digital Storytelling with Tangible Interaction: Challenges and Opportunities. In: Nunes N., Oakley I., Nisi V. (eds) Interactive Storytelling. ICIDS 2017. Lecture Notes in Computer Science, vol 10690. Springer, Cham DOI: https://doi.org/10.1007/978-3-319-71027-3_51
Federico Fabiano, Hannah R. M. Pelikan, Jelle Pingen, Judith Zissoldt, Alejandro Catala and Mariët Theune. Designing a Co-Creative Dancing Robotic Tablet. 6th International Workshop on Computational Creativity, Concept Invention, and General Intelligence. Madrid, December 15th 2017.