The R3D3 project aims at investigating situated natural language dialogue systems that combine limited natural language understanding with the understanding of the nonverbal behaviour of the users in a real-life context. Inspired by Star Wars’ R2D2, we propose to develop an even more advanced robot, our R3D3. R3D3 takes the form of a Receptionist Robot that can have an intelligent dialogue with the visitors to a museum, shop or office building. These dialogues are ‘double’ in the sense that they involve a third party in the form of an avatar or a person shown on a screen by the robot. The R3D3 project addresses scientific ICT challenges in the areas of computer vision, spoken language interaction, and data processing, as well as practical challenges in robot development.