Narrative from the perspective of multimodality: what makes a (multimodal) story?
From the perspective, for example, of narratology, it is generally accepted that narratives can unfold in many media and using many expressive forms. This raises interesting questions concerning both the possible boundaries of what might be called `narrative' and the potential for new media to give rise to new kinds of narrative. One case of this is the uneasy truce that now holds between narrative and computer gaming, where for some time there was heated discussion concerning just to what extent a digital game could be considered to be narrative at all. Similar points of discussion are raised when moving across other media, such as when considering the narrative potential of static images, music, dance and so on. In this talk I address these issues from the perspective of multimodality, an emerging field that adopts an approach in many respects the reverse of previous proposals for dealing with complex communicative artefacts and performances. Rather than starting with individual `modalities' such as language, image, movement, music, etc. and considering how these might interact with others, the view of multimodality I present instead takes the phenomenon of multimodality itself as the central organising principle. This has several theoretical and methodological consequences that allow modally-complex communicative situations to be addressed in a new light. The talk will set out the fundamental concepts in this account, particularly those of semiotic mode, medium and genre, and position narrative multimodally within this drawing on examples from a broad range of media.
Biography of John Bateman will follow soon.
Storysourcing: Telling stories with humans & machines
Stories connect us with other people, guide us in exploring unfamiliar places, bring us to discover new things. They are all around us - in our daily lives, in museum exhibitions, in movies and songs. Everybody has them, but only few used to tell them. The Web has changed this. Now, we co-create and share stories everywhere - blurring the lines between physical and online worlds with our social media timelines and video streaming. It is a symbiotic cooperation between humans and machines. However, we still do not understand the implicit and creative aspects of narration. In this talk I connect serendipitous discovery, creative thinking and human computation in the context of narrative building. This is illustrated with examples from smart culture, such as DIVE+ (http://diveproject.beeldengeluid.nl/), where humanities scholars explore and discover stories with cultural heritage objects from media collections online. DIVE+ is the result of a interdisciplinary collaboration between computer scientists, humanities scholars, cultural heritage professionals currently integrated in the Dutch national CLARIAH (Common Lab Research Infrastructure for the Arts and Humanities) research infrastructure. In this talk, I also show how human computation can help acquire, capture and harness diversity in human interpretation to make such explorative journeys creative and serendipitous. The experience with the crowd leaves one thing absolutely clear - there is no single notion of truth, but rather a spectrum that has to account for context, opinions, perspectives and shades of grey. CrowdTruth (http://crowdtruth.org) is a widely used crowdsourcing methodology adopted by industrial partners and public organizations, e.g. Google, IBM, New York Times, Crowdynews, The Netherlands Institute for Sound and Vision, in a multitude of domains, e.g. AI, news, medicine, social media, cultural heritage, social sciences. The central characteristic of CrowdTruth is harnessing the diversity in human interpretation to capture the wide range of opinions and perspectives, and thus, provide more reliable and realistic real-world annotated data for training and evaluating machine learning components. Unlike other methods, we do not discard dissenting votes, but incorporate them into a richer and more continuous representation of truth. Creating this more complex notion of truth contributes directly to the larger discussion on how to how to distinguish facts from opinions, perspectives and ultimately to make the Web more reliable, diverse and inclusive.
Lora Aroyo is a Full Professor in Human-Computer Interaction. She is heading the User-centric Data Science group (http://ucds.cs.vu.nl/) at the Department of Computer Science, Vrije Universiteit Amsterdam, The Netherlands. Within the framework of the Amsterdam Data Science (http://amsterdamdatascience.nl) and the Network Institute (http://www.networkinstitute.org), she is involved in a number of research projects and organized various workshops and tutorials focussed on crowdsourcing and human computation, collecting data, data quality assessment, and especially hybrid human-AI systems for understand text, image and video. She has led major research projects in semantic search, recommendation systems, event-driven access to online multimedia collections, and through these has become a recognized leader in human computation techniques for specific domains, such as digital humanities, cultural heritage, and interactive TV. She is a three time holder of IBM Faculty Award for her work on CrowdTruth (http://crowdtruth.org/): Crowdsourcing for ground truth data collection for adapting IBM Watson system to medical domain.
Twitter: @laroyo and slideshare: http://www.slideshare.net/laroyo