Human Factors

https://upload.wikimedia.org/wikipedia/commons/e/e6/Flickr_-_Official_U.S._Navy_Imagery_-_Doctors_perform_surgery_together..jpg

  • BHF1 – Artificial Intelligence Conversational Agents: A Measure Of Satisfaction In Use (4 students)

    SUPERVISOR: DR. SIMONE BORSCI, SECOND SUPERVISOR: JULE LANDWEHR, MSC


    Background

    Conversational agents, such as chatbots and voice interfaces, can be used for multiple purposes e.g., support customer experience with services etc. These new tools are growing and more and more integrated into systems such as websites, social networks, cars. Smart and AI-based conversational agents are shaping the future of human-computer interaction however little is known about how to assess people reaction and satisfaction after the use of these systems.

    Goals

    You will build upon previous work done on a new scale to assess satisfaction with chatbots. Your experimental work will focus on the evaluation of conversational agents to further streamline the reliability and validity of the scale. You will perform remote usability testing on 5 different chatbots (of your choice) by using Qualtrics (a template will be provided) to collect data.

    The current validate version of the scale is composed by 11 items and it is available in Dutch, Spanish, German and English, and Italian (the last one is still under validation).

    The target is to involve a large number of participants of different ages working (potentially) in a team and perform a factorial analysis and descriptive and inferential statistics considering, for instance, elements that can affect the satisfaction during the interaction with chatbots such as:

    -        Effect of age on satisfaction
    -        Workload
    -        Task achievement

    You can also potentially contribute to the ongoing development of the scale by adding new languages. Moreover, you can propose to add research questions of your interest to the current evaluation protocol.

    Requirements

    You should be aware of the statistical methods regarding factorial analysis and for inferential statistics and be able to use R for such purposes.

    Previous work

    Borsci, S., Malizia, A., Schmettow, M., Van Der Velde, F., Tariverdiyeva, G., Balaji, D., & Chamberlain, A. (2021). The Chatbot Usability Scale: the Design and Pilot of a Usability Scale for Interaction with AI-Based Conversational Agents. Personal and Ubiquitous Computing, 1-25.

    Kerwien Lopez, S. M. (2021). Confirmatory Factor Analysis of a new Satisfaction Scale for conversational agents and the role of decision-making styles (Bachelor's thesis, University of Twente).

    Bos, M. A. (2021). Testing a scale for perceived usability and user satisfaction in chatbots: Testing the BotScale (Master's thesis, University of Twente).

  • BHF2 – Artificial Intelligence Conversational Agents: Using Card Sorting To Evaluate The Chatbot Usability Scale (2 students)

    SUPERVISOR: JULE LANDWEHR MSC, SECOND SUPERVISOR: DR. SIMONE BORSCI


    Background

    Conversational agents, such as chatbots and voice interfaces, can be used for multiple purposes e.g., support customer experience with services. These new tools are growing and more and more integrated into systems such as websites, social networks, cars. Smart and AI-based conversational agents are shaping the future of human-computer interaction however little is known about how to assess people reaction and satisfaction after the use of these systems.

    Goals

    You will build upon previous work done on a new scale to assess satisfaction with chatbots. Your experimental work will focus on the evaluation of the satisfaction scale using card sorting to further streamline the validity of the scale. You will perform a card sorting using the items of the scale to collect data.

    The current validated version of the scale is composed by 11 items, and it is available in Dutch, Spanish, German and English, and Italian (the last one is still under validation).

    The target is to involve a large number of participants working (potentially) in a team to evaluate the face and construction validity of the satisfication scale and possibly get some new insights into additional factors and items.

    Requirements

    You should be aware of the statistical methods regarding factorial analysis and for inferential statistics, and be able to use R for such purposes.

    Previous work

    Borsci, S., Malizia, A., Schmettow, M., Van Der Velde, F., Tariverdiyeva, G., Balaji, D., & Chamberlain, A. (2021). The Chatbot Usability Scale: the Design and Pilot of a Usability Scale for Interaction with AI-Based Conversational Agents. Personal and Ubiquitous Computing, 1-25.

    Bos, M. A. (2021). Testing a scale for perceived usability and user satisfaction in chatbots: Testing the BotScale (Master's thesis, University of Twente).

    Kerwien Lopez, S. M. (2021). Confirmatory Factor Analysis of a new Satisfaction Scale for conversational agents and the role of decision-making styles (Bachelor's thesis, University of Twente).

    Beerlage-de Jong, N., Kip, H., & Kelders, S. M. (2020). Evaluation of the perceived persuasiveness questionnaire: User-centered card-sort study. Journal of Medical Internet Research, 22(10). https://doi.org/10.2196/20404

    Agarwal, N. K. (2011). Verifying survey items for construct validity: A two-stage sorting procedure for questionnaire design in Information Behavior Research. Proceedings of the American Society for Information Science and Technology, 48(1), 1–8. https://doi.org/10.1002/meet.2011.14504801166

  • BHF3 – Website Navigation Structures : Eliciting Mental Models Using Card Sorting (2 students)

    SUPERVISOR: MARLISE WESTERHOF, MSC

    Mental models are abstract, inner representations that people have in their mind regarding things from the external world, such as objects or systems. In other words: a mental model is an explanation of how an individual thinks something works. Card sorting is a method to elicit mental models of specific domains, and is frequently applied for organizing the structure of informational websites in a way that it suits the user's expectations. If the navigation structure of a website matches the mental model of users, it helps users to find what they are looking for.

    In this bachelor thesis assignment, you will apply the method card sorting to elicit user's mental model of a specific domain (e.g. websites of GGD/Health Services, webshops, municipalities or universities). Based on the obtained insights, you will formulate design recommendations for the chosen domain.