Software Testing

Conversational AI systems, such as chatbots and virtual assistants, are becoming increasingly common across various applications, from customer support over education to healthcare. Ensuring their reliability is essential, as failures can lead to user frustration or misinformation. 

Model-Based Testing (MBT) is a well-estbalished technique that offers a structured approach to testing of (black-box) software systems by using formal models to define expected behaviors and generate test cases.

The aim of this project is to explore how MBT can be applied to conversational AI to ensure consistency and accuracy in interactions, focusing on the unique challenges posed by natural language processing.

Some example research questions are the following:

Dialogue Modeling: How can we create formal models that capture the flow of conversation, handling context changes, user intent, and response generation? In particular, how can the model represent complex dialogue structures found in real-world conversational agents?

Test Case Generation: What strategies can be used to generate meaningful test cases that simulate realistic conversations? How can we ensure that generated test cases cover key dialogue scenarios, including edge cases where the agent may fail to respond appropriately? (Natural language generation)

Response Evaluation: How can the behavior of a conversational agent be validated against the formal model? In particular, what metrics or techniques can be used to assess if the agent’s responses align with the expected outcomes defined in the model? (Natural language processing)

The research will consist of the following steps:

1. Conducting a literature review to explore existing model-based (testing) techniques that can be applied to conversational AI.
2. Developing your own approach or methods based on the chosen research question.
3. Formulating an answer to the research question by applying the developed methods to a conversational AI system.

Context

Technolution, an electronics/digital logic/software technology integrator, regularly creates
embedded systems for customers. Technolution tests these systems to verify conformance to the specification. Currently, these tests are automated, but hand-crafted. Technolution wants to
investigate feasibility of further automating test of embedded systems using model-based testing, with the aim of reducing cost and mistakes. The novelty lies in the application of model-based testing to embedded systems. As embedded systems play a critical role in society, further improving testing of embedded systems is highly relevant.

As a first step, we want to apply model-based testing to a sensor for measuring voltages and
currents in the electrical grid. Part of the behaviour on the sensor interfaces is expected to be
representable in typical and existing MBT concepts; part of it is atypical for MBT, for example the (digitized) electrical input signals, and arithmetic operations on the digitized signals like offsets, multiplications, and filtering. Important challenges of this assignment are finding a way to model these atypical aspects and inventing a method to generate tests for them, where the number of combinations is potentially infinite.

Goal

The research aims to answer the following questions:

1. Is it feasible to model the embedded systems’ behaviour so that it can be automatically tested using the model as input? How does this model look, what concepts are needed to describe the behaviour?
2. What algorithm can be invented, or existing method be used, to automatically test the system using the model as input? What coverage can be achieved using this algorithm or method?
3. Can the algorithm or method be optimized for a specific goal? For example for duration (typical use case would be development testing), or for coverage (typical use case would be release testing).
4. How does the model-based testing method evaluate against hand-crafted tests?

Activities

• Study the interfaces of the sensor and identify the independent parts;
• Postulate initial ideas for modelling the interface parts;
• Study, understand, summarize (document) and present the basic concepts of model-based testing;
• Determine what interface parts can be modelled using an existing approach, and what parts need a new approach;
• For each interface part, define the modelling and testing strategy and develop a prototype (consisting of a model, an automated testing tool, adapters to the system and a script that runs the model-based test) that demonstrates the test strategy on the actual sensor;
• Evaluate the model-based testing strategies on effectiveness compared to the handcrafted tests;
• Provide recommendations for further developing the model-based testing approach and generalizing it to other embedded systems.

For prototyping and demonstrating, an actual sensor with software adapters to its interfaces will
be made available.

Company info

Technolution B.V.
Burg. Jamessingel 1
P.O. Box 2013
2800 BD Gouda
The Netherlands

T +31 (0)182 59 40 00
E info@technolution.com
I www.technolution.com

Traditionally, requirements for software systems are translated by hand into a set of test cases. This is a lot of work, and prone to errors in manual translation. With model-based testing, test cases can be derived automatically from a model that captures all requirements. However, obtaining a model suited for model-based testing from the set of natural language requirements is still a challenging task, because it requires interpretation of natural language and modelling skills to construct such a testing model from requirements. In this assignment you will investigate how well Large Language Models can close the gap, i.e. translate between requirements and models for testing. You will work on a case study with requirements provided by Sioux (Eindhoven).