NWO has granted CTIT research Marielle Stoelinga of the FMT group, a TOP research Grant for her proposal on Better Testing with Game Theory (BEAT project). TOP grants are awarded for innovative, ground breaking and risky research not limited to a specific topic.
Software testing, more urgent than ever
Due to the shift from embedded to cyber-physical systems, from single to multi-core, and to run-time adaptation, system complexity is ever increasing. Therefore, effective and efficient test methods are more urgent than ever.
Testing is an important means to assess and improve the quality of ICT systems --- which are vital in today's society. At the same time, testing is costly and time consuming, often taking 50% of all project resources. Therefore, improving test effectiveness and efficiency yields higher system quality and a shorter time-to-market.
Testing as a two-player game
Testing is naturally phrased as a game, where the tester tries to find faults, playing against the system-under-test (SUT). Also, testing is by nature an optimization problem, asking for a test suite with maximal impact within a given budget. We take a game-based approach, where test cases coincide with game strategies, allowing the rich methods of algorithmic game theory and strategy synthesis to be deployed for test optimization. This will lead to a rich, model-based framework that allows one to automatically derive an optimal test suite for a wide range of optimality criteria, including cost, impact, coverage, and risk. Thereby, it will unify the enormous variety of model-based testing theories.
In the BEAT project, we will develop an effective and efficient framework for model-driven test case optimization. This framework will unify the various test theories that exist, for functional, symbolic, timed, hybrid, etc), and will be implemented within the MBT tool JTorX.
Finally, quantitative assessment of our framework is important, and we will measure and improve the actual gains in effectiveness and efficiency of our test optimization methods, by using realistic benchmarks provided by industry.
Picture by Gijs van Ouwerkerk