Digital game-based learning in crisis management decision-making: Unobtrusive assessment measures and intervention evaluation
Due to the COVID-19 crisis the PhD defence of Johannes Steinrücke will take place (partly) online.
The PhD defence can be followed by a live stream.
Johannes Steinrücke is a PhD student in the research group Instructional Technology (IT). His supervisors are prof.dr. A.J.M. de Jong and prof.dr.ir. B.P. Veldkamp from the Faculty of Behavioural, Management and Social Sciences (BMS).
In digital game-based learning (DGBL), adaptivity allows for a more efficient and effective learning experience. That means, that the game can change dependent on, for example, the current performance of the players regarding a certain skill. For example, if a player performs badly, the game can adapt to become easier, and therefore more appropriate for the player’s skill level. However, to be truly adaptive, the game also needs to know the current performance level of a player. To achieve this, we need to assess the players regarding the skill we want to measure and train. One way to do this, would be to interrupt the game and to administer some kind of test, like a multiple-choice test. Unfortunately, this rather obtrusive method of assessment would break the flow of the gameplay and would keep the players aware of the fact that they are being assessed. Alternatively, we can try to assess the players without them noticing it. In doing so, players would remain in their flow state and could fully focus on playing the game. This is called unobtrusive assessment.
By unobtrusively assessing the players’ current skill levels, games can adapt to the players needs and offer instructional interventions. Which intervention is offered, depends on the context of the game, on the target skill and on the target population. While it could be argued that flow is interrupted even with an unobtrusive assessment, that would be done without the player noticing it and only to trigger a change in the players’ in-game behavior.
The goal of this dissertation has been to investigate the potential use of unobtrusive assessment measures in a DGBL environment, as well as the potentially beneficial effect of one specific instructional intervention, namely self-reflection through social comparison. Self-reflection not only fitted the DGBL environment that we used, but it also fits the natural way of crisis management decision-making training of Veiligheidsregio Twente (VRT; Twente Safety Region), whose members are used to critically reflect on their own approaches in real- life training sessions.
Overview of conducted studies
In the context of this dissertation, we conducted three studies. While the first study was conducted with university students from University of Twente, the other two studies were conducted with professionals in crisis management at VRT. We focused on three important competencies in the context of crisis management decision-making: Analytical skills, stress resistance, and information literacy (IL).
In the different studies, we used various evolutionary stages of the Mayor Game, later on referred to as the Dilemma Game. In the first study, the original Mayor Game was slightly modified to fit the goal of the study. In the second study, the Mayor Game lost its original name, and was revised and further developed to become the Dilemma Game. These two versions ran on the flash-based SDM and DDM game engines from T-Xchange by Thales respectively. In Study 3, the Dilemma Game used their newest HTML5 based game engine, ensuring compatibility on most devices. As in real-life crisis management decision-making situations, the Mayor Game, or Dilemma Game, makes use of dilemmas, where there is no correct decision to take. The only thing that matters, is how players reach a decision.
Study 1
In the first study, we investigated the effect of induced stress on analytical skills, which we measured within the Mayor Game, using a control and an experimental condition. We aimed to develop an unobtrusive assessment method of experienced stress using participants gameplay behavior. Although participants in the experimental condition tended towards a more intuitive and less accurate decision-making strategy, in-game indicators of gameplay behavior only partially predicted the through self-reports and physiological sensors collected measures of stress. More powerful analysis tools and a more targeted method of inducing stress could help to gain more insight into the unobtrusive assessment of stress in digital serious games.
Study 2
As finding and evaluating information and its sources is crucial in crisis management decision-making, we aimed to develop an unobtrusive assessment method to measure IL in a digital serious game in our second study. To validate our unobtrusive assessment method, we employed a self-report measure of IL. Using latent profile analyses (LPA) in a data-driven bottom-up analysis, we were able to adequately relate different patterns of gameplay behavior to higher or lower IL skill levels. Although the employed self-report measure still needs to be validated in future research, this study and its results are a starting point for making the training of crisis management decision-making more effective and efficient.
Study 3
In the third study of this dissertation, we focused on investigating whether a self-reflection through social comparison influences the in-game behavior of participants and subsequently on the in-game IL skill level, as unobtrusively measured using the in the previous study obtained results. In contrast to participants in the control condition, participants who were shown a dashboard facilitating this self-reflection after completing half of the game, increased the amount of information and number of consulted advisors in the second half of the game. Further, these participants also took slightly more time to reach a decision, which only makes sense if they consult more advisors and read more information. However, no meaningful between-condition differences regarding the in-game IL skill levels could be found in terms of change from first to second half of the game.
Conclusion
Real adaptivity in DGBL environments is hard to achieve. Especially in decision-making situations where all decision options are equally viable, we cannot base adaptive elements on a directly observable outcome. Therefore, in-game behavior of players has to be used, such that DGBL environments can assess players continuously and adapt accordingly. With the research presented in this dissertation, we set some valuable starting points to further develop DGBL environments that can be used for crisis management decision-making training.
We observed that players who were less stress resistant, lost accuracy in evaluating information. Furthermore, we found that unobtrusive assessment of skills like IL is feasible using players’ in-game behavior. This allows for further development of instructional interventions that can be incorporated in DGBL environments. Last, we observed that self-reflection through social comparison can be a valuable tool in triggering change in players in-game behavior. However, to trigger long lasting behavior change that is reflected in the actual skill level, we conclude that the dashboard used to guide the self-reflection should be more targeted and more specific. In an optimal case, interpretations on the in-game behavior, supported by specific suggestions for improvement can be displayed to the player. Then can players effectively use the information to critically evaluate their own in-game behavior and adjust it accordingly.
