MASTER Assignment
automatic short answer grading using text-to-text transfer transformer model
Type : Master M-BIT
Period: Jan, 2019- Oct, 2020
Student : Haller, S.M. (Stefan, Student M-BIT)
Date Final project: October 6, 2020
Supervisors:
Abstract:
We explore in this study the effects of multi-task training and domain adaptation on Automatic Short Answer Grading (ASAG) using the text-to-text transfer transformer model (T5). Within this study, we design an ASAG model and evaluate its applicability to a practice dataset. We fine-tuned a multi-task model that is trained on a profound selection of related tasks and an extensively pre-trained model. We evaluate the performance of the models on the SciEntsBank dataset and achieved new-state-of-the-art results. With the best performing model, we showed that domain-independent fine-tuning is preferable to domain-specific fine-tuning for data-sparse cases. The optimized model was used and demonstrated in the university context. The prediction behavior of the model was explained with different model-agnostic methods which resulted in several hypotheses. The reported results reveal that the model is biased towards correct answers and has problems with partially correct answers. Through the knowledge about the decision behavior, the model's robustness was evaluated and tested. Within a validation study, we asked students to generate manipulation answers. Our findings emphasize the susceptibility of the model towards manipulations and difficulties with handling imbalanced and sparse data. We observe that for a functional ASAG model balanced and extensive data are necessary.