There are several reasons to consider the possibilities of digital testing in order to assess knowledge and skills of students. One reason is that automated grading is very efficient for teachers, because grading takes a lot of time. Besides this, it is easier for teachers to monitor weak and strong subjects, and monitor the level of students and their progress. Another reason is that digital testing makes it possible to create more different kinds of test items (adaptive testing, using pictures or video, randomize numbers in case of a math test). Students can receive immediate feedback on their knowledge and skills, and therefore digital testing stimulates the learning process.
In collaboration with the TELT team the educational program Applied Math has performed a pilot with Chromebooks to establish whether chromebooks could be used effectively and securely to deliver a math exam using the online application MyLabsPlus. Almost 60 Electrical Engineering students took part in the pilot.
Why Chromebooks? Chromebooks have excellent battery life, quick boot times and, because they are designed primarily for cloud-based applications, they are much cheaper than a Windows-based laptop. In order to conduct the digital test with a safe solution, the UT purchased Chromebooks, combined with a Google Education Device Management Console. These Chromebooks where placed in a separate network, from where the web-based exam environment of MyLabsPlus was being accessed.
The Chromebook solution functions as an "web-only" exam-kiosk machine. With the help of the Google Device Management Console, all local and remote resources (e.g. USB ports, internet) can be blocked with little effort. Students were only given access to specific parts of the exam environment.
The Chromebooks were pre-assigned to a dedicated network, and therefore were assigned a specific IP-range. In this pilot setup, access to the network was username/password based, as certificate authentication was still under development.
The MyLabsPlus exam environment was only opened for the specific time of the examination, and was only accessible from the dedicated assessment network.
The pilot was conducted by taking the following aspects into account:
A Penetration / Hack test was done to test the security of the system
In the test session, the students got acquainted with the system and learned how to start up and enter the exam environment
Students executed the exam using the Chromebook devices
The test was scheduled in a large room to accommodate all the students, and with an appropriate storage / working room to accommodate the Chromebooks
The Pilot has been surveyed among the students. Some of the results are summarized in the figure below:
Note: The students could rate the questions on a scale of 1-5. (1 = Fully disagree; 5 = Fully agree)
- During the test session there was a small technical issue. During these early stages, all parties involved in setting up and supporting the examination environment must be on standby when performing assessments.
- When using a standalone assessment system (like MyLabsPlus), students may have forgotten their system- specific password. It is important to instruct students in advance to test their login details. Even better would be integrating the assessment system with the UT authentication server.
- It takes about 30 minutes for one person to supply 60 Chromebooks to the students, and another 30 minutes for collecting them afterwards.
- Access to the wireless network was username/password based. It is recommended to use certificate authentication.
- Students need bigger tables to use the Chromebook and also write down solutions on paper before entering them in MyLabsPlus.
- Designing a math test takes time. It is not useful to use the written exam and only change it into a digital test. Test items should be suitable for digital testing.
- Students are not used to doing a digital math test. Although the learning objectives are the same as last year, it is very important to train them to do the digital testing to get acquinted to the new kinds of test items. There should be an alignment between the learning methods during the module and the digital test.
- It’s obvious that students do not completely trust that the digital exam is safe for testing. Students believe that there are always ways to undermine the digital environment. We should involve students more in this new way of testing to let them experience the testing is safe.
- 60 Students were safely assessed with this solution
- The system was technically stable
- Both lecturers and students are very satisfied with the technical solution
- The Therm as the assessment location provided good facilities for conducting the examination, as well for the logistics around it.
- Starting up the assessment environment was smooth and straightforward, without needing specific instructions.
- The management of the whole setup is relatively low.
- The grading time decreased significantly.
- The grading is more reliable (the answer is eiterh good or false) and therefore more fair to students.
- There are more possibilities for digital testing to discover which can optimize the learning process of students (adaptive, more frequent testing for example).
- Digital testing makes it possible to randomize the numbers in test items. That could be another good way to prevent cheating.
During the pilot only the Electrical Engineering students were assessed by using Chromebooks. The other 500 students took the same test on paper. The UT will continue running more pilots with this setup. More Chromebooks have been purchased to also facilitate bigger groups of students. Additional pilots on assessing skills will be executed using desktop software on a Chromebook.