==== June 2017. This page is for reference only. We have concluded NOT to go forward with the UTENSO solution as described below. We are steering our efforts in a different direction. See Digital testing by using Chromebooks. ====
Curricula have changed significantly, and specialized software is part of many courses (e.g. programming, math). A project is not suitable for assessing all technical skills. Students should be able to carry out test assignments independently and under time constraints. A project does not offer those restrictions.
Until now, the University of Twente did not have an option to use specialized software during exams. The University of Twente does not have dedicated digital assessment facilities.
However, students do carry their own laptops where this software is installed. But student laptops also contain programmes that should not be available during an exam: e-mail, Dropbox, and other communication tools.
A digital test can be more effective as a solution for the situation described above. The department of Advanced Technology (AT) and Applied Physics (TN) initially developed a solution (further developed by UNTESO) which allows the student laptops to run under a controlled operating environment by means of a USB stick.
Bring your own device (BYOD)
These laptops were started with a distributed boot, which limited their functionality. Only a few programmes were available, which were necessary for the test, such as MatLab and Libre Office Writer. The Internet connection was controlled, making it possible to deliver the results via the Internet, while making all other channels of communication (local and remote) inaccessible.
We conducted pilots with this system in the 2015 and 2016 course.
Training student assisstants
Student assisstants were trained to assist with any technical questions and to work as an invigilator / proctor during exams
In the test session, all students got acquinted with the system and learned how to boot the test environment
Actual test / Resit
Students executed the exam using their own devices
A delivery portal was created to safely transfer students’ work to the lecturers
The test was scheduled in a large room to accommodate all the students
Spare laptops were on site for students with technical issues and MAC users
A penetration / hack test was done to test the security of the system
The numbers of the pilot are summarized in the following figure:
The Pilot has been surveyed amongst the students. The results are summarized in the figure below:
During the pilot conducted in 2016, we had the following points of concern and lessons learned.
Points of concern
- It takes time for students to get used to the Linux GUI. For example copy / pasting materials is different.
- It is important to only give students with technical issues a spare laptop. When given the choice, students preferred to use a spare laptop. This resulted in a large number of spare laptops needed (30%).
- The system does not work on MacBooks.
- Spare laptops should be available when implemented campus wide.
- Logistics around spare laptops definitely need to be resolved.
- General organization and communication should not be underestimated.
- Over 100 students were safely assessed with this method.
- The system is technically stable (apart from Matlab-related issues, which were mainly solved in the last session with updated OpenGL drivers).
- Lecturers are very satisfied with the solution.
- We recommend using one dedicated location for digital testing with sufficient power connections (Therm).
- The USB sticks’ bar codes are scanned. Logistics around this worked fine.
- Booting the USB stick did not cause trouble with students, thanks to a dedicated instruction and test session.
- The exam submission portal works well.
==== January 2017. The pilot period has been concluded. Several pilots with the UTENSO method have been conducted. The TELT team is now evaluating the experiences and the results.
==== June 2017. We have concluded NOT to go forward with the UTENSO solution. We are gearing our efforts in a different direction. See Digital testing by using Chromebooks.