Experiences

Digital Testing with USB Sticks

==== June 2017. This page is for reference only. We have concluded NOT to go forward with the UTENSO solution as described below. We are steering our efforts in a different direction. See Digital testing by using Chromebooks. ====

Introduction

Curricula have changed significantly, and specialized software is part of many courses (e.g. programming, math). A project is not suitable for assessing all technical skills. Students should be able to carry out test assignments independently and under time constraints. A project does not offer those restrictions.
Until now, the University of Twente did not have an option to use specialized software during exams. The University of Twente does not have dedicated digital assessment facilities.

However, students do carry their own laptops where this software is installed. But student laptops also contain programmes that should not be available during an exam: e-mail, Dropbox, and other communication tools.

Solution

Digital assessment

A digital test can be more effective as a solution for the situation described above.  The department of Advanced Technology (AT) and Applied Physics (TN) initially developed a solution (further developed by UNTESO) which allows the student laptops to run under a controlled operating environment by means of a USB stick.

Bring your own device (BYOD)

These laptops were started with a distributed boot, which limited their functionality. Only a few programmes were available, which were necessary for the test, such as MatLab and Libre Office Writer. The Internet connection was controlled, making it possible to deliver the results via the Internet, while making all other channels of communication (local and remote) inaccessible.

Pilot Set-up

We conducted pilots with this system in the 2015 and 2016 course.

Aspect

Description

Training student assisstants

Student assisstants were trained to assist with any technical questions and to work as an invigilator / proctor during exams

Test session

In the test session, all students got acquinted with the system and learned how to boot the test environment

Actual test / Resit

Students executed the exam using their own devices

Delivery portal

A delivery portal was created to safely transfer students’ work to the lecturers

Logistics

The test was scheduled in a large room to accommodate all the students

Spare laptops

Spare laptops were on site for students with technical issues and MAC users

Hack test

A penetration / hack test was done to test the security of the system

results

The numbers of the pilot are summarized in the following figure:

The Pilot has been surveyed amongst the students. The results are summarized in the figure below:

Lessons learned

During the pilot conducted in 2016, we had the following points of concern and lessons learned.

Points of concern

  1. It takes time for students to get used to the Linux GUI.  For example copy / pasting materials is different.
  2. It is important to only give students with technical issues a spare laptop.  When given the choice, students preferred to use a spare laptop. This resulted in a large number of spare laptops needed (30%).
  3. The system does not work on MacBooks.
  4. Spare laptops should be available when implemented campus wide.
  5. Logistics around spare laptops definitely need to be resolved.
  6. General organization and communication should not be underestimated.

Positive points

  1. Over 100 students were safely assessed with this method.
  2. The system is technically stable (apart from Matlab-related issues, which were mainly solved in the last session with updated OpenGL drivers).
  3. Lecturers are very satisfied with the solution.
  4. We recommend using one dedicated location for digital testing with sufficient power connections (Therm).
  5. The USB sticks’ bar codes are scanned. Logistics around this worked fine.
  6. Booting the USB stick did not cause trouble with students, thanks to a dedicated instruction and test session.
  7. The exam submission portal works well.

Notes

====  January 2017. The pilot period has been concluded. Several pilots with the UTENSO method have been conducted. The TELT team is now evaluating the experiences and the results.

==== June 2017. We have concluded NOT to go forward with the UTENSO solution. We are gearing our efforts in a different direction. See Digital testing by using Chromebooks.