(by dr. Sjoerd van den Heuvel, dr.ir. Jan de Leede, Marloes Luttikhuis MSc, drs. Chris Rouwenhorst)
Within the course "Strategic HR Analytics" we conducted a pilot with digital summative assessment. The lecturers (Van den Heuvel en De Leede) were looking for an innovative way to make the assessment process more efficient. The main reason was to improve the grading process by making it more reliable and less dependent on illegible student handwriting. The pilot group consisted of 20 master students in an exam, and a resit with essay questions. The solution provided was a combination of the assessment function in BlackBoard combined with the Respondus Lockdown Browser.
For the exam and resit, the students were allowed to bring their own device (BYOD). Students needed to download an external program called 'Respondus Lockdown Browser' to ensure secure access to the test environment in BlackBoard.
The following precautionary measures were taken:
- To let the students feel comfortable with the systems, a trial session was scheduled during the lectures.
- For any technical issues which couldn’t be solved during the exam, spare laptops were provided.
- Students who were unable to bring an appropriate device could also use a spare laptop.
- If all else failed, a paper-based version of the exam was available.
- The technical and organizational lay-out of the lecture hall was inspected and prepared before the session.
- In collaboration with funcational application management, the test settings in the systems were re-checked.
During the trial session no real problems occurred. Almost all students were able to access the systems.
During the actual exam, students working with iOS experienced some difficulties with the screen size and ability to scroll down with the mouse. Because of this issue 5 students were unable to finish the exam on their own device.
One of the students experienced technical problems with 'Respondus' and wasn't able to finish the exam on a laptop.
None of the he above issues occurred in the resit. All students were able to finish the exam.
Experience of the lecturers:
The lecturers were satisfied with the process of this pilot. Digital assessment helps to make the assessment more objective. They recommend to use this solution in small student groups and involve support staff in the process.
Experience of the students are shown in the following figures:
Experience of TELT team:
Setting up this pilot was an organizational challenge; the initial time investment was high. Overall we are satisfied with the results.
- Always check the grading settings upfront
- Make sure that the trial session is long enough so that students experience all aspects (and issues) of the exam
- Ask support services to re-check the settings of the exam
- A lot of organizational challenges arise with the use of spare laptops. Consider using a paper-based backup.
- More research is necessary to determine if this solution is safe and secure enough and can be applied to a larger setting. There is some evidence that Lock down browsers are not soo safe as we thought.
- Always check the exam location upfront to see if it meets the technical requirements (WIFI, charging possibilities etc.) as well as organizational ones (number of chairs, distance between seats, etc.)