Testing and remote assessment framework

We use the term remote testing for any test activity in which the students are not phys- ically present and supervised. In contrast, the ”normal” test circumstances are called controlled testing. This document sets out the guidelines for summative remote testing to be used at the University of Twente.

Change in test schedule

In most cases, remote testing involves a change in the way a module or course is assessed. Switching to remote testing therefore involves a change in test schedule. This has a number of consequences:

• The nature of tests has to be changed so that they are suitable for taking remotely.
• A controlled test will always be necessary to accommodate students who are for technical reasons prevented from taking the remote test.
• If under the new test schedule, some learning goals are insufficiently covered, this should be explicitly made clear.
• The Examination Board has to approve the changed test schedule.

Grades obtained from remote tests are never automatically final grades: the teacher will have to judge which of the results, in his or her opinion, are reliable enough to  be raised to the level of a final (Osiris-registered) grade. Thus, a remote test grade is initially a preliminary grade. This may range from:

• (Only) those results that raise no suspicion of fraud.
• (Only) those results that are in line with the general level of performance of the student.
• None of the results, in case there are too many doubts about the reliability of the entire test.

If a preliminary grade is not promoted to a final grade, essentially the test has retroactively been turned into a formative test. Students must be made aware of this situation. See also ‘On the web: global explanation’, which points out the need for a web site on which the remote test policy is explained.

At all times, it may occur that a student is prevented from taking part in a remote test because of obstacles of a technical nature: the system they are using fails, the network fails, the support software for the test fails. In fact, we can’t even begin to check a student’s claim that one of these things has happened. We can’t do much else than conclude that such students have been unable to take part because of reasons outside their control, meaning they have to be allowed another opportunity.

It follows that a remote test can never serve as the only test opportunity: we will always have to organise a resit under controlled circumstances. Given, however, that we may not be able to offer such circumstances in the foreseeable future, we serve the interests of the student by enabling them to make progress.

As part of the measures to increase the reliability of remote testing (while acknowledging that this will never get close to 100%), we will request students to confirm that they are aware of our expectancy regarding their ethical behaviour when taking a remote test. This request will take the form of an integrity statement that will be part of every remote test. This involves the following steps:

1.       On the web: global explanation

There is a public web page where the policy of remote testing and the expected students’ behaviour is explained in some detail (but not legalese, we want a readable text). Students should be pointed to this before they partake in a test, at every possible occasion, and also from every test (see below).

The global document should also clearly state that anyone helping another is culpable.

2.       In every test: allowed sources

A source, in this paragraph and the next, is the analogon of ”material” but extended to humans: it refers to where information comes from that a student uses to make a test. Some sources are allowed (a cheat sheet, a book in case it’s  an open-book test,  a calculator, possible more) and some are not. Typically, the test schedule of a course already lists the allowed sources for each test.

For the purpose of remote testing, the allowed sources should be explicitly mentioned on the test itself — the reason being that the “additional question” discussed in subsection 3 (In every test: additional question) below can then with confidence refer to it. The list of allowed sources should be comprehensive, so it is clear that anything not on it is not allowed. We do not want to try and compose the complementary list of disallowed sources.

What should allowed sources be? Closed-book remote tests are very strongly discouraged. No matter how dire the warnings, the ease of looking up an answer in the book and the unlikelihood of detection (in the students’ perception) make it clear that no closed-book remote test can ever be reliable enough to yield final grades.

3.       In every test: additional question

Every (digital and written) test should contain the following as a first question. The an- swer cannot be wrong, but the absence of an answer invalidates the test for summative use.

• In case the test has an answer form:

Please read the following paragraph carefully, and tick the box to acknowledge that you have done so. To find more information, please consult [URL of state- ment, see subsection 3.1 above]
By testing you remotely in this fashion, we express our trust that you will adhere to the ethical standard of behaviour expected of you. This means that we trust you to answer the questions and perform the assignments in this test to the best of your own ability, without seeking or accepting the help of any source that is not explicitly allowed by the conditions of this test. Please tick: [checkbox]

• In case the student submits his answer in his own sheet or format:

Please read the following paragraph carefully, and copy the text below it verbatim to your answer sheet. To find more information, please consult [URL of statement, see subsection 3.1 above]
By testing you remotely in this fashion, we express our trust that you will adhere to the ethical standard of behaviour expected of you. This means that we trust you to answer the questions and perform the assignments in this test to the best of your own ability, without seeking or accepting the help of any source that is not explicitly allowed by the conditions of this test. Text to be copied:
I will make this test to the best of my own ability, without seeking or accepting the help of any source not explicitly allowed by the conditions of the test.

Every remote test should be accompanied by an additional time slot in which complementary orals can be planned. The purpose of a complementary oral is twofold:

• The teacher can check whether the understanding of the student is in line with the quality of the submitted test, and use this as a basis for deciding on the reliability of the test (in other words, whether the preliminary grade can be promoted to a final one).
• The students will realise that they can possibly be asked to elaborate on their answers, which should serve as another incentive against cheating.

In most cases, only a selection of students can be tested in this way. There are two distinct strategies to come to a selection:

• A random selection is made from all students who submitted as solution, and the oral is held very quickly after the test;
• The selection is based on the results of the test, out of the students whose grades are not deemed reliable enough to be turned into final grades. In this case, the oral can only be held after the assessment has been finished.

It is recommended that in the second case it also includes randomly selected students. This avoids that students immediately perceive their selection as a suspicion and it gives the teacher a way of comparing how the average student responds in these oral exams compared to the students for whom a flag was raised.

In the current situation due to Corona, it is expected that the number of oral exams might increase significantly.

In most programmes the rules and guidelines of the exam committee require the presence of a second person besides the examiner during an oral exam. Sometimes the recording of the oral exam is explicitly mentioned as an alternative for the second person. If we have many oral exams, presence of a second person will not always be feasible. Note that that the second person may be a teaching assistant or PhD, not necessarily with the university teaching qualification.

It is recommended to make a recording of an oral exam with only one student and one examiner present. Otherwise, there is no way to evaluate the oral exam from the context of quality assurance nor is there a reasonable procedure when a student objects to his grade.

Based on the privacy laws in the Netherlands the oral exams can be recorded provided that this is clearly communicated beforehand to the student and that it is clear how long these recording will be saved and who will have access to them. Based on the above the following procedure is recommended:

1. Oral exams are recorded when only one examiner is present. A student can refuse to an oral exam with recording but then might have to wait until the situation normalizes before the student can have his exams.
2. The recordings are removed after one year. The examination board is responsible to ensure that these recordings are deleted.
3. The recordings are only used for quality control (exam committee, accreditation committee) or for processing an appeal by the student.

For organizing and recording an oral exam, TELT is recommending the following tools.

These recordings are clearly sensitive information that need to be securely stored. It is recommended to store those in a central location with proper authorization. The technical realization still needs to be arranged with LISA.

The UT offers students the opportunity to use ”additional features” for written tests, extra time among them. See the Examinations Office website for the full list. These do not directly transfer to oral exams. However, personal circumstances might apply (ranging from extreme nervousness or stuttering to medical conditions) under which a student might need extra time in an oral exam. The advice for dealing with this is to be flexible with granting extra time, in such a way that every student has a fair chance to demonstrate his or her knowledge and skills regarding the subject.

• Student do have right to review their exam also in an online environment. You can use the speed grader in Canvas or annotations to facilitate this process. In case of force majeure (i.e. exams written before 17 March 2020 which were not reviewed yet), you can explain to the student that he/she has to wait until we can resume off-line activities. In case this leads to insurmountable problems the student can contact the study adviser/programme coordinator. In this way we create an overview of the extent of the issue.
• Give information on the organization of the exam as soon as possible. Do not wait until just before the exam.
• You can use Microsoft Teams to get in touch with students. There is no need (anymore) to ask students for their phone number as part of a procedure to avoid fraud.

Links

• Request Form Examination Board
• Information for students about remote written assessment