Remote assessment | Choosing remote assessment method & tool

Choose the suitable remote assessment method

Changing a course from face-to-face to online teaching poses particular challenges. One challenge will be finding a suitable way to assess your students. Below you find a 'decision scheme’ to help you with this challenge. Additional we present points to consider when remotely assessing students and advice on how to keep your assessment valid, reliable and transparent. The final part of the guide offers more elaborate explanations on various assessment methods, including the advantages and disadvantages. Keep an eye on the TELT-site for updates and more information.
This guide has been drawn up most of all with the lecturers in mind who work with large groups and used to test by means of written tests with open or closed questions.

Important: if, as an examiner, you want to adjust your assessment, check what the current procedure is for adjusting the assessment for your own study programme. What freedom do you have as an examiner in the current situation? In the event of drastic changes, coordination with and permission from the programme director and/or examination board will be necessary in advance.

If you're wondering which specific tool to use (like Canvas Quiz or Assignments or Remindo), this flowchart may help; click on the picture left.

Click on the scheme below for a (larger) PDF version.

This website is created by CELT/TELT (University of Twente). With permission, grateful use has been made of ideas and texts from Y.P. Hsiao (Tilburg University) and G.A. van de Watering (Eindhoven University of Technology).

Some points of concern when choosing a remote assessment method
- We all need time to get used to the new situation; not getting discouraged and success experiences are essential for students, teachers and everyone involved in education. Right now, it is essential to keep an eye on the essence of a programme (and module/quartile) on the one hand versus using a dose of realism and pragmatism on the other hand, and this will sometimes lead to concessions. Maybe as guiding principle for now: If it can't be done in the best way, then for now the best way is how it can be done.
- Keep an eye on the bigger picture and the final qualifications. What is the essence of a specific test within a larger framework? Is the test used to make important decisions about the individual students? Are knowledge and skills assessed that are also covered and assessed further on in the programme (often at a higher level)? Will the assessment in a later phase provide enough confidence that students meet the final attainment targets? Is it possible to omit a certain test or to use it in a formative way? The assessment programme at programme level can provide insight. Are the knowledge and skills also largely assessed in the project work of a module or another assignment in a course? Or could this be the case with some slight adjustments? A test plan at module/course level provides insight.
- Students need to be accurately informed about every new course of action. Especially when it comes to assessment. But let's certainly not forget that we can also actively involve the students. They can think along with us about other forms of assessment based on learning objectives and attainment targets.
- Preventing and detecting fraud is important in order to make a well-founded, reliable statement about someone's knowledge and skills. Although 'fraud' receives a great deal of attention in the following texts, it should not be forgotten that committing fraud is certainly not common practice and that the majority of students can be expected to act honestly and ethically. They are also keen to acquire sufficient knowledge and skills to be able to start a further education or a job market.
Considerations - Keeping the quality criteria in mind
Validity. The assessment method should be appropriate to assess the achievement of the learning objectives. The learning objectives should be covered. Considerations:
o    Can you also assess your learning objectives differently?
o    Do you assess at the same level or another (higher) level when you use a different
assessment method?
o    Is it possible to extend a test within one module so that multiple learning objectives
can be tested together or integrated that were first assessed separately?
A test plan and test specification table (examples) will help to gain insight into how learning objectives are assessed and with which alternative forms of assessment it is possible to assess (almost) the same learning objectives.

Reliability. How can you ensure that the assessment is as reliable as possible, that the marks obtained are a good reflection of the student's level? How consistently and objectively are the results assessed? Is inter-rater reliability ensured if there are multiple assessors?
o   An important point of attention in remote assessment is the susceptibility to fraud.
How can preventive measures be taken to increase reliability? What can we do to
   check whether fraud takes taken place at the time the test is taken? How can we
   detect fraud based on the results afterwards?
o   What degree of uncertainty about the reliability do we accept in the remote
assessment?
Efficiency. The chosen assessment method should be feasible from the teacher's perspective. The time required for reviewing the work of the students should be do-able. Efficiency and the concern for high reliability can sometimes be at odds with each other. Considerations:
o    Can additional assessors be used?
o    What are the optimal or maximal numbers for a particular assessment method?
o    How does the test-taking time relate to the reviewing time? An oral test for a larger
group of students takes a lot of time, but is probably more reliable than an online
multiple choice test and it will not need a lot of time to assess written work
afterwards.
Transparency. If the students are not familiar with the new assessment method, it is essential that the procedures and expectations are accurately explained (e.g. in a video) and that examples are made available. This also applies if the questions will be asked at a different level than would normally be expected. Make sample questions and if possible, discuss these questions online with the students.
Technical possibilities and potential technical problems. When using online options for assessment, technical problems can occur on the system side, but also every individual student can encounter technical problems. It is important to consider how to deal with these problems in advance. What will be alternatives for students who will encounter problems? Could, on an individual and incidental basis, an oral test be used? A special point of attention is the facilities for students with special circumstances. How can these students be accommodated?
Recommended: Use only the existing systems supported by the UT (see TELT website). Students are more often already familiar with these systems and support is available. For systems that are 'public', alienation of data and privacy issues are an issue.

More detailed explanation of the different assessment methods

In this section we provide a quick overview of different remote assessment methods to help you choose. The options are not exhaustive and combinations are possible. When you want to discuss what will be a good choice in your specific situation, you can contact the CELT educational advisor in your faculty. If your questions are about technical possibilities, the e-learning specialist or members of the TELT-group can support you.
The information we present next will be updated regularly. For example, a new tool may become available or we have additional tips & tricks, based on experiences form teachers.

NB. In the descriptions below, no attention is paid to facilities for students with special circumstances. As before, attention will have to be paid to these facilities as well. If necessary, the use of an alternative test form may be considered for specific circumstances. At the moment (31-3-2020) many study programmes offer tailor-made solutions. Work is being done to collect possible solutions and good practices

Written test online – open book test (resources allowed); open questions. Simultaneous online examination for a whole group. Sit-down / limited-time test

What	Students complete tasks and/or answer questions and submit their work within a strict time limit (hours) while working off-campus. The whole target group will get the test at the same time. The tasks or questions are asking for in-depth knowledge, insight, application, analyses, retrieval and transfer skills (higher order cognitive skills). Students can use their books or notes. The emphasis is on the use and application of what is studied and the extent to which the students are able to find their way and make use of the available materials.
How	Depending on the subject matter and type of tasks or questions, the answers can be written out by hand (e.g. when drawings have to be made or formulas have to be used) or online forms with questions can be used (Canvas/Remindo). If handwritten answer sheets are used, a clear procedure must be drawn up for starting and handing in the materials. Handing in the materials can be done for example by asking the students to hand in a clearly legible photo and/or scan of the delivered work.
Time	Think about what a realistic period of time is for the exam. If questions are more complex, cases are used or a student is required to look up certain information, the students will need ample time for that. Expect them to have read and understand the learning materials, but do not expect them to have learned it by heart. Ask for short answers or limit the answer space, so that the students are not busy writing down an answer for longer than necessary or, when they don't know an answer, start writing down as much as possible in the hope that the correct answer can be found somewhere in between.
Student preparation	Inform the students thoroughly about the procedure. Provide practice during the education process. Be aware that students may falsely assume that an open book test will be easy because they can look up answers. During test taking: make support available (both technical, and for clarification in case of factual errors in questions etc.) for the duration of the assessment.
Advantages	Allows students to demonstrate their knowledge, abilities and use of sources. Allows for assessment of higher order cognitive skills (argument, application, comparison, critique etc.). Can be marked by more than one assessor if necessary.
Dis-advantages	It offers opportunities for fraud. The procedure needs to be carefully thought out. There may occur technical problems.
Concerns	You have to think very carefully about the formulation of the questions or tasks. Also you have to make very clear what kind of answer you expect (e.g. just the answer or a worked out solution that will show how the student came up with this answer). See below at More information for some practical guidelines. Think about beforehand what to do when technical problems will occur. What if a student says his computer crashed? Can an oral exam be used as a second chance?
Ideas	An MC exam and open exam can be combined, if students are asked to explicitly justify a certain answer. The justification largely determines the score per question.
Grading, reliability and quality assurance	Use an answer model or rubric to assess. If more than one assessor is involved, pay attention to inter rater reliability. Make the answer model together or discuss this , compare the results for the first few students or for a sample. Discuss border cases (like a 5.5). Online tests carry an increased risk of unauthorized consultation or collaboration between students. Preferably the tasks and questions are of such a nature that outside assistance or cooperation will not help much or can be detected afterwards. Ask for own examples or elaborate explanation for instance. Similar (testing the same learning objectives) but different tasks and questions, can be used to prevent or discourage collaboration and might make it easier to detect. Depending on the assessment policy and judgements of the Examination Board regarding the use of this, online proctoring can be considered. But this provides no guarantee. To point to a code-of-conduct or let students sign it, might help, but does not give guarantee either. Check the answers afterwards: Any particularities stand out? Similar answers? Different styles or use of language in one answer sheet? Consider what steps might be taken if fraud is detected or suspected. Inform and warn the students beforehand that attention will be paid to fraud and that fraud will have consequences.
Tools	Answer sheets or photo’s can be delivered by way of “assignments” in Canvas. You can make the assignment visible at a certain time and let students submit their work before a certain deadline. Advantage: plagiarism check can be used. Canvas and Remindo provide possibilities for online test taking and submissions with time limits. Grading and giving feedback can also be conducted online. for support with your choice, click on the picture. The advantage of Remindo is that it keeps a record in the logbook of what the student does in the test (what he types in the answer field) and you can use more complicated question formats. The choice may also depends on your experience with a certain system. For more information about the (available) tools, see the TELT site.
More support & information	Step-by-step guide from CELT: How to design and execute an online open book exam Guidelines: A guide for academics – Open book exams. The university of Newcastle, Australia. Example for information to provide to the students: Open Book and Take Home Exams. USNW Sydney.

Written test online - open questions (no resources allowed) ; testing lower and basic higher order cognitive skills // Simultaneous examination for a whole group

What	Students complete tasks and/or answer questions and submit their work within a strict time limit (hours) while working off-campus. The whole target group will get the test at the same time. The tasks or questions are asking for factual knowledge, insight, application. Here and there maybe aiming at higher order cognitive skills. Students are not allowed to use resources of any kind.
How	Canvas / Remindo provide possibilities for online test taking. Test taking on paper, handwritten, is also an option. An online proctoring tool can be used in combination with the test instrument in use.
Time	Think about what a realistic period of time is for the exam. In general this will not differ much from common practice for test taking. Extra time can be allowed for the technical actions.
Student preparation	Inform the students thoroughly about the procedure. Make support available (both technical, and for clarification in case of factual errors in questions etc.) for the duration of the assessment.
Advantages	Allows students to demonstrate their knowledge, abilities. Can be marked by more than one assessor if necessary.
Dis-advantages	It offers opportunities for fraud. The procedure needs to be carefully thought out. There may occur technical problems.
Concerns	Main concern: opportunities for fraud. Think about beforehand what to do when technical problems will occur. What if a student says his computer crashed? Can an oral exam be used as a second chance?
Ideas	Combining and MC exam and open exam by asking students explicitly to justify a certain answer. The justification largely determines the score per question.
Grading, reliability and quality assurance assurance	Recommended to use an answer model or rubric to assess. If more than one assessor is involved, pay attention to inter rater reliability. Make the answer model together or discuss this , compare the results for the first few students or for a sample. Discuss border cases (like a 5.5). Although the chance of fraud will be slightly lower than with MC exams (a letter is easier to pass than a text; writing answers will leave less time for other actions), the chance is still considerable. Fraud by 'cheating' (book, notes, second computer close by or just being able to use Internet), identity fraud, unauthorized cooperation (someone passes on all the answers via e.g. a phone). It seems almost impossible to exclude the possibility for fraud. At most, it can be made more difficult. > use different tests in which the questions are randomized; > preferably the tasks and questions are of such a nature that outside help or cooperation will not help much or can be detected afterwards. Ask for own examples or elaborate explanation for instance. > similar (testing the same learning objectives at the same level) but different tasks and questions, can be used to prevent or discourage collaboration and make it easier to detect fraud. > online proctoring – if available for the UT (see TELT site) - can be considered but provides still no guarantee. For online proctoring there is a lot to consider, like for instance privacy rules en technical issues (see Whitepaper Online Proctoring. Surf.) > point out to or let students sign a cover sheet about code-of-conduct for test taking (send in scan or photo) might make students more aware of ethical behaviour and the consequences of misconduct, but does not give a guarantee. Check the answers afterwards: Any particularities stand out? Similar answers? Different styles or use of language in one answer sheet? Consider what steps might be taken if there is doubt or fraud is detected. Inform and warn the students beforehand that attention will be paid to fraud and that fraud will have consequences.
Tools	Answer sheets or photos can be delivered by way of “assignments” in Canvas. You can make the assignment visible at a certain time and let students submit their work before a certain deadline. Advantage: plagiarism check can be used. Canvas and Remindo provide possibilities for online test taking and submissions with time limits. Grading and giving feedback can also be conducted online. for support with your choice, click on the picture. The advantage of Remindo is that it keeps a record in the logbook of what the student does in the test (what he types in the answer field) and you can use more complicated question formats. The choice may also depends on your experience with a certain system. For more information about the (available) tools, see the TELT site.
More information	Tips for constructing questions can be found [here]

Written exam online with closed (for instance multiple choice) questions; no resources allowed Simultaneous examination for a whole group. Sit-down / limited-time test

What	Students complete tasks and/or answer MC questions and submit their work within a strict time limit (hours) while working off-campus. The whole target group get the test at the same time. Whether students can use their books or notes has to be decided.
How	Canvas / Remindo provide possibilities for online test taking. An online proctoring tool can be used in combination with the test instrument in use.
Time	Think about what a realistic period of time is for the exam. This will be not much differ from common practice for test taking. A general guideline for MC questions is 45 sec. per question. But this depends very much on the type of question; how much time it takes to read through the question properly + thinking time. If something has to be calculated, it will take more time.
Student preparation	Inform the students thoroughly about the procedure. What is and what is not allowed during the test taking. Make support available (both technical, and for clarification in case of factual errors in questions etc.) for the duration of the assessment.
Advantages	The results can be assessed quickly and objectively. This will be especially an advantage for large groups.
Dis-advantages	It offers ample opportunities for fraud. There may occur technical problems. It's not easy to construct good closed questions. Especially when it comes to testing higher cognitive skills.
Concerns	Fraud is the main concern. How to prevent and detect? Think about beforehand what to do when technical problems will occur. What if a student says his computer crashed? Can an oral exam be used as a second chance?
Grading, reliability and quality assurance	Grading will be easy and when Remindo is used, it can be done automatically. Test analysis results will be provided automatically. Online tests with closed questions (like MC questions or other types), testing for factual knowledge and insights, carry a high risk of fraud. Fraud by 'cheating' (book, notes, second computer close by), identity fraud, unauthorized cooperation (someone passes on all the answers via e.g. a phone). It seems almost impossible to exclude fraud. At most, it can be made more difficult. If closed question tests are taken online, the following measures can help to at least limit the possibility of fraud: > use several tests in which the questions are randomized; they are coded, but the code can't be identified by students; if possible, the answer options for a number of questions can also be changed. NB: This might make automatic assessing afterwards more complex, but systems like Remindo are able to use tests with randomized questions and the coding may help to combine tests and key answer lists. > use a test bench if available so that similar tests can be taken with different questions. > limit the time per question, after this time the question will disappear; in general 45 sec. is a good estimation, but this depends very much on the type of question. Ask a colleague to try it out and then add a little more time. NB. These kind of option should be technically possible. At the moment this option does not seem available yet for the UT systems (Canvas, Remindo). > the questions are presented one by one and after a chosen answer one cannot browse back. > an online proctoring tool can be used as way to prevent and detect fraud. > let students sign a cover sheet about code-of-conduct for test taking (send in scan or photo), this might make students more aware of ethical behaviour and the consequences of misconduct, although it will not give a guarantee. Important: Be transparent towards the students. Prepare them for the measures taken. For example, arrange for a practice test with a time limit per question. Be aware: time pressure and limited time for each question, may cause more tension and anxiety for many students. This may influence the results and make the results less reliable. Check the answers afterwards: Any particularities stand out? Consider what steps might be taken if there is doubt or real fraud is detected. By doubt an extra (short) oral might eliminate the doubt. Inform and warn the students beforehand that attention will be paid to fraud and that fraud will have consequences.
Tools	Canvas and Remindo both provide possibilities for online test taking and submissions with time limits. Grading and giving feedback can also be conducted online. For support to choose a system, click on the picture. The advantage of Remindo is that it keeps a record in the logbook of what the student does in the test (what he types in the answer field) and you can use more complicated question formats. The choice may also depends on your experience with a certain system. For more information about the (available) tools, see the TELT site.
More information	Tips for constructing questions can be found [here]

Take home exam based on complex tasks, a case or higher order questions. Restricted period of time.

What	Students can do the test individually in their own time, although within a restricted period of time (a day or week for instances). The test is based on questions or tasks asking for applying higher order cognitive skills and showing in-depth knowledge and understanding of the what is learned. The exam can for instance be based on one or more cases on which questions are asked. The answers are often extensive and may involve the application of knowledge or skills, problem solving, arguing a choice, comparing theories etc.
How	The exam is distributed at a given moment and students are required to submit their answers before a given deadline. The length of the answers should be limited to a set number of words/pages (specify font and line distance) to limit the time needed for assessing the work, to encourage students to present their answers concisely and precisely and to prevent students from writing down as much as possible in the hope that there is a meaningful part somewhere in the entire text they present.
Time	Depends on the tasks. Suitable in relation to the learning objectives and education process. The time period can be the same for every student or it can be arranged that students have limited time available to fulfill the tasks or answer the questions (for instance 8 hours) but they can start when it is convenient for them. The clock then starts ticking when students download the exam paper.
Student preparation	It should be clear for students what they should focus on in their preparation. The exam should be clear. The procedure should be clear. There should be a possibility for students to ask the teacher questions if the tasks or questions are not clear.
Advantages	Possible to test integrated use of knowledge, attitude and skills. Especially aiming for testing higher order skills: application, analysis, evaluation and creation. It provides students a chance to ‘shine’.
Dis-advantages	Assessing the work will take a lot of time. Difficult to come up with the right tasks, cases, questions.
Ideas, specifics	The tasks and questions can be made interesting and motivating for students. Maybe allowing some choices (but at the same time make sure that this will not entail the risk that the learning objectives will no longer be tested).
Grading, reliability and quality assurance	Recommended: use an answer model or rubric to assess. If more than one assessor is involved, pay attention to inter rater reliability. Make the answer model together or discuss this , compare the results for the first few students or for a sample. Discuss border cases (like a 5.5). Take home exams carry an increased risk of students getting help from someone or even someone else may do the exam. Preferably the tasks and questions are of such a nature that outside help or cooperation will not help much or can be detected afterwards. Similar (testing the same learning objectives) but different tasks and questions, can be used to prevent or discourage collaboration and to make it easier to detect. To have students sign a code-of-conduct might help, but does not give guarantee. Plagiarism in Canvas can be checked with Simcheck. It is recommended to check the student results afterwards: Any particularities stand out? Similar answers? Different styles or use of language in one answer sheet? By doubt, an additional oral exam (to check) might be considered. Students should be informed in advance if this is a possibility. Consider beforehand what steps might be taken if fraud or plagiarism is detected. Inform and warn the students beforehand that attention will be paid to fraud and that fraud will have consequences.
Tools	Canvas Assignments provide possibilities for online submissions with time limits. Online grading and feedback is possible.
More information	Guidelines: Take home examination papers: guidance for tutors and course conveners. LSE Teaching & Learning Centre.

Oral exam

What	An oral exam assesses student’s learning based on the spoken word. Students are asked to use the spoken word, guided by questions or small tasks, to show they have achieved the learning objectives.
How	Some consideration, tips and tricks can be found <here>.
Time	An oral exam might take you about 30 minutes including preparation and grading.
Student preparation	Inform the students explicitly and elaborate about the expected procedure. If possible, let them practice in a (online) lesson. Maybe show a simulated setting in a video to give an impression. Tell them that the oral is meant to provide opportunity to show what they have learnt.
Advantages	You can test understanding of the topic individually in a reliable way. Efficient up until 100 students.
Dis-advantages	With a larger group of students it will take a lot of time and it's quite strenuous for the assessor. When oly one assesor will assess the oral and more than one assessor will be involved, consider how to guard the inter rater reliability. For instance: do the first few together. Use a clear checklist or rubric.
Concerns	A rule in the bachelor EER right now is: the oral examination is public, unless the Examination Board has determined otherwise in a special case. Is this rule workable in the online situation? How will you deal with this?
Ideas, specifics	An oral exam can be done in small groups. Interaction can be stimulated. An oral exam can also be combined with an assignment, for instance a (group) report. A tip by Kristina Edström: reverse the burden of proof. Ask the students in the first 7 minutes of an oral exam to show you that he/she has reached the learning outcomes. Follow-up questions will then pop up. Ask students kindly how they think it went after the exam. Make sure they know they have to show ”real” understanding, in real time.
Grading, reliability and quality assurance	It is recommended to use a grading scale or rubric to stay consistent and fair. A filled in standard form will help to substantiate your judgment in retrospect and upon request. A recording can be considered. The use of two assessors will increase reliability, but this might not always be possible. Assessment might be biased because of assessing pitfalls if the student is not anonymous. Student’s articulateness, shyness, speed of thought, gender, language skills, etc. can influence the judgment. Keep in mind during the oral exam that a student can be very nervous. So be aware of influencing factors which should not influence the determination of the figure. Point of attention: the first student can pass on the questions to other students. Consider whether this is objectionable and whether the order may disadvantage or benefit the earlier, later students. Similar but different question scenarios can be used to overcome this point.
Tools for online	Skype, Microsoft Teams, BlueJeans, Canvas Conference.
Information & tips	Step-by-step guide from CELT: How to design and execute an online oral exam A short guide to oral assessment. Gordon Joughin, 2010. Leeds Metropolitan University/University of Wollongong.

How to... guides for the design and execution of various online exams
Some brief step-by-step guides have been developed to support examiners with the design and use of remote assessment methods.
1. Step-by-step guide: How to design and execute an online open book exam
2. Step-by-step guide: How to design and execute an online oral exam
3. Step-by-step guide: How to design an group assignment
Good practices and experiences of other examiners
Examiners share there experiences and good practices in a Canvas-course.
See this website for more information and share your experiences, suggestions and good practices!
https://www.utwente.nl/en/telt/online-lectures/#share-your-experience

If you are an examiner and have questions or you like some support, you can contact:
for choosing a tool and technical issues	The e-learning specialist(s) in your faculty // Specialists from TELT
for helping you to choose a suitable assessment option in relation to you learning objectives and education process	The CELT educational advisor in your faculty. See for names and contact information: https://www.utwente.nl/en/ces/celt/who-we-are/
for assistance with the design of the test or assignment, for constructing the questions, for developing criteria and rubrics etc.	The CELT educational advisor in your faculty. See for names and contact information: https://www.utwente.nl/en/ces/celt/who-we-are/

Choose the suitable remote assessment method

If you are an examiner and have questions or you like some support, you can contact: