Quality requirements and test quality improvement

In addition, we naturally also aim to develop efficient exams. This means that the information obtained by testing must justify the investment (in time and money) required.

Each test format has its own characteristics. In some cases validity is the dominant characteristic, in others it is reliability. The goal is to find the optimum formula. SMG has a tool that you can use to investigate the quality of your own test. All you have to do is to answer a few questions.

TIPS for optimizing the quality of exams

The following tips can help you to improve the quality of tests:

·

Tips to optimize a test’s validity

·

Tips to optimize a test’s reliability

·

Tips to optimize a test’s transparency

·

Before your students take the test, check it using the evaluation tool

Tips to optimize a test’s validity

With regard to study units taught by SMG: specific learning objectives must be formulated for each subject; the test formats must be derived from the learning objectives; and that the questions and assignments used in exams must reflect the learning objectives.

What can you do, as the examiner of a particular subject, to optimize the validity of your exams?

o

You can classify learning objectives by type, e.g. knowledge, skill, and attitude objectives, and/or a blend of these. This makes it easier to select an appropriate test format (or combination of formats).

o

In general, the subject matter tends to focus most on major learning objectives or topics. This suggests that the test should include more questions on major topics. That would increase the representativeness of the test.

o

A specification table (or test blueprint) is a tool for ensuring that tests are valid, both in terms of their content and of the concepts used. You can prepare such a table before you start compiling the test questions and/or assignments.

o

Provided the subject material accurately reflects the objectives (and learning objectives), it will be an excellent starting point when compiling test questions and assignments. In this, you should avoid of course a too strong bias on (knowledge) reproduction.

Tips to optimize a test’s reliability

With regard to study units taught by SMG, each exam (or part thereof) must have a scoring rule, i.e. a scoring key for closed-ended questions, a scoring system for open-ended questions, and assessment criteria for assignments. If a study unit is to be assessed by several examiners, they should all use the same criteria and the same standards.

What can you do, as the examiner of a particular subject, to optimize the reliability of a test?

o

To help compile effective test questions and assignments, use the tips and checklists for closed-ended questions, open-ended questions, and assignments.

o

If possible, ask a colleague to check the test questions. Key issues: 1) is there a discrepancy in the way that you and a colleague interpret questions?; 2) are the test questions sufficiently specific, i.e. they can only be answered correctly by students who have mastered the subject matter?

o

Prepare an assessment tool and use it to assess the exam (or part thereof). This is particularly important when it comes to open-ended questions or assignments. Scoring systems (open-ended questions) or standards (assignments) are essential to the reliability of the assessment process, especially where several individuals are involved in the assessment.

o

Look out for, and avoid, common assessment errors, to ensure that the test results are as objective as possible.

o

Evaluate the questions after the test. If possible and where appropriate, perform an item analysis. This will generate information about the reliability of the test. If, for example, very few students have been able to answer a particular question correctly, this may be the result of poor preparation on the part of the individuals in question. However, it may be that the question was unclear or more difficult than intended. Questions that are more often answered by high scorers than by low scorers are questions that clearly distinguish between groups.

Tips to optimize a test’s transparency

With regard to study units taught by SMG, subject-specific information is posted on Blackboard and/or OSIRIS before the start of the course of study and/or academic year. This includes details of the learning objectives of the subject in question, the test formats selected to test the learning objectives, how the various test formats contribute to the determination of the final grade, and the requirements to be met by assignments. In written examinations, the maximum achievable number of points for each question is indicated alongside the question.

What can you do, as the examiner of a particular subject, to optimize the transparency of a test?

In general, the transparency requirements can be satisfied by providing comprehensive information.

o

Post as much information about the assessment as possible on Blackboard.

o

In written tests, give clear instructions and provide clear information (if possible, on the test’s cover page). This should include details of the testing procedure, criteria and scoring information, the distribution of points between questions or assignments, the cut-off score, and the time allowed for the test.

o

Ensure that the exam has a clear layout, such that the questions are clearly distinguishable from one another, the numbering is clear, references to texts or illustrations are correct, and that there is a clear distinction between a question (or components thereof) and the answer options (MC).

o

Also ensure that examination supervisors are well informed about the test procedure, the requisite conditions for the test, and the time allowed for the test.

Sources:

Berkel, H. van, and Bax, A. (ed.) (2006). Toetsen in het hoger onderwijs (Testing in higher education). Houten: Bohn Stafleu van Loghum, 2006.

Handboek toetsplanontwikkeling Digitale Universiteit (Digitale Universiteit test development handbook), October 2004

## go to summary of job aids