Enhancing the reliability of graduation projects assessment

Prior to organizing an activity, it is important to consider what goal is or goals are intended. What is the focus of the activity undertaken? Ultimately, all these goals may contribute to improving the quality of the overall graduation process. Nevertheless, the perspective varies and this may determine the design and implementation of an activity.

Possible goals can be for instance: 

• To investigate whether the assessments do justice to the quality of the work done; possibly also in comparison with the quality of graduation projects at similar programmes elsewhere (national or international).
• To investigate whether the assessments of assessors involved or whether the assessors of a programme are on the same wavelength and, if there are differences, find out what leads to observed differences.
• To calibrate; having thesis assessors discuss with each other how they assess, to promote consistency in assessment.  This can be specifically important when there is a lot of change in who is assessing and to “induct” new assessors. Reflecting on one's assessment methods can also be valuable for experienced assessors. 
• To examine ambiguous situations and circumstances. Are there aspects that make an assessment especially difficult? If someone takes much longer on a task than planned, do you factor this into the assessment? In what ways? Does everyone do so equally?
• To check whether the assessment criteria and assessment forms used are adequate and provide a good guide for the assessments. Are there any ambiguities? Differences in interpretations of certain criteria? Difference in the assignment of weight to different criteria? Are adjustments needed?
• To discuss the assessment criteria and assessment forms used, possibly also the vision regarding thesis work, the supervision process and important conditions (e.g. how much time is envisaged for supervision), procedures and support (e.g. documentation for supervisors and assessors, but also the manuals for the students), in order to arrive at a common frame of reference, similar working methods and sufficient support for the processes.
• To discuss bottlenecks experienced by thesis supervisors and assessors and possible measures. Or discussing measures planned by the programme (for example, the need to regulate the number of supervision hours) and the expected bottlenecks or added value of these.

Depending on the goal(s), it can be interesting to involve external parties. For example, when you want to investigate whether the level and quality of the graduation work of your own students is comparable to the graduation work of students from similar programmes in the Netherlands or internationally. And whether you give comparable grades for comparable quality.  
An external party without familiarity with the student population, educational context, certain traditions, and customs, looks at the final works more objectively. Externals with specific expertise can be involved to further examine certain areas of concern. For example: is the methodological part of the research that students conduct in order and up to standard?

How can you proceed?
The involvement of an external party can be on the basis of mutual exchange. This does not necessarily have to happen, but promotes working with “closed exchanges”. On both sides, for example, a few faculty members with sufficient knowledge of the discipline and with experience in grading theses are asked to look at a few theses (varying in quality and grades) and give an assessment of them using a form.
A separate form can be prepared for the exchange so that both parties assess based on equal and pre-agreed assessment criteria. Or the assessment form used at the particular programme can be used. Assessment forms can then also be discussed afterward.
The assessments are subsequently compared and discussed at a meeting. The process and results are documented and provide input for possible improvement measures. Good documentation is important for demonstrating measures taken in the context of quality care and assurance.

Variants:

• Even without looking at specific theses, it may be interesting to check with a similar programme elsewhere to see how they design the thesis process and assess the work delivered. This can be done through desk research based on requested or publicly available information or by organizing an (online) meeting with one or more external parties.  
• A calibration session can be organized by taking the same thesis (or 2 or 3) and having both parties review it. With or without using their own assessment tools. Are their different judgments for the same thesis?  What causes this?
• A single expert external assessor (familiar with the discipline and experienced in assessing theses) may be asked to assess a number of theses.

Purpose: To verify that teachers interpret criteria and expected standards similarly. Calibrating assessments by learning from each other. Possibly also: to ascertain the extent to which the assessment tools are satisfactory or where improvement is needed.

In advance: One or more (2, 3) theses are chosen and all reviewers are asked in advance to complete the assessment form (possibly with additional questions) for the same anonymized thesis(s). Items such as “process” and “presentation” can be omitted.
The judgments can be analyzed in advance to bring out some points of interest for the session. Another option is to omit analysis beforehand and ask employees to bring the form with them to discuss on the spot. 

During session: The assessments are discussed (in groups). Participants explain to each other how they arrived at the assessment. Perspectives are compared and differences are discussed on the basis of a number of presented questions.
Findings are recorded using, for example, the questionnaire (or a PPT presentation, flip chart sheet, etc.).  
At the end of the session, the main findings are reviewed and whether they warrant action. A measure does not always imply a concrete decision. More research may be needed first. 

Variant: Discussion in a “world cafe setup. At different tables with a moderator, specific concerns are discussed. Participants visit the different tables to give their input. Previous input is built upon.

After the session: A report is made and and actions are determined.

Variants: This calibration can also be well organized in a smaller format, for example for a chair group. This is particularly useful if new reviewers have joined.

Purpose: To ascertain inter-rater reliability. Are equal grades given by different evaluators? To ascertain whether teachers interpret criteria and expected standards similarly. Calibrating assessments by learning from each other. Possibly: assessing the extent to which assessment tools are adequate or where improvement is needed.

In advance: A sample of theses will be identified. Varying in quality (moderate, good, excellent), coming from different chair groups, possibly with further variations, e.g. internal versus external assignments. 
For the selected theses, two lecturers per thesis (e.g. one belonging to the same chair group as the original assessors and one from a different chair group) are asked to independently assess the particular thesis and to fill out the usual forms for this purpose. Possibly with additional questions, such as: how do you estimate the complexity of the assignment? 
A set moment (half a day) can be scheduled for completion and a space for everyone to work independently. This can help teachers schedule time for this action. It can also be left to those involved and only a deadline is given.   
After the “reassessments”: The completed forms are compared with those completed by the original assessment committee. As far as possible, the comparison is made at the criteria level. Consideration is given to:
> Differences between the re-assessors themselves. In terms of overall assessment and at the criteria level.
> Differences between the re-assessors and the original assessment committee. In terms of overall assessment and criteria level.

Discussing results: Several variants are conceivable for this. Among them:

• An independent interviewer/conversation leader speaks with the assessors in the case of differences greater than a set standard (e.g., 1 grade point) to explore what explanations may be for the differences. Interviews may be done with the evaluators individually or with the re-evaluators plus the original evaluators together. Findings are noted and all findings together are recorded in a final report for the client (program management and/or examination board).  The findings can form input for a later session with those involved in the graduation process. A session - for example during an annual 'study day' for all teachers - that focuses specifically on identified points of attention and points for improvement.
• A session is organized in which all original evaluators and re-evaluators discuss their judgments in groups. Using a question form, they record the findings for each group. The findings at a somewhat higher level of abstraction, not specific to a particular thesis, but for example, “Group x: We noticed that the process is very decisive for the final grade,” are discussed in plenary.  
  Afterwards, a report presents the main findings.