1 The testing and assessment system
Over the years we have at the Electrical Engineering department developed an assessment system that guarantees that tests and judgements of the tests give a reliable indication of the learning achievements of students. Students have a clear picture of the requirements of the tests and the Board of Examiners is able to guarantee the quality and consistency of the exams.
After briefly introducing our overall vision on assessment and the course file (‘vakdossier’), we will describe how the assessment policy of our EE study is shaped by the various stages of the so-called Deming cycle (Plan-Do-Check-Act).
1.2 Vision on assessment
Central to our vision is that a test in first instance is an assessment tool for the study of EE, meant to assess the knowledge, insight and skills of a student. This means that summative tests play an important role. Since a test is also an effective means to influence student behaviour, also formative tests play a part in our testing. Especially in the recently introduced educational innovation called the Twente Education Model (TOM) we have increased the role of formative tests. Marking is by summative marks (typically 1-10), performance marks (e.g. for labs; typically 6-10), and diagnostic marks (1-10). It is possible that a final grade for an educational unit can be composed of a summative mark and a performance mark, whereby the summative figure should be equal to or above a 6. The idea is hereby that a summative mark can be improved by a performance mark, but that an insufficient summative mark cannot be compensated by a performance mark.
1.3 Course file
The main instrument for quality assurance is the course file, which is maintained by the lecturer on each course or the module coordinator on each module. This file contains information that is accessible also for students (learning objectives, course description, organization of teaching and assessment) as well as information that is not accessible for students, in particular a logbook on activities in the course file, test scheme, test assignments and peer review thereof, answer models and marking rules, all evaluation results, and the reflection of the lecturer on test results and evaluation results. The program director has access to all course files, monitors the activities, and reports his findings to the Board of Examiners, which, if necessary, takes action.
2 Test preparation and development (PLAN)
For each module in the bachelor’s curriculum the lecturer(s) prepare a test scheme, in which the learning objectives, the weighing factors and the assessment are described. Each lecturer is expected to submit the test as well as its answer model to a fellow lecturer with knowledge on the subject, who inspects the test on content, transparency and validity. The final versions of the test schemes are established on the authority of the Board of Examiners. The test scheme forms the basis for the development of the annual testing and has to be re-entered if circumstances thus redesign of a module require (e.g. at redesign of a module).
2.1 Implementation of the test (DO)
Osiris (the student information system) and Blackboard (the electronic learning environment) provide information to the students about the learning objectives for the module/course, the test methods used, the test evaluation and the manner in which the final grade is established. It is expected that the lecturers provide representative trial tests and practice assignments, including marking rules, to the students. The written test provides test instructions in which at least the available time and the number of points per assignment are mentioned. Assessment of written tests is made on the basis of a test scheme and answer model. When other types of assessment are used, a scoring model is established on the basis of evaluation criteria derived from the learning objectives for the course. In this case it is clearly documented how the mark is arrived at. The establishment of the final mark is also described in the test scheme. In the education a number of subjects exists where students work in groups on assignments. The assessment in these subjects may be composed from various components such as a presentation, oral examination, a final report and the course of the process. For these subjects a manual is available listing the assessment criteria of the various components and indicating for which components the student will get an individual or a group assessment. In principle the students are assessed individually. In components where a group assessment takes place, the supervisor monitors whether each individual student has made a full contribution to the process. If this is not the case, then the student will receive an individual mark on the basis of an oral examination. Examiner-supervised peer review amongst group members is also used to arrive at individual marks. Finally, rules for the various types of tests can be found in the R&R (Rules and Regulations) of the Board of Examiners.
2.2 Evaluation and analysis of the test (CHECK)
By the examiner
The examiner analyzes the test results, thereby considering average and distribution of marks, performance on separate test parts and any other conspicuous details, and considers an adjustment of the assessment. Specific problems for a test, especially if they lead to an adjustment of the assessment by the examiner, are submitted by the examiner to the Board of Examiners. The examiner himself notes points of improvement for the assessment for a subsequent educational year, and takes these up in the course file.
In the Bachelor each module is annually evaluated through a standard questionnaire. In these completion of the course surveys the students also evaluate the quality of testing, considering transparency of requirements, validity/representativeness, clarity of questions and presence of sufficient exercise material.
By the student association
The student association organizes meetings for the students of every module discussing the quality of education including testing. Feedback is given to the lecturers, the education quality committee, the education committee and the director of education.
By the education quality committee
A comprehensive evaluation of a module takes place at least once every five years (or more frequently if circumstances require), by the education quality committee every five years (‘onderwijskwaliteitscommissie’, OKC), which resorts under the Board of Examiners. These evaluations involve multiple meetings of a committee member with the students as well as a course lecturer. The OKC reports to the lecturer(s), who provide a written reaction and, if needed, propose measures for improvement. The results of the (extensive) assessment plus reaction are discussed in the Board of Examiners and the Education Committee (Opleidingscommissie, OLC) and are included in the course file (‘vakdossier’) (see Section 1.3). If needed, the Board of Examiners meet the responsible lecturer.
Other quality control measures
Besides the above mentioned, students always have the right of inspection, which implies the right to visit the examiner to discuss the results of the test and to see what the outcome is based on. If the student and the examiner do not reach mutual agreement, the student can deposit his/her concern or complaint at the Board of Examiners, which then rules on the case.
2.3 Improvement or modification of the test (ACT)
Based on the evaluation results described in the Section 2.2 (‘CHECK’), the lecturer determines whether testing needs improvement and if needed, how. The lecturer includes the planned improvements in the course file and in this manner reports them back to the Board of Examiners and the Education Committee (OLC). In case of substantial changes a request is submitted to the Board of Examiners. Educational experts are available when the lecturer wishes to receive advice or support to improve the assessment. Additionally the lecturer is free to approach the program director for advice. Opportunities for formal training or courses in the area of testing and assessment are provided by the Educational Service of the UT.
3.3 Bachelor assignments
The assessment of the bachelor projects is in the hands of the Graduation Committee whose composition is described in the ‘OER’, the Rules of Examinations and Assessment for the BSc UT and in the EE specific version of this document . The Graduation Committee for the assessment uses a list of key criteria, which aims to harmonize the assessment within EE, and serves as a guideline for lecturers and students during the project. The marking is based on scientific quality (35%), organizational skills (35% ) and communication including reporting and presentation (35%). To arrive at the marks the graduation committee discusses each component mentioned above separately and then determines the final grade. The written motivations of the mark are briefly noted down and stored in DECOS. The final presentation takes place for a general public, so that the academic community can take note of the level. To ensure the assessment quality, a member of the Board of Examiners regularly inspects the written motivations for the marks. If needed, the graduation lecturer is contacted.