1 The testing and assessment system
Over the years we have at the Electrical Engineering department developed an assessment system that guarantees that tests and judgements of the tests give a reliable indication of the learning achievements of students. Students have a clear picture of the requirements of the tests and the Board of Examiners is able to guarantee the quality and consistency of the exams. After briefly introducing our overall vision on assessment and the course file (‘vakdossier’), we will describe how the assessment policy of our EE study is shaped by the various stages of the so-called Deming cycle (Plan-Do-Check-Act).
1.2 Vision on assessment
Central to our vision is that a test in first instance is an assessment tool for the study of EE, meant to assess the knowledge, insight and skills of a student.
1.3 Course file
The main instrument for quality assurance is the course file, which is maintained by the lecturer on each course or the module coordinator on each module. This file contains information that is accessible also for students (learning objectives, course description, organization of teaching and assessment) as well as information that is not accessible for students, in particular a logbook on activities in the course file, test scheme, test assignments and peer review thereof, answer models and marking rules, all evaluation results, and the reflection of the lecturer on test results and evaluation results. The program director has access to all course files, monitors the activities, and reports his findings to the Board of Examiners, which, if necessary, takes action.
2 Test preparation and development (PLAN)
For each course in the master’s curriculum the lecturer(s) prepare a test scheme, in which the learning objectives, the weighing factors and the assessment are described. Each lecturer is expected to submit the test as well as its answer model to a fellow lecturer with knowledge on the subject, who inspects the test on content, transparency and validity. The final versions of the test schemes are established on the authority of the Board of Examiners. The test scheme forms the basis for the development of the annual testing and has to be re-entered if circumstances thus require (e.g. at redesign of a module).
2.1 Implementation of the test (DO)
Osiris (the student information system) and Blackboard (the electronic learning environment) provide information to the students about the learning objectives for the module/course, the test methods used, the test evaluation and the manner in which the final grade is established. It is expected that the lecturers provide representative trial tests and practice assignments, including marking rules, to the students. The written test provides test instructions in which at least the available time and the number of points per assignment are mentioned. Assessment of written tests is made on the basis of a test scheme and answer model. When other types of assessment are used, a scoring model is established on the basis of evaluation criteria derived from the learning objectives for the course. In this case it is clearly documented how the mark is arrived at. The establishment of the final mark is also described in the test scheme. In the education a number of subjects exists where students work in groups on assignments. The assessment in these subjects may be composed from various components such as a presentation, oral examination, a final report and the course of the process. For these subjects a manual is available listing the assessment criteria of the various components and indicating for which components the student will get an individual or a group assessment. In principle the students are assessed individually. In components where a group assessment takes place, the supervisor monitors whether each individual student has made a full contribution to the process. If this is not the case, then the student will receive an individual mark on the basis of an oral examination. Examiner-supervised peer review amongst group members is also used to arrive at individual marks. Finally, rules for the various types of tests can be found in the R&R (Rules and Regulations) of the Board of Examiners.
2.2 Evaluation and analysis of the test (CHECK)
By the examiner
The examiner analyzes the test results, thereby considering average and distribution of marks, performance on separate test parts and any other conspicuous details, and considers an adjustment of the assessment. Specific problems for a test, especially if they lead to an adjustment of the assessment by the examiner, are submitted by the examiner to the Board of Examiners. The examiner himself notes points of improvement for the assessment for a subsequent educational year, and takes these up in the course file.
Questionnaires, panel discussions and exit interviews
Evaluation occurs by means of a questionnaire six months after graduation, a panel discussion with selected students after 12 months, a survey after 18 months and an exit interview after 24 months. In each of these occasions problems can be identified problems with testing in certain courses, in which case the OKC will conduct a course evaluation.
Other quality control measures
Besides the above mentioned, students always have the right of inspection, which implies the right to visit the examiner to discuss the results of the test and to see what the outcome is based on. If the student and the examiner do not reach mutual agreement, the student can deposit his/her concern or complaint at the Board of Examiners, which then rules on the case.
2.3 Improvement or modification of the test (ACT)
Based on the evaluation results described in the Section 2.2 (‘CHECK’), the lecturer determines whether testing needs improvement and if needed, how. The lecturer includes the planned improvements in the course file and in this manner reports them back to the Board of Examiners and the Education Committee (OLC). In case of substantial changes a request is submitted to the Board of Examiners. Educational experts are available when the lecturer wishes to receive advice or support to improve the assessment. Additionally the lecturer is free to approach the program director for advice. Opportunities for formal training or courses in the area of testing and assessment are provided by the Educational Service of the UT.
3 Master assignments
The assessment of the master projects is in the hands of the Graduation Committee whose composition is described in the OER, the Rules of Examinations and Assessment for the MSc EE. The Graduation Committee for the assessment uses a list of key criteria, which aims to harmonize the assessment within EE, and serves as a guideline for lecturers and students during the project. The marking is based on scientific quality (50%), organizational skills (20%) and communication including reporting and presentation (30%). To arrive at the marks the graduation committee discusses each component mentioned above separately and then determines the final grade. The written motivations of the mark are briefly noted down and stored in DECOS. The final presentation takes place for a general public, so that the academic community can take note of the level. o ensure the assessment quality, a member of the Board of Examiners regularly inspects the written motivations for the marks. If needed, the graduation lecturer is contacted. To ensure the quality and consistency of the assessment, one member of the Graduation Committee is also a member of the permanent academic staff from a different chair than the one where the student graduates.
The evaluation of the internship is based on three components
- an assessment of the written report via a list of criteria derived from the learning objectives of the internship assignment
- an individual interview with the UT supervisor where possible ambiguities or doubts rising from the report can be discussed
- the assessment of the external supervisor, who is asked to give an opinion on the quality of the contribution of the student on a six-point scale with clearly defined criteria including self-motivation and substantive depth
After consultation with the external supervisor the internal supervisor determines the final grade using an assessment form, which is stored in DECOS.