Automated Multiple Choice exams: reports and analysis by ConTest

Automated Multiple Choice exams: reports and analysis

The University of Twente offers a service for automated processing and grading of multiple choice exams. Students fill out their answers on scannable forms, and lecturers receive automatically calculated grades and reports. This document contains a brief overview of the available reports.

The scanning service is located in Spiegel 414. Please see the TELT website for a description of the service, as well as all the forms you will need to use.

REPORTS AND ANALYSIS

The scanning service uses the ConTest application. It can deliver several reports, which are explained below.

1.

The answers for each student in CSV format (testdata.csv)

This file contains the raw data. The second row contains the correct answer (participant nr, 0). These data can be used to run your own analysis in Excel or SPSS. Please see an example below, or download the sample csv file.

Participant

Received

1

2

3

4

5

6

7

21

22

23

0

1-1-2017

A

B

C

C

D

B

C

 

A

C

B

0123456

1-1-2017

D

D

C

D

D

B

C

A

A

A

0234567

1-1-2017

A

B

A

D

D

B

C

 

A

C

A

2.

The grades per student as calculated by ConTest, in Excel format (ReportBook.xls)

This file contains the grades as calculated by ConTest. Below is a small example, and you can also download the sample xls file.

 

 

 

 

 Score

Grade

 

Participant

Name

Open

 

 

 

Bonus

 

Tot.

 

Bonus

Tot.

2345678

Jansen, A.

0

 

15,00

 

0,00

 

15,00

5,8

0,0

5,8

2678901

Pietersen, B. 

0

 

17,00

 

0,00

 

17,00

6,9

0,0

6,9

2456789

Li, C

1

 

11,00

 

0,00

 

11,00

3,7

0,0

3,7

The formula used to calculate the grade is a commonly used formula for multiple choice exams with four possible answers. Please contact your faculty education advisor if you have questions about this, or about other calculation methods (for example, more complex exams that combine multiple choice questions and open questions).

3.

Exam analysis in PDF format

This report is also available as a sample PDF. It contains the following pages:

·

Meta data report. This report contains the score to grade transformation.

·

Test analysis report. Statistical details for the entire test, including Crohnbach’s Alpha, P values, etc.

·

Score frequency report

·

Item analysis report. P values and Rit values for each question.

·

Item advice report. The exam questions plotted on a graph by P x Rir values, and a report on the quality of each question (see image below).

·

Grading report. The same information that is contained in the Excel GradeBook (grades per student).

General information about test analysis and psychometric data analysis can be found on this website at https://www.utwente.nl/ces/toetsing/Docenten/.

Below is an example of the item advice report. It gives an indication of the quality of the questions (e.g. too easy, another alternative may have been plausible, etc.).

If the lecturers decide that a certain exam question contained an error and needs to be excluded from the exam, the exam can be recalculated and new reports can be generated.

4.

A PDF containing the scans of all student forms

This PDF can be useful for archiving and/or student review. One page is included as an example in this sample PDF (the actual file is quite large, as it contains one page per student).

MORE INFORMATION

If you’d like to use this service, please see the TELT website for instructions.

If you have any questions about the reports, please contact your faculty education advisor.