Course: Analysis of Measurement Instruments (Glas and Fox)

Analysis of Measurement Instruments: Introduction to classical test theory, item response models and multilevel item response models

Target: The course provides an introduction on classical and modern test theory and its major applications, such as the construction of measurement instruments, linking and equating measurements, evaluation differential item functioning, optimal test construction, computerized adaptive testing and the evaluation of large-scale dicational surveys such as PISA and TIMSS. The introduction is broad and on a conceptual level, but references will be provided to the more technical aspacts such as estimation and testing of the models.

The second part of the course will focus on Bayesian item response modeling. An introduction will be given to the important features of the Bayesian modeling approach, which will be illustrated with real data examples using software packages WinBUGS and R. The practicals will focus on standard software, but interested students will also be given the opportunity to get acquinted with Baysian modeling software.

Tutors:

Prof. Dr. Cees Glas (http://www.utwente.nl/gw/omd/afdeling/Glas/)

Dr. Jean-Paul Fox, (www.Jean-PaulFox.com)

Maximum number of participants: 25

Date and Venue: November 15th – 18th 2010, University of Twente, Enschede, Netherlands.

Prior Requirements:

Knowledge of statistics and methodology in the field of the behavioral and social sciences

Literature:

J. Scheerens, S. Thomas, & C.A.W. Glas. Educational evaluation and monitoring: A systemic approach. Swets and Zeitlinger Publishers.

Fox, J.-P. (2010). Bayesian Item Response Modeling: Theory and Applications. Springer, New York.

Work method

The course takes 4 full days. Every session consists of a lecture of approximately 2.5 hours and a computer workshop of approximately 3 hours.

For your registration, please use the registration form below. Participants will be accepted by order of application date, the number of participants is limited to 25 persons.

Participants are required to bring their own laptop with Windows (XP or higher) as operating system (also Mac with Windows via Boot Camp). The programs R and WinBugs have to be installed. Instructions for the installation of both packages will be send. Programs from the tutors will be given during the course.

Accommodation: There are various hotels in Enschede, some are located near the University of Twente (we will send information after registration). Seminar fee and payment: The participation fee is 150,00 € for non-IOPS students. Reductions or refunds are not possible. Travel costs, accommodation and meals are not included. Please make your own arrangements.

Contents

In general, the course provides an overview of classical and modern test theory discusses the coherence of the various theories and deals with the application of test theory for developing and evaluating measurement instruments for the social sciences.

Lecture 1	Introduction
Subjects	Overview of classical test theory and item response theory, the relation between the two approaches and some important applications such as evaluation of reliability and validity, item analysis, test equating, and optimal test construction
Workshop 1	Reliability Analysis and Test equating using the MIRT package.
Literature	Basic elements of educational measurement, Measurement models in assessment and evaluation, Applications of measurement models, Chapters 6, 7 and 8 from J. Scheerens, S. Thomas, & C.A.W. Glas. Educational evaluation and monitoring: A systemic approach. Swets and Zeitlinger Publishers.

Lecture 2	Item Response Theory
Subjects	Introduction to maximum likelihood estimation and testing methods. Applications of IRT: adaptive testing, analysis of item bias (differential item functioning, DIF) and handeling of missing data.
Workshop 2	DIF analysis using the MIRT package.
Literature	As for Lecture 1.

Lecture 3	Introduction to Bayesian Item Response Modeling
Subjects	Introduction to Bayesian item response modeling with applications: the testlet model, models with mutiple raters, models for missing data, among other things.
Workshop 3	MIRT package and Bayesian software using Winbugs and R-packages.
Literature	Fox, J.-P. (2010). Bayesian Item Response Modeling: Theory and applications. Springer, New York.

Lecture 4	Multilevel IRT modeling
Subjects	Introduction to general multilevel models and multilevel IRT models, working with plausible values.
Practice 4	MIRT package and Bayesian software using Winbugs and R-packages.
Literature	As for Lecture 3.