THE TOPIC

Research evaluation occurs regularly. Governmental programs have ex-ante, midterm and final evaluations. Institutes, centres and people are evaluated. University research groups are assessed.

Research evaluation is important in terms of accountability, quality control, strategic decision making, and for identifying strengths, weaknesses, opportunities and unintended consequences of research. Research evaluation has also become important in assessing the appropriateness of research policy measures, and for identifying systems failures.

The challenge to professional evaluators, and to government and other officials responsible for research evaluation, is to design and conduct tailor-made evaluation exercises that serve policy needs and strategic aims. The course builds on the methods and experiences in various research evaluation practices to show that research evaluation can be done systematically. It offers a catalogue of research evaluation methods, techniques and rationales, as a firm base for designing, assessing and envisioning its multiple uses in policymaking.

Course objectives

The course includes lectures by experts in research evaluation, study of case material
in small groups and international exchange of experiences. The aim is to provide:

  • introduction and overview of research evaluation frameworks, models and approaches
  • in-depth discussion of key issues and methods,
  • systematic approaches to the design of research evaluation,
  • insight in the use of research evaluation as a strategic policy tool.

WHO SHOULD ATTEND?

  • Government officials who commission research evaluation studies, or are required to interpret the results of such studies
  • Actual or potential research evaluation professionals who want to improve their understanding and skills.

course material

The course language is English.
Please bring your own laptop.
Course material includes the lecture notes, background publications and the book Learning from science and technology policy evaluation. Cases will be studied with the help of selected original data and documents. Data and documents from actual evaluation studies will be used to work in groups on design and methods of research evaluation

The international team of teachers provides access to state-of-the-art research evaluation in Britain, United States, Scandinavia, Germany, the Netherlands, and of European programs.

COURSE ITEMS

  • Design of research evaluation, selection of methods, in relation to the policy problem and general context.
  • Output identification and measurement: methods and problems. Scientific output and technological effects, economic and non-economic effects.
  • Policy contexts of evaluation. Implementation of evaluation results and policy use of evaluation.
  • Experiences with evaluation. Evaluation of universities; evaluation of institutes; evaluation of research programs.
  • Development of research evaluation and corresponding changes in national and international research systems.

COURSE EXERCISES

Data and documents from actual evaluation studies will be used to work in small groups on design and methods of research evaluation.

The groups will apply what they have learned in the lecture sessions, and will report their results, and receive comments from the lecturers.

The aim is to provide hands-on experience: this is training in methods, and a way of letting participants recognise the difficulties and pitfalls of research evaluation and stimulate them to reflect on it.

course program

Monday, 12 November

  • Evolution of research evaluation
  • Design of research evaluation, selection of methods in relation to the policy problem and general context
  • Exercise I on evaluation design: logic chart

Tuesday, 13 November

  • Output identification and measurement: methods and problems. Scientific output and technological effects, economic and social impacts
  • Exercise II on evaluation of institutes and university research

Wednesday, 14 November

  • Evaluation of policy tools and systems
  • Ex-ante evaluation
  • Policy uses of evaluation: how to make an evaluation effective
  • Exercise III on communicating evaluation results and policymaking
  • Course dinner

Thursday, 15 November

  • Presentation exercise III
  • Future of research evaluation

Informal interaction with the teachers and among participants with different backgrounds during meals, etc. provides an important learning experience.