See Postgraduate and professionals

Research Evaluation 2021

research evaluation 28 - 29 - 30 June 2021
3 half day online courses 
09:00 - 13:00 CEST timezone

3 consecutive half day online courses on new developments in research evaluations in context for Research Evaluators and Government Officials on 28 - 29 - 30 June 2021

The topic

Research evaluations often take a standardized approach. With research policies linked to societal challenges and economic goals, new approaches in research evaluation have developed.
How can they help you in improving your research policy and make it more evidence-based?
Think of evaluations of policies that fund research to strengthen regional economic development.
Or peer reviews of international research programs in medical science. How to evaluate a research institute for transitions in agriculture? In such evaluations, questions of research quality and research management are connected to issues of impacts, effectiveness, policy logic, implementation and policy legitimacy.

The objective

Explore the new developments in research evaluation beyond the standardized forms.
Using real-life cases, we bring together experts who work at the frontier of research policy, management, and evaluation theory & practice.

Course 1 - June 28: Designing research evaluations for policy-making
In course 1 we start off with a case related to a program for international collaboration in medical science. How to develop an evaluation that really is useful for the future of the program? We discuss the use of logic charts and program logics to connect policy objectives, research activities and outcomes - and discuss the limits of the approach

In the second part of course 1, we move to situations in which research is embedded in a mix of policies and the relations between policies, research efforts, and impacts are not straightforward. In such cases, the design of the evaluation often is part of the evaluation itself and a first contribution to the policy process. We also introduce the idea of ex ante evaluations, at the start of programs, that help to develop better policies and facilitate the policy use of future research evaluations.

Course 2 - June 29: New evaluation methods for research organizations and programs
Outcomes of research evaluations are increasingly used as evidence for policy makers and managements. Because of this use, there is a clear need for appropriate and responsible research evaluation methods. In Course 2 we explore how these new expectations affect the more regular methods of bibliometrics and peer review, and what are the possibilities for evaluating social impacts.

In three lectures, experts show the limits of mainstream evaluation methods, and discuss ways forward. - Bibliometrics have a key role in research evaluation. Yet they are criticized for their effect on research management and research careers. Can they be useful in more advanced research evaluations? - Methods of peer review are well known in the evaluation of science. But how to manage peer panels for policy-oriented research evaluations? - While claims about economic and societal relevance of science have always been made, the need to evaluate the impact is a more recent phenomenon. What methods are available as impact of scientific research are essentially uncertain?

Course 3 - June 30: Evaluating research for system interventions and social transformation
In Course 3 we discuss the evaluation of institutes, programs, policies with a mission to innovate systems. Engineering universities, agricultural research institutes, programs for public health: their aim is to strengthen regional innovation systems, push sustainable transitions in agriculture, and contribute to meeting SDGs locally, nationally and globally. How can we evaluate the impact of research on such overarching system goals.

In three lectures experts in research management and system evaluations present and discuss approaches for evaluating system effects of research programs and organizations, and their use for policy making. Can institute evaluations help to strengthen research impacts on sustainable transitions? How to capture economic impacts of research at regional level? How to asses the impact of research funding organizations on the research system? Questions beyond the usual forms of research evaluations on which our experts like to share their answers with you.

Course faculty

Erik Arnold, Technopolis Group & Royal Institute of technology (KTH)

Kate Barker, University of Manchester

Stefan Kuhlmann, University of Twente

Thed van Leeuwen, Leiden University

Mireille Matt, INRA National Institute of Agronomic Research

Philip Shapira, University of Manchester & Georgia Tech

Barend van der Meulen & Gonzalo Ordóñez-Matamoros, course-directors, University of Twente