UTFacultiesBMSResearchBMS Research Support#7: Capturing and evaluating impact

#7: Capturing and evaluating impact

Let’s Talk About Impact is a series that explores various aspects of making impact. We strive to make a change for the better, to make an impact, but what lies beyond this buzzword? Today we will talk about capturing and evaluating impact. Of course, we cannot talk about 0.3 or 0.7 impact (or 2.3, or 100), but then how can we “measure” impact?

Let’s get this out of the way: the impact indicator does not exist. There are multiple reasons to make this bold claim: talking about impact varies per discipline, is time-bound, there can be unexpected impacts, and impact is such a broad term that one indicator would not cover that breadth.

Indicators

A key word when talking about measuring impact is “context-sensitive”. A one-size-fits-all measurement does not exist, but indicators may work within a shared discipline, or a shared type of project. Impact indicators should be aligned with the goal of a project. For instance: a health intervention may use “Quality-adjusted life year (QALY)” as an indicator for impact, whereas an educational innovation’s impact may lie in the number of schools that have adopted this innovation and used it for 5 years. These indicators both go beyond just outcome, so how many people used something, how many citations, how much money, but really try to capture the change that took place. To capture the impact.

The paragraph above presents a story about societal impact, but what about academic impact? Citations are limited, as they don’t capture if your work made a lasting change. They give an insight in the reach of your work, but not the significance. Efforts to include this relevance and field-specificness have led to new indicators like the Field-Weighted Citation Index (FWCI). The FWCI tries to place a citation in context, by showing its relevance in the field. This is a step in the right direction, but is weak on representing the thought of lasting change. Also when talking about academic impact one can reflect on the sustained effect of their work. Has your theory or model been adopted by relevant peers? Do they keep using it? Did you change a certain standard?

Tool: Impact Indicator Inspiration Framework

A working group of Strategy and Policy, ITC, BMS, SBD and TechMed colleagues developed the Impact Indicator Inspiration Framework. This is a list of indicator categories that you can use to underpin an impact pathway. The tool hopefully gives some inspiration for you to find metrics that fit your work. You will notice how input, activity, output and outcome indicators are offered, not impact indicators. This is because a UT-level tool would inherently be too generic to properly fit the work of all disciplines. BMS Research Support can help with that last step.

Again, please see this as a tool, not a checkbox exercise.

Narrative

One can notice how the impact measurement discussion is moving away from rigid indicators (e.g., the h-index is not preferred in research evaluations anymore), and moving towards a more narrative-based approach. You may have heard of a “narrative CV”, or of an Impact Case. At BMS there is increased attention to the latter. An impact case tries to capture the impact of your work, project, research line, in a well-underpinned narrative. As such, you can provide context while also providing pieces of evidence. The latter is important, a narrative still needs to be well-underpinned and explicit.

At BMS we offer Impact Case workshops. You will walk out the door with your draft impact case in your hand! New workshop moments are planned in quartile 3 on:

I would like to join an Impact Case workshop
Register now!

This shift from only metrics to narrative is also seen in research evaluations. Curious how we approached this at BMS? Have a look.

Evaluating BMS impact in the midterm research evaluation

At BMS we have ran multiple Impact Case Workshops, where colleagues drafted their own case. We used these cases in the midterm research evaluation. In the self-evaluation report we tried to evaluate our impact using a mix of numerical indicators (output, output/staff ratio, FWCI) and impact cases. We then placed theses cases in context, describing how they were “typical for BMS”. We have yet to receive the final committee report, so we are curious how they view this type of impact evaluation.

Colophon

This series is composed by BMS Research Support. While most content applies to a wider array of disciplines, some may resonate more with a BMS audience. If you work at BMS and have any impact-related questions, reach out to Tom Boogerd. If you work at another faculty, you can still reach out and we can find a colleague of your faculty who can help.

This news item is also available as a page on our website.