It looks like a ranking, and it is often read that way. Yet that picture is wrong. For a start, not every country takes part in every round, so a headline like ‘we dropped from tenth to twelfth place’ compares two different line-ups. On top of that, countries sometimes score so closely together that a few places’ difference falls within the margin of error.
The Dutch national report itself is careful, and serious media picked up that nuance. When the last round came out, the NOS ran the headline that reading skills had ‘declined further’, with the caveat that countries are hard to compare. In the retelling, one figure did the rounds above all. The Netherlands had slipped from 26th to 34th place for reading, and that figure travels more easily than the nuance around it.
Martina Meelissen is the project leader of the Dutch PISA study at the University of Twente. She says the report sets out exactly what you may and may not read into the figures. “A number on a global ranking is not one of those things,” she says.
How do you measure the same thing in dozens of countries?
If it is not a ranking, what is it? PISA tries to measure the same skills in the same way, across dozens of countries and languages at once. That is a tricky measurement problem, and it is exactly what the UT specialises in for the Netherlands. Its education researchers have been working on studies like this since the 1970s.
Careful measurement starts with the sample. PISA first picks schools at random, and pupils within them at random, so that no country can push its best pupils forward and the results stay representative. Specially trained test administrators run the test. An international monitor visits a sample of schools in every country to check that the test is run the same way everywhere.
Once the answers are in, they are marked. “Strictly speaking, we shouldn’t call it that,” says Jolien Valk, a researcher at the Cognition, Data & Education group (CoDE). “In the end, PISA decides whether an answer is right or wrong.” Each answer gets a code from a fixed list, so the same answer counts exactly the same in the Netherlands as in Japan. UT student assistants mark the tricky open questions by hand. Our physics and maths students spot the nuance a computer misses.
The value is in the trend over the years
A small rise or fall in performance compared with the previous round says little. Only across several rounds can you see whether a drop was chance or a real pattern. Meelissen has coordinated the related TIMSS study for more than twenty years, and prefers to look at the longer-term trend. Sometimes, though, a drop is so large, as with reading in PISA-2018 and -2022, that it does need to be taken seriously.
That kind of trend is what makes comparison with other countries useful. If a country, say England, climbs steadily across several rounds, it is worth looking at how they organise their schools there. That is what studies like PISA are really for.
More than reading and maths
A place on a list, by the way, shows only a fraction of what PISA measures. Alongside the tests, pupils fill in extensive questionnaires about their school and themselves. In PISA-2025, for instance, about living and learning in a digital world. Jolien and her colleague Nathalie Maassen looked into the well-being of Dutch fifteen-year-olds. How do they feel at school, and is that linked to their performance? “We saw, for example, that boys are more satisfied with their lives than girls. Boys also feel more that they belong at school.”
Those figures also steer policy. Recently the state secretary put language teaching ahead of maths, partly because of the disappointing reading results. That is exactly why it is worth knowing how carefully those figures are put together. So the next time PISA comes around, you can do something other than scan the ranking. Look at the line across the years, and at what sits beneath the figures. That is where the real story is.
