Learning data is a key component for informing national policies and global dialogue in education. With that in mind, a new World Bank paper A Global Data Set on Education Quality (1965‑2015) by Nadir Altinok, Noam Angrist, and Harry Patrinos compiles data of 163 countries and regions from 1965 to 2015 – an impressive achievement.
This work illustrates some important possibilities for measuring Sustainable Development Goal (SDG) 4.1 (to ensure that all girls and boys complete free, equitable and quality primary and secondary education leading to relevant and effective learning outcomes), particularly for developing countries that have participated in regional or international assessments. The work, however, also reveals some limitations in available data.
New methodology for linking regional and international assessments
The authors demonstrate an innovative approach to compiling a "global data set" on student learning, by linking international and regional assessments. They demonstrate that it is possible to derive common measures of student learning while retaining a flexibility for education systems to choose assessment programs that suit their context. This flexibility requires investment and innovation in developing sophisticated, fit-for-purpose linking approaches. The rigorous body of work on which the paper is based is a cogent example of how it is possible to monitor SDG 4.1 for many countries by building on existing learning assessments and using linking approaches.
The limitations arise from what is missing from the paper. The main outputs from the analysis are comparisons of results between education systems, including between developing and developed countries. The finding that even the highest-performing developing countries are performing below the developed world is unsurprising in itself. Of course, by quantifying these differences it provides compelling information on the scale of the learning crisis that some countries are facing. However, it leaves unanswered the more important and complex question of what can be done about this.
The paper does not engage with the different ways in which learning in the relevant domains is described in the various assessment programs under analysis, or what these differences might mean for understanding the results. Instead, it relies on certain tests as reference points, which risks orienting the view of learning towards what is operationalized in those tests, rather than piecing together a global view of learning, based on its operationalization around the world. To inform system improvement, the outcomes of learning assessment must be more than numbers, but must also provide meaningful descriptions of learners' proficiency in the relevant domains.
Moving from comparing assessment results to understanding them
To this end, the UNESCO Institute of Statistics Reporting Scales (UIS RS), which are being prepared under the auspices of the Global Alliance to Monitor Learning, focus on providing a shared global view of learning progress in reading and mathematics. The UIS RS are not new assessments, but empirically-developed reference points against which other learning assessments can be aligned. As more assessment programs undergo alignment to the UIS RS, a global conversation about how students progress in their learning becomes possible, moving from comparison of results to understanding of their meaning. The UIS RS have many possible uses in informing educational improvement, and guiding systems to support their learners to reach higher levels of proficiency.
A further major advantage of the UIS RS is that any robust assessment program may be suitable for alignment, including national assessment programs.
With three points of measurement in SDG 4.1 (grades 2/3, end of primary, and end of lower secondary) across 193 education systems, it is essential to keep our reporting options open. Reliance on international and regional assessment programs only risks excluding education systems that may have most to gain.
Harmonizing learning assessment data globally requires a multi-pronged approach
The value of the UIS RS was endorsed by the global assessment community in Hamburg in October 2017. Another big step forward in Hamburg was the agreement to link existing cross-national assessments, with two to three countries per region participating in both Trends in International Mathematics and Science Study (TIMSS) and a regional assessment in 2019.
As well as statistical linking, this initiative will require detailed analysis and discussion of how learning domains are conceptualized in each assessment, including in developed and developing economies. The draft UIS RS will provide a valuable, "program-neutral" point of reference to inform this analysis. This solution could be extended to assessment programs in other domains (PIRLS and LaNA) in the future, whenever regional and international assessment cycles coincide. Capitalizing on these opportunities will depend upon the rapid mobilization of funding, as well as in-kind collaborative support.
The Hamburg discussion confirmed that the global harmonization of learning assessment data is a multi-faceted process, in which a variety of methods must be developed and applied. Approaches such as those in A Global Data Set on Education Quality (1965-2015) have a valuable place within the suite of tools needed to meet the challenges of measuring progress towards the SDGs, alongside the UIS RS and Hamburg initiative.
They may provide useful interim measures, as global understandings of learning progress are under development; and also have enduring value as a reference point in the global assessment landscape. The paper also demonstrates some valuable possibilities for finer-grained analysis of the SDGs, including benchmarks to show dispersion within systems, disaggregation of data by gender, and comparison of results over time.
At the same time, as the global conversation about how to measure learning gathers momentum, it is also important to remember why. The main purpose of SDG 4.1 is not to compare one system to another, but to gain the best possible diagnostic of where children and young people are at in their learning, so we can plan for where to go next. This means that the numbers used in SDG reporting must be meaningful descriptors of learners' location on a continuum of progress; not just a ranking on the world stage. From this perspective, the efforts by the Global Alliance to Monitor Learning to link regional and international assessments using the UIS RS are critical, and warrant strong international support.