This blog was also posted on Brookings' website

The UNESCO Institute for Statistics (UIS) convened the fifth meeting of the Global Alliance to Monitor Learning (GAML) in Hamburg on October 17-18, 2018. Including representatives from over 20 developing countries, this meeting saw a major shift in focus of GAML – from development and refinement of the SDG 4 indicators toward implementation.

UIS plays a critical role in the Education 2030 Agenda by producing cross-nationally comparable education data and working with partners to develop new indicators, methodologies, statistical approaches, and monitoring tools to better assess progress toward international education targets. It is the lead agency for producing the indicators needed to track global progress toward Sustainable Development Goal 4 (SDG 4), while sharing responsibility for some targets with other agencies.

A primer on GAML

GAML was convened by the UIS in early 2016 to provide concrete solutions to develop new indicators on learning needed to achieve SDG 4 and set standards for good practices on learning assessments by bringing together technical experts from around the world. Counting more than 250 members representing a wide range of stakeholders from UN Member States, academia, international organizations, NGOs, civil society, and the private sector, GAML is developing the standards and methodologies needed to measure learning globally while helping countries to produce and use the information to achieve SDG 4.

During the meeting, about 20 countries provided their perspectives to inform decision-making across the learning and skills-related indicators. In addition, four countries—Cambodia, Chad, The Gambia and Tanzania—explained the data they would likely draw on to inform these indicators, as well as the challenges and needs that confront them. Inclusion of a large country contingent and a new decision-making approach differentiated GAML5 from previous convenings.

Aligning national and international measures of proficiency in math and reading

Day one of the meeting focused on procedural papers that would enable countries to align their learning outcomes data against content in the reading and mathematics areas, and to establish procedures that would enable the verification of appropriate standards of data collection tools. Of interest was the proposal that both statistical and non-statistical methods be used in these processes.

This was a welcome approach for countries that do not engage in international or regional large-scale assessment programs (LSAs), or that use data from national assessments, citizen-led assessments, or early grade reading/mathematics assessments.

Questions raised by countries included how to manage differences in identification of grade versus years of schooling for reporting—for example on Indicator 4.1.1 for Grades 2/3, end of primary, and end of lower secondary, and whether the proposed frameworks for reading and mathematics would provide sufficient guidance to countries in their local comparisons with their national assessment data.

Working out the details of several learning indicators and respective roles in GAML

During the second day, the Task Force working on indicator 4.4 presented extensive research on how to approach digital literacy and ICT indicators, with a subsequent proposal for adoption of a Global Framework on Digital Literacy Skills. The definition proposed for discussion was: "The ability to access, manage, understand, integrate, communicate, evaluate, and create information safely and appropriately through digital technologies for employment, decent jobs, and entrepreneurship. It includes competences that are variously referred to as computer literacy, ICT literacy, information literacy, and media literacy."

Target 4.6 calls on countries to ‟ensure that all youth and a substantial proportion of adults, both men and women, achieve literacy and numeracy” by 2030. Task Force 4.6 described its methodological work, and the potential identification of different minimum proficiency levels according to country economic levels. Member States did not view establishing different minimum proficiency levels positively, especially given that that many countries vary significantly in their adult literacy proficiencies. However, reporting more than one proficiency level appeared sound, which would enable countries to make visible progress over time in a manner that setting just one level might not.

Task Force 4.7 presented extensive research to inform the global frameworks of education for sustainability and global citizenship. Addressing Indicator 4.7.5 in particular which relates to the “percentage of 15-year-old students showing proficiency in knowledge of environmental science and geoscience,” the Task Force discussed a proposal for use of Trends in International Mathematics and Science (TIMSS) Grade 8 assessment of biology and earth science. TIMSS biology includes the topics of ecosystems and human health, while earth science includes the topics of earth’s resources, their use, and conservation.

Recognizing the role of UNICEF rather than UIS as the custodian agency for Indicator 4.2.1 (proportion of children under five years of age who are developmentally on track in health, learning, and psychosocial well-being, by sex), it was proposed that Task Force 4.2 dissolve and transfer its materials and research to UNICEF as the latter finalizes reporting approaches and methods. 

Similarly illustrating GAML’s dynamic and responsive nature, there was a proposal that the Assessment Implementation Task Force, a cross-cutting group, re-form as the Capacity Development Task Force to focus on implications of the decisions made by the Task Forces, on country implementation, and on the needs that such decisions will stimulate within countries. Brookings and the Global Partnership for Education (GPE) proposed to jointly lead this new task force.

As Day 2 drew to a close, countries presented a microcosm of the concerns that had been raised throughout the two days and the challenges they faced. Their questions included:

• What are the continuing gaps in data collection and how might these be addressed?
• What are the issues related to consolidation of data sets across agencies and how to resolve them?
• How to develop simple monitoring tools that could be used at decentralized level to capture learning outcomes?

Remaining challenges in SDG 4 monitoring

Beyond these immediate concerns, reporting against the learning and skills-related indicators of the SDGs raised significant philosophical issues for countries. At the national level, some countries noted a move away from reporting the percentage of students reaching particular levels of proficiency, in favor of more meaningful reporting that indicates actual student proficiencies. Although the positive potential implications of SDG reporting can help countries prioritize education, the apparent inconsistency between global reporting needs and constructive use of assessment nationally may need to be reconciled.

Similarly, where national curricula are not well aligned with the curricula that may increasingly be implied by the frameworks for reporting against the SDGs, will national education systems shift their own learning goals accordingly? When this does not happen, how indicative will data cherry-picked from national resources be of real learning outcomes?

Beyond technical questions and capacity needs, participants raised concerns around identification of stakeholders’ roles and responsibilities within countries. This highlighted the need for coordination across agencies (e.g. assessment units/agencies, EMIS units, national statistics offices), and clear communication about not only the indicators, but also what data collection tools and data would meet required levels of quality for global reporting standards.

What's the verdict?

While it may have been daunting for country representatives who were new to this platform to constructively engage in discussion during the packed two-day agenda, it was a successful start thanks to the excellent leadership and coordination of UIS.

Now it's time for diverse stakeholders within countries to have access to similar information, pull it apart, and consider the best options for their own countries to identify data resources and use their reporting as levers for change.

Note: GPE is an active member of GAML, including serving on its Strategic Planning Committee and contributing to the deliberations of various task forces. GPE and Brookings have also proposed to co-lead the new GAML Task Force on Capacity Development.