The recent symposium Innovations in Global Learning Metrics: A Focused Debate among Users, Producers and Researchers, hosted by Arizona State University’s Center for Advanced Studies in Global Education (CASGE), brought together a wide range of stakeholders to discuss how to more effectively use global learning metrics for education policymaking. Some of the most interesting discussions focused on the options to achieve globally-comparable reporting on Sustainable Development Goal 4 (SDG 4) based on a paper prepared by the UNESCO Institute for Statistics (UIS).
The paper generated excellent discussion during the symposium, as well as pre-symposium commentary from Kadriye Ercikan (University of British Columbia), Tünde Kovács Cerović (Belgrade University and Open Society Foundations), Radhika Gorur (Deakin University) and William H. Schmidt (Michigan State University). The paper and written comments are available on the Arizona State University website. But the discussion need not end with the symposium – and in fact, it shouldn’t. In that spirit, we would like to share some additional insights on the need for comparability and standardisation in measurement, the importance of measurement in achieving on-the-ground improvements, and whether (and how) the academic and research community can influence policy setting.
First, the UN (and other) institutions who are custodians of the measurement at this point have a mandate. There is not much choice but to follow that mandate. The UN system is comprised of member states which ultimately decide on the work programme of the organizations. Specific agencies have been given the mandate for the measurement and tracking of performance for a set of fixed SDG 4 indicators, and in a manner that is as standardised and comparable as reasonably feasible. Within the Report of the Inter-Agency and Expert Group on Sustainable Development Goal Indicators, the language is very specific: “Global monitoring should be based, to the greatest possible extent, on comparable and standardised national data, obtained through well-established reporting mechanisms from countries to the international statistical system”.
As was wisely noted at the meeting, though, it is true that, while this is a policy or even political mandate now given to the professionals, professionals (and academics) shape the agenda and provide policymakers and political leaders with a sense of what is possible. Professionals cannot entirely “hide” behind a mandate. But we honestly think that, had the policymakers truly been responsive to a technocratic agenda instead of having opinions of their own, the indicators would not be nearly as demanding on us as they are. We are being forced to stretch, especially in areas such as adult learning, civic engagement and sustainability, and digital skills. We are not sure the public and NGO researchers and officials necessarily wished this difficult challenge on themselves.
But more importantly, and as was also wisely noted during the symposium, professionals ought to have the moral courage to engage with their mandate, not just “obey”.
To us, one of the most important reasons to have comparability and standardisation has nothing to do with efficiency, cost savings or accountability but with equity and social justice, taking as a point of departure the contents and skills to which children and youth are entitled. If we did not have the standardised and comparable measurements that we already have, which allow us to talk based on a common language and understanding, for instance, we would not know some of the things we increasingly know, in a comparable and multi-country (that is, pretty generalisable) manner, such as:
You cannot know how much inequality exists, or what drives it, unless you measure it – with a standardised measurement stick, otherwise it is literally difficult to judge that two things are not of equal length. But, we also note that the ideal might be “as much localisation as possible, as much standardisation as necessary.” That is why the UIS’ emphasis has been on supporting the comparability of existing (and future national) assessments rather than on backing, adopting, “imposing” or even endorsing specific global assessments.
Second, it was noted that measurement isn’t really the issue – action by teachers and systems is. This is true, and we would certainly back the idea that there be more funding of the “improvement” function than the “measurement” function. However, improvement can more easily gain traction if one knows what is going on. (There is, of course, already far more backing of the “regular business” aspects of education systems: assessment would be a tiny fraction of that cost. However, there is under-investment in how one actually uses assessments – the right combination of assessments – to improve.) But there is still a measurement mandate aimed at making the problem visible so resources for improvement are dedicated and, since there are efficiencies in specialisation, institutions such as the UIS (and their equivalents at WHO, FAO, etc.) have to focus on measurement. But perhaps such specialised bodies ought to reach out more and support others, whose mission is to use the data to support teachers (or doctors and nurses, agricultural extension agents, etc.). Along those lines, though, we also suggested (with tongue only partially in cheek) that perhaps international assessments ought to be less, not more, relevant or at least less determinant. That is, they ought to be only a reference point (albeit a useful one), and national assessments ought to have center stage. This is the UIS’ position.
A last major issue that was discussed, partly in reaction to the paper but partly also because it was “in the air”, was whether (and how and why) policy research and academic input influence policy. Some were skeptical or pessimistic. Others not as much. In our view, there is impact. Not, perhaps, immediately. And few, if any, policymakers make decisions solely based on evidence. Nor is the impact of research typically traceable to particular academics, books, papers or conferences – it is a much more diffuse process, which can contribute to the sensation that one is not having impact. And, of course, political economy and just plain politics have a lot of influence. But J.M. Keynes got it about right: “Madmen in authority, who hear voices in the air, are distilling their frenzy from some academic scribbler of a few years back… Not, indeed, immediately, but after a certain interval… soon or late, it is ideas, not vested interests, which are dangerous for good or evil.”
We can cite a few optimistic examples or suggest ways to think about the impact of research:
There is no time, resources and energies for questioning the commitments themselves. The 2030 Agenda is a call for everybody. Academia is not the exception, and initiatives such as the GPE’s KIX are stressing the relevance of knowledge exchange and areas where academia can play a critical role if focused on building human capacities at all levels.