By Hannah-May Wilson, Senior Technical Consultant, PAL Network
When the Sustainable Development Goals (SDGs) were drafted in 2015, there was broad agreement that the new global goals needed to evolve from measuring increased access, investment in infrastructure and reporting average learning gains, to measuring learning with a focus on the most disadvantaged children. The focus on ensuring that no child is left behind is crucial. Evidence from many low-income countries shows that learning inequalities are visible before children even start school, primarily driven by disparities in wealth. When wealth disparities interact with other forms of disadvantage such as gender, geographic location, disability, and ethnic and linguistic minority status, they reinforce and exacerbate disadvantage, with the consequence that disadvantaged children have little chance of ever catching up.
Failure to achieve the Millennium Development Goal of universal primary education in many low-income countries, coupled with the uncomfortable fact that millions of children who are in school are not learning the basics, have resulted in a ‘global learning crisis’ affecting more than one-half of all children and adolescents, according to estimates from the UNESCO Institute for Statistics (UIS). The new data have set alarm bells ringing and are the central focus of the 2018 World Bank Development Report. The eye-watering fact is that today, of the 617 million children and adolescents worldwide not achieving minimum proficiency levels in reading and mathematics, two-thirds are in school.
It is crucial that children learn to read and do basic mathematics in the first few years of education, as these skills provide the foundation for all future learning. If this window of opportunity is missed, learning these skills later in life becomes difficult or impossible without remedial support. In this blog, I explain how the citizen-led assessments, which began as a grassroots effort in India, have become a model for a participatory, civil society led effort to gather data on basic reading and mathematics skills, and how this model can support global monitoring efforts.
What are citizen-led assessments?
The citizen-led assessment model started in India in 2005 with Pratham’s Annual Status of Education Report (ASER). Citizen-led assessments are conducted in the household and not in schools, offering a complementary and contextually-relevant method of assessing learning outcomes that is grounded in the realities of developing countries, where not all children are enrolled in school or attending regularly. Citizen-led assessments focus on assessing the acquisition of a few basic skills for all children (rather than on subject-wise, grade-level outcomes like other assessments) and are conducted orally, one-on-one.
Over the past 13 years, the ASER approach has been borrowed and adapted to produce nationally-representative assessments in Kenya, Pakistan, Senegal, Tanzania and Uganda; as well as assessments representative at the state level in Botswana, Mali, Mexico, Mozambique and Nigeria. These initiatives all belong to the People’s Action for Learning Network or PAL Network. A recent global mapping study conducted by the PAL Network Secretariat has also uncovered citizen-led assessment tools being used in more than 80 education projects in over 33 countries worldwide (see map below).
PAL network studies and other citizen-led assessment projects
Ensuring technical rigour in citizen-led assessments
In the past, citizen-led assessments have been too easily dismissed as lacking the necessary rigour or quality to be compared with school-based assessments. Three key design features of citizen-led assessments often limit the extent to which the model is accepted as a robust and reliable assessment in national and international policy circles.
The first is the involvement of citizen-volunteers in conducting the assessment. This feature is often used to cast doubt on the quality and reliability of the data. However, citizen-volunteers are trained to administer the assessment consistently across sampled households and children, producing reliable estimates of children’s learning. Each citizen-led assessment initiative has a documented monitoring and re-check framework, ensuring that quality data are collected in real time with mechanisms for both planned and random spot-checks. Furthermore, the citizen-volunteers add credibility to the validity of the study by providing unbiased and independent assessments of children’s learning.
The second design feature that is often criticised is the simplicity of the assessment tools, which is often seen as evidence of a lack of sophistication and rigour. However, what may appear to be a simple approach is actually backed up by highly-sophisticated processes to ensure the reliability of the data generated. Technical rigour is a high priority for citizen-led assessments. Sampling strategies and frameworks are designed by, or in consultation with, the National Bureau of Statistics or an equivalent agency. Learning assessment professionals and consulting experts from government agencies are involved in the creation and piloting of new tests. Countries participating in citizen-led assessment typically publish their assessment methodology, training materials, sampling frameworks, survey materials and tests online to ensure complete transparency in the approach. Data analysis methods are disclosed in the published reports, with datasets available either online or upon request, together with codebooks and accompanying notes to ensure that the analysis can be verified by other experts.
The last design feature that is often criticised is that the same foundational ‘floor-level’ assessments are given to every child within a sampled household, regardless of their age or grade. This means that they start with a simple task, such as identifying letters, and build up to reading a short paragraph. This is often used to cast doubt on the scope of citizen-led assessments as they are unable to provide any information about the learning competencies of children who are able to complete the highest level of the basic assessment. However, given that Indicator 4.1.1 requires information on minimum proficiency levels only, the floor-level approach can provide enough information to determine whether a child has reached this minimum proficiency.
In May 2018, the PAL Network published the first network-wide Data Quality Standards Framework (DQSF) as a central element to the network-wide commitment to producing high-quality, robust data to be reported at regional, national and global levels. The DQSF supports citizen-led assessment initiatives to ensure technical rigour, while allowing flexibility to accommodate the diversity of processes and adaptations to local contexts that are central to the citizen-led assessment model. The DQSF is accompanied by implementation and monitoring plans, with peer-support mechanisms to enable every initiative to meet the required standards, ensuring that citizen-led assessment data are included as a complementary source of reporting against Indicator 4.1.1. Furthermore, data from citizen-led assessments are included in the Catalogue of Learning Assessments (CLA) developed by the UIS. This marks an important step towards the inclusion of learning data for children in developing countries who are less likely to be found in traditional educational settings.
Using citizen-led assessments to meet the SDGs
In order to ensure quality learning for all and meet the targets set out in SDG 4 by 2030, we need to know who is on track and who is in need of extra support. More and better data are needed to identify the children who are either not attending school or in school and not learning. Data also enable us to see at what stage learning gaps emerge and how the schools system itself can help to narrow learning gaps. Active efforts are needed to identify the most disadvantaged children and that, in turn, means understanding where they are most likely to be found. In most low-income countries, the children who are the hardest to reach are often found in the areas that are hardest to reach and are unlikely to be in school.
With the adoption of the SDGs, we have an opportunity to ensure every country gathers this type of data. However, the SDG indicator that would report on basic reading and mathematics in early primary school, Indicator 4.1.1a, is still categorised as Tier 3, meaning a global methodology is not in place to measure it. The good news is that the UIS is developing a methodology to report on this indicator which will adequately address the remaining challenges of measuring and reporting on learning levels in Grades 2 and 3. The proposed methodology will support countries in using data from national, regional and international assessments in which they already participate, as long as these assessments meet identified quality standards. This includes the citizen-led assessments. We hope others will join us in supporting the upgrade of Indicator 4.1.1a at the IAEG-SDG meeting in Stockholm, 5-8 November 2018. This is a critical moment for children around the world to be able to realize their right to acquire foundational skills, regardless of whether or not they are in school, as stated in the SDG framework.
Leave a comment