The Context

The idea of ASER first came up in 2004, when Pratham was already close to ten years old. General elections took place in India that year and in May 2004, the Congress-led coalition won the elections to the 14th Lok Sabha and formed the first UPA (or United Progressive Alliance) government.

On 8 July 2004, Shri P Chidambaram, Finance Minister for the UPA government, presented the first budget to the country. In his budget speech he said:

In my scheme of things, no issue enjoys a higher priority than providing basic education to all children. The NCMP mandates Government to levy an education cess. I propose to levy a cess of 2 per cent. The new cess will yield about Rs.4000 - 5000 crore in a full year. The whole of the amount collected as cess will be earmarked for education, which will naturally include providing a nutritious cooked midday meal. If primary education and the nutritious cooked meal scheme can work hand in hand, I believe there will be a new dawn for the poor children of India.

Read the full speech here

Source: http://www.thehindu.com/2004/05/14/stories/2004051409780100.htm

Clearly, resources earmarked for the primary education sector were about to increase significantly. But already by this time, well over 90% of all children in the 6-14 age group were in school. Would the additional resources help ensure that every child was in school AND learning well?

Gross Enrollment Ratio (2005-2015) source: link
Gross Enrollment Ratio (2005-2015)
Source: http://dise.in/Downloads/Trends-ElementaryEducation-2014-15/AllIndia.pdf

The new government seemed keen to ensure that outlays would in fact lead to outcomes. In his budget speech on 28 February 2005, Shri Chidambaram said: 

At the same time, I must caution that outlays do not necessarily mean outcomes. The people of the country are concerned with outcomes. The Prime Minister has repeatedly emphasised the need to improve the quality of implementation and enhance the efficiency and accountability of the delivery mechanism. During the course of the year, together with the Planning Commission, we shall put in place a mechanism to measure the development outcomes of all major programmes. We shall also ensure that programmes and schemes are not allowed to continue indefinitely from one Plan period to the next without an independent and in-depth evaluation. Civil society should also engage Government in a healthy debate on the efficinecy of the delivery mechanism.

Read the full speech here

What could be considered as appropriate outcomes for the elementary school sector? Ten years ago there was no large scale data available on children’s learning. But from ten years of work Pratham knew that although children were enrolled in school, they were not acquiring even foundational skills in reading and math. Since progress through the formal school system builds heavily on these basic abilities, this was the logical place to start. Pratham co-founder Madhav Chavan recalls:

By 2004, Pratham had already developed a simple tool to assess children’s basic reading ability. This tool had been used extensively by Pratham teams working in thousands of villages across the country, to enable them to understand whether their work was helping children acquire basic reading skills, and to establish immediate, short term objectives for each child they were working with. The simplicity of the tool had another major advantage: with the help of community members, the reading assessment could be done with every child in a village, and the resulting 'village report card' could be used to discuss the problem of quality education with parents and community members in a very concrete way.

Read more about the village report card process here.

Village Report Card
Village Report Card

The Challenge

Pratham knew that across the country, children were not learning. But because children were seen to be enrolling in school in greater numbers than ever before, the problem was not visible to most others, whether parents or policy makers. In a 2014 interview with Rajya Sabha TV, Rukmini Banerji (now CEO, Pratham Education Foundation) explained:

 

 

How to make this problem visible to the rest of the country?

As an organization that has always believed in using evidence to guide action, the way forward was to generate evidence on scale that would highlight the problem.

And so the Pratham leadership set about the task of designing a national survey of children’s learning. Many decisions had to be taken. Who would be assessed? On what? Who would do the testing? Where would the assessment take place? These decisions responded not only to the technical aspects of designing the assessment tools,  but equally if not more to the need to make the problem visible to a wide range of people. Not just experts but also parents, community members, and “ordinary citizens” needed to engage in discussions of children’s learning.

Aser Decisions
Aser Decisions

In addition to decisions on the actual assessment, the question of how to get a representative picture of the entire country was a key part of the design process. Thus sampling was a critical part of the survey design. The Pratham leadership roped in Dr Wilima Wadhwa, an econometrician by training, to design the sampling.

Wilima recalls being offered a challenge that was impossible to refuse:

One major decision was that the survey had to generate representative estimates at the district level, meaning that the data had to provide an accurate picture of each district in terms of children’s schooling and learning status. This was a huge challenge. Even today, no other major survey in India generates district level estimates. For example the NSS rounds, which produce India’s poverty estimates, and NFHS which generates data in the health sector, are representative only at the state level. This is because the sample size required to generate state level estimates is much smaller, and therefore the scale of the survey becomes much more manageable.

But the Pratham leadership felt that district estimates were essential because India’s annual planning for elementary education is done at the district level. Estimates of children's learning could become an important input into the planning process.  

The RTE Act provides timelines for meeting certain standards that it has prescribed. For example, the Schedule of the Act provides a three year window for creating and developing the infrastructure as well as availability of teachers. Similarly, a five year timeline has been set for ensuring that all teachers are professionally qualified. Thus, States in their Plan shall reflect how they are planning to meet the timeline. Within these timelines, annual plans are to be prepared focusing on the gaps and the available resources (e.g. funds, adequately qualified and experienced human resource, capacities etc.) to meet these gaps. Similarly, in respect of quality and equity aspects, while maintaining vision and long term and medium term goals,States and districts shall set differentiated,need based annual targets and plan for achieving them.The Annual Plans will thus be a need based prioritised Plan, based on a broad indication of resource availability to a district in a particular year.

-Extract from MHRD (2011), Sarva Shiksha Abhiyan: Framework for Implementation. See the full document here.

After consulting with eminent economists, the sampling methodology for ASER was finalized: in each district, twenty randomly selected villages were to be visited; and in each village, twenty households were to be randomly selected and surveyed, so as to generate a sample of 400 households in each district. That first year, ASER surveyed 28,434 children aged 6-14 in 15,166 households (more details here). In one of the only major design changes during the last ten years, from 2006 onwards the ASER sample size was increased to thirty villages per district, for an estimated 600 households per district. This is about double the size of the rural sample of NSS. 

Now the what, who, and where questions had been sorted out. But how would this tremendously ambitious plan actually be implemented on the ground?

ASER rolls out!

ASER Centre did not exist in 2005; it would be born only three years later. From October to December 2005, Pratham teams dropped their normal work and fanned out across the country to find and train partner organizations to do the survey. 

There are hundreds of amazing stories of these young people traveling to remote rural locations, armed only with a sketchy list of NGOs and a firm determination to get the job done. Some of these stories were written up for a previous edition of this blog [see here]. Rukmini Banerji, Pratham CEO and a key architect of the survey, summarizes the process:

From house to house, family to family, we meet mothers, fathers, brothers and sisters, some literate, some not, some contented, some with dreams, some homes that have nothing but still offer us tea. But everywhere, every time, people gather around when children are asked to read or to do sums. In the poorest house in the village, in the school teacher’s verandah, in the garden of the police constable, there are serious discussions about schools, about learning and about the future of children.

-Extract from R Banerji, S Bhattacharjea, and W Wadhwa (2013), ‘The Annual Status of Education Report’. Read the full article here.