People often look at ASER’s simple, intuitive front end and the fact that it is conducted by volunteers, and come to the wrong conclusion.
They assume that the easy, user-friendly front end means that data collection processes are not standardized or sufficiently rigorous, and conclude that ASER data is not accurate or reliable. Nothing could be further from the truth.
Over the years, enormous amounts of time and effort have gone into designing, reviewing, and strengthening procedures for ensuring the quality of ASER data. Just one example is the fact that this year, as ASER Madhya Pradesh team member Mahendra Singh Yadav points out, 80% of all villages surveyed in the state were monitored or rechecked or both. And MP is not exceptional in this regard. Similar information is not always available for other large scale surveys, but this quantum of checking is probably significantly more than what is done in most of them.
This short presentation by Senior Research Associate Anuradha Agrawala provides a more detailed overview of the processes that Mahendra mentioned:
[embeddoc url=”http://aserblog.com/wp-content/uploads/2016/09/Ensuring-data-quality-and-generating-reliable-estimates.pptx” download=”all” viewer=”microsoft”]
To ensure that ASER provides accurate estimates of children’s learning levels in India, its data has to be ‘good, clean and reliable’. The survey’s enormous coverage and short time frame means that a set of coordinated, streamlined processes is needed to implement and track the survey across the country.
This is what the annual ASER calendar looks like. Every step has a specific set of processes and criteria intended to ensure quality.
A snapshot of the timeline of ASER
As Dr Manjistha Banerji, Senior Research Associate says, the process begins with intensive training, so that every person involved with the survey has the same understanding of what information is being sought as well as how to obtain and record it.
Practicing the administration of the survey is a critical aspect of ensuring accurate data collection. Field practice begins at the national level training where state team members are taught, trained and tested so that they can in turn go back to their respective states to replicate the training process. All ASER staff members participate in the training, including field pilots, regardless of their previous experience. This is to ensure that everyone, from the newest intern to the most experienced staff member, is familiar with the nitty gritties of survey administration.
Most importantly – at national, state and district level trainings, all participants are assessed on their understanding of the survey process as well as their ability to implement them in the field. While quizzes assess participants’ theoretical understanding, mock trainings and field visits throw light on their training and communication skills and their ability to interact appropriately with communities and children.
Table showing the evaluation processes
Once the survey is in the field, Master Trainers (MT) and ASER state team members monitor the survey as it is being conducted. MTs monitor surveyors via phone and field visits, and ASER state teams in turn, monitor and support the MTs as needed.
Though the basic processes for survey implementation and monitoring are the same across states, each state has the flexibility to customize these if contextual conditions make this necessary. Geographical terrain is often an important factor. For example, weekend-only surveys are not possible in Manipur because villages are extremely hard to reach. Restricting the work to weekends would make it impossible to complete the survey on time. So Manipur planned their survey timetable in such a way that the field survey finished on schedule, while at the same time monitoring implementation of the survey in more than 40% of sampled villages. Kevin Donald Swain, from the ASER team in Manipur says:
Multiple Layers of Recheck
ASER introduced ‘recheck’ in 2008. Four of the thirty surveyed villages in each district were randomly selected to be revisited at the end of the survey, to check if the process of survey had been conducted accurately.
Over time, this evolved into a comprehensive three-step process which begins with MTs desk checking the survey booklets for their district and then making phone calls to surveyed households to verify that surveyors had in fact visited them.
Rechecks are no longer exclusively random, because ASER teams have learned over time what kinds of locations are likely to have data quality issues. For example: villages that are very remote, or those where surveyors’ quiz marks were unsatisfactory, are deliberately selected for recheck.
In addition, summary statistics for each district are carefully examined by ASER staff as soon as the survey concludes. This often leads to the identification of potential problems, so that these villages can be revisited immediately.
Currently, at least 12 of the 30 surveyed villages in each district are revisited. As in the case of monitoring, although general guidelines are the same across the country, recheck processes can vary depending on the specific context of each state. Uttar Pradesh, for instance, is the largest state in India comprising of 69 districts. So to manage data quality in a state as big as UP is no small feat. Sunil Kumar from the ASER team in Uttar Pradesh says:
But that’s not all. Further to this three-step mechanism of recheck, cross-state rechecks (where ASER teams recheck villages in each others’ states) and external rechecks (where external organizations recheck surveyed villages) provide additional layers of verification in selected states and districts where more checking is thought to be needed.
Collecting accurate data does not stop with the survey booklets; the process of entering data on more than 600,000 children from about 16,000 villages demands equal attention and care. At ASER, data management begins as soon as the survey booklets are handed over to the state team, after the recheck process. Once the state teams have reviewed the booklets, the MTs send them to data entry centres. Because of the huge volume and the different languages involved, data entry is done in different locations across the country. ASER 2016 data entry was done in 11 locations.
Data entry and consolidation protocols are the responsibility of Sunai Technologies, based in Patna. Sudhakar Sinha, now Director of Sunai, has looked after ASER data entry since the very first year, 2005. Here Sudhakar gives an overview of the processes that are in place to ensure accuracy and completeness of data entry, long before the data arrives at ASER Centre for analysis:
Data starts flowing into the ASER Centre office in December of each year, when numbers are checked, text is written, tables are generated, and the report is laid out. All ASER staff are told –even before they are hired! – that there will be late nights and seven day work weeks from mid December to mid January, when the report is released.
The ASER report contains an estimated 30,000 numbers – excluding annexures. To ensure that there are no typos or copy/paste errors, before the report finally goes to print, every single one of these numbers is checked – twice. So during what is holiday season for many people around the world, this is an example of what you might hear very late at night in the ASER Centre office:
Ensuring Quality is about Building Trust…
Each number on the pages of the ASER report has a long history that reflects not only who and where it originated from, but also the number of people and processes it went through before it was allowed into print.
Those familiar with the ASER quality control processes find it hard to believe how thorough, how detailed, and how painstaking these processes are. But for those who have the opportunity to see these exhaustive quality control systems at work, these are the steps that build trust. One of the participants in the external recheck for ASER 2016, IDinsight’s Ron Fauber, writes:
The recheck itself is an extension of their deceptively simple model. Going from house to house, we ask previously-surveyed students if they remember seeing the ASER tool and if they remember what level they were able to complete. But think about what that suggests to those communities. First, ASER visits their community to test the learning levels of their children, then they return to make sure that they are providing the most accurate information to the country’s leaders. To the students, parents, and teachers in the villages we rechecked, ASER is affirming: every piece of data matters. It matters so much that we come back to make sure we’re getting it right.