Assessing the assessments: Taking stock of learning outcomes data in India

Data on learning outcomes is essential for tracking progress in achieving education goals, understanding what education policies work (and don't work), and holding public officials accountable. We assess the accuracy and reliability of India's two nationally representative surveys on learning outcomes, ASER and NAS, so that users of these datasets may better understand when, and for what purposes, these two datasets can reasonably be used. After restricting our sample to maximize comparability between the two datasets, we find that NAS state averages are significantly higher than ASER state averages and averages from an independently conducted and nationally representative survey (IHDS). In addition, state rankings based on NAS data display almost no correlation with state rankings based on ASER, IHDS, or net state domestic product per capita. We conclude that NAS state averages are likely artificially high and contain little information about states' relative performance. The presence of severe bias in the NAS data suggests that this data should be used carefully or not at all for comparisons between states, constructing learning profiles, or any other purpose. We then analyze the internal reliability of ASER data using variance decomposition methods. We find that while ASER data is mostly reliable for comparing state averages, it is less reliable for looking at district averages, or changes in district and state averages over time. We conclude that analysts may use ASER data with confidence for comparisons between states in a single year, constructing learning profiles, and assessing learning inequality but should exercise caution when comparing changes in state scores and avoid using ASER district-level data.