Choosing a valid and feasible method to measure child developmental outcomes is key to addressing developmental delays, which have been shown to be associated with high levels of unemployment, participation in crime, and teen pregnancies. However, measuring early childhood development (ECD) with multi-dimensional diagnostic tests such as the Bayley Scales of Infant and Toddler Development III (Bayley-III) can be time-consuming and expensive; therefore, parental screening tools such as the Ages and Stages Questionnaire (ASQ-3) are frequently an alternative measure of early childhood development in largescale research. The ASQ is also becoming more frequently used as the first step to identify children at risk for developmental delays before conducting a diagnostic test to confirm. However, the effectiveness of the ASQ-3 is uncertain. In this study, we evaluate the accuracy of the ASQ-3 as a screening measure for children at risk of developmental delay in rural China by age group. To do so, we administered the Bayley-III, widely considered to be the “gold standard” of ECD diagnostic tests, to a sample of 1,831 five to twenty-four monthold children and also administered the ASQ-3 to their caregivers. We then compared the outcomes of the ASQ-3 test to those of the Bayley-III. We find that the ASQ-3 was significantly though weakly correlated with the Bayley-III and that the strength of this correlation increased with child age and was stronger when the mother was the primary caregiver (as compared to the grandmother). We also find that the sensitivity and specificity of ASQ-3 ranged widely. The overall findings suggest that the ASQ-3 may not be a very accurate screening tool for identifying developmentally delayed children, especially for children under 13 months of age or children whose primary caregiver is not the mother.