Using Extant Data to Improve Estimation of the Standardized Mean Difference
Abstract
This paper presents methods for using extant data to improve properties of estimators of the standardized mean difference effect size. Because samples recruited into education research studies are often more homogeneous than populations of policy interest, the variation in educational outcomes can be smaller in these samples than is reflective of the true variation in the population. This affects effect size estimation since the sample standard deviation is used in the denominator of the standardized mean difference. We propose leveraging extant data on sample variance estimates from multiple studies, made available via clearinghouse databases such as the What Works Clearinghouse, to standardize a mean difference. This allows effect sizes to be benchmarked across a common and broad population, thus enabling better comparability across studies and interventions. We derive new estimators of the population variance and the corresponding standardized mean difference, which pool sample variances from multiple studies using both an ANOVA and a meta-analytic framework. We demonstrate the properties of these estimators via analytic and simulation results and offer recommendations for when these estimators are appropriate in practice.
