Consequences of Selective Reporting Bias in Education Research
Selective reporting is a critical concern in the scientific literature. Imagine if a study only reported on the positive effects of an educational intervention, while hiding negative or inconclusive results. This bias would give others an inaccurate understanding of the intervention’s effectiveness. This problem is especially salient in meta-analyses that combine quantitative results across many studies. Meta-analytic conclusions may appear comprehensive but can be biased due to selective reporting in the scientific literature reviewed.
The project’s methods and tools will help applied meta-analysts diagnose and adjust for selective reporting bias, while addressing major limitations of related existing methods and empirically examining the consequences of these biases in education research.
Our project aims to develop new methods and tools for identifying and adjusting for selective reporting in modern meta-analyses. Related existing methods are insufficient; they traditionally assume that selective reporting only occurs at the study level, such as deciding to not submit an entire study for publication. But selective reporting can also occur at the outcome level, such as publishing a study but reporting on only the outcomes with positive effects.
Our project will modify a promising technical approach called selection models to address both types of selective reporting. Another technical approach called robust variance estimation will account for the complex statistical issues that arise when a study includes multiple outcomes, which is a widespread occurrence.
Our project will also:
- Conduct simulation studies to assess the new methods’ performance across a range of plausible meta-analytic scenarios;
- Develop software tools for applied meta-analysts to use the new methods in the statistical software R; and
- Empirically evaluate the consequences of selective reporting in education research by reanalyzing data from prior meta-analyses.
Overall, our project’s methods and tools will help applied meta-analysts diagnose and adjust for selective reporting bias, while addressing major limitations of existing methods. These resources can help promote more rigorous evaluations of evidence in meta-analyses, ultimately supporting improvement in decision making in education.
