NAEP Validity Studies Panel Responses to the Reanalysis of TUDA Mathematics Scores

This paper serves as a response from the NVS Panel to the Trial Urban District Assessment (TUDA) reanalysis study conducted by Dogan (2019) that was designed to explore whether content misalignment might be a possible reason for the mismatched results for the TUDAs on the National Assessment of Education Progress (NAEP) and the respective state assessments in 2017. The study results can be found in the appendices of this paper.

The paper is organized into three sections.

• The first section is a background section that provides context for the motivation behind conducting such an analysis. This section covers important historical background on the alignment of standards and assessments, the implications of the college and career ready standards for NAEP, and the value of alignment studies to investigate the validity of NAEP.
• The second section provides comments on and caveats for the methods used in Dogan’s analysis.
• The final section considers the implications of Dogan’s results for NAEP and the reporting of results.

The conclusion of the paper is that the secondary analysis done by Dogan for the NAEP TUDA scores is important and worthy of further exploration as part of ongoing efforts to monitor the validity of NAEP. However, the authors state that such analyses should not be used in the reporting of any official statistics or even as a recurring set of ancillary results or appendix material. To the extent that there is a real and educationally significant mismatch between the content covered on NAEP and that in the states, the best way to ameliorate this is by modifying the NAEP frameworks, not through post hoc reweighting of the NAEP results.

Updated April 2023: Appendix B was added with updated results from Dr. Enis Dogan.