Skip to main content
  • About Us
  • Careers
  • Contact

Search form

American Institutes for Research

  • Our Work
    • Education
    • Health
    • International
    • Workforce
    • ALL TOPICS >
  • Our Services
    • Research and Evaluation
    • Technical Assistance
  • Our Experts
  • News & Events

You are here

  • Home
30 Oct 2019
Report

NAEP Validity Studies Panel Responses to the Re-analysis of TUDA Mathematics Scores

Gerunda Hughes, Howard University
Peter Behuniak, Criterion Consulting, LLC
Scott Norton, Council of Chief State School Officers
Sami Kitmitto and Jack Buckley, AIR

The National Assessment of Educational Progress (NAEP) is meant to be reflective of the entirety of what is taught in the United States. However, with many changes over time, such as the adoption of rigorous state college and career readiness standards like the Common Core State Standards (CCSS), there are questions about the extent to which NAEP continues to meet this objective. During this shift in the national educational landscape, the NAEP Validity Studies (NVS) Panel has monitored, studied, and commented on issues such as this potential mismatch between the states’ and NAEP’s assessment content.

Based on the 2017 NAEP Mathematics TUDA (Trial Urban District Assessment) results, it appeared that student performance trends on NAEP were not as positive as student performance trends on the state assessments that were aligned to college and career ready standards. At that time, several leaders in the affected urban districts called for what amounted to a “recount.”

The results of the recent NVS Panel study by Daro et al. (in press) document the extent of alignment between NAEP and state mathematics assessments according to several important dimensions, one of which is content distribution. These results provide an opportunity for further analysis, that was taken up by Dogan (2019; reproduced in the appendix). Dogan’s TUDA reanalysis study was designed to explore whether content misalignment might be a possible reason for the mismatched results for the TUDAs on NAEP and the respective state assessments.

The following research questions were asked:

  1. How would the 2017 mathematics Grades 4 and 8 TUDA mean scores change if the NAEP subscales were weighted according to the content distribution of selected state assessments?
  2. How would the mathematics Grades 4 and 8 TUDA mean scores change in 2013, 2015, and 2019 if the NAEP subscales were weighted according to the content emphasis of selected state assessments, assuming the content emphasis of those assessments and NAEP were similar in these years compared with 2017?

This report, authored by NVS panel members and AIR staff, serves as a response from the NVS Panel to the analysis conducted by Dogan (2019).

First, an extensive background section provides context for the motivation behind conducting such an analysis. This section covers important historical background on the alignment of standards and assessments, the implications of rigorous college and career readiness state content standards for NAEP, and the value of alignment studies to investigate the validity of NAEP.

The second section provides two major comments on, and caveats for, the methods used in Dogan’s analysis: 1) statistically overweighting a domain is not likely to produce the same result as creating a test blueprint with a greater emphasis in that domain; and 2) the analysis is limited due to reliance on state assessments as proxies for the opportunity to learn.

The final section considers the implications of Dogan’s results for NAEP and the challenges that would be inherent in considering the reporting of results from such analysis in any official or systematic manner.

The conclusion of the report is that the secondary analysis done by Dogan for the NAEP TUDA scores is important and worthy of further exploration as part of ongoing efforts to monitor the validity of NAEP. However, such analyses should not be used in the reporting of any official statistics or even as a recurring set of ancillary results or appendix material. To the extent that there is a real and educationally significant mismatch between the content covered on NAEP and that in the states, the best way to ameliorate this is by modifying the NAEP frameworks, not through post hoc reweighting of the NAEP results. Such an updating of the Mathematics framework is already underway by the National Assessment Governing Board.

PDF icon NAEP Validity Studies Panel Responses to the Re-analysis of TUDA Mathematics Scores (PDF)

Related Projects

Project

NAEP Validity Studies (NVS) Panel

AIR formed the NAEP Validity Studies Panel in 1995 under contract with the National Center for Education Statistics. The panel provides a technical review of NAEP plans and products and identifies technical concerns and promising techniques worthy of further study and research.

Further Reading

  • NAEP Validity Studies (NVS) Panel
  • 50 Years of NAEP Use: Where NAEP Has Been and Where It Should Go Next
  • Validity Study of the NAEP Mathematics Assessment: Grades 4 and 8
  • State Implementation of NCLB Policies and Interpretation of the NAEP Performance of English Language Learners
  • Using State Assessments to Impute Achievement of Students Absent from NAEP: An Empirical Study in Four States
Share

Contact

Sami Kitmitto

Principal Researcher

Topic

Education
College and Career Readiness
NAEP

RESEARCH. EVALUATION. APPLICATION. IMPACT.

About Us

About AIR
Board of Directors
Leadership
Experts
Clients
Contracting with AIR
Contact Us

Our Work

Education
Health
International
Workforce

Client Services

Research and Evaluation
Technical Assistance

News & Events

Careers at AIR


Search form


 

Connecting

FacebookTwitterLinkedinYouTubeInstagram

American Institutes for Research

1400 Crystal Drive, 10th Floor
Arlington, VA 22202-3289
Call: (202) 403-5000
Fax: (202) 403-5000

Copyright © 2020 American Institutes for Research®.  All rights reserved.

  • Privacy Policy
  • Sitemap