In 2011, the Massachusetts Department of Elementary and Secondary Education partnered with American Institutes for Research (AIR) to perform an independent evaluation and redesign of their existing Early Warning Indicator Index (EWII). The EWII previously identified only students who were entering ninth grade off track, defined as at risk of dropping out of high school based on pre-high school data. For the redesign, AIR considered a much broader definition spanning the entire continuum of schooling. Under this new conception, the EWII defines “off track” as at risk for failing to progress at their age-appropriate developmental level, being retained in a grade, failing to earn a competency determination and/or meet local high school graduation requirements, and risk of dropping out.

To create the new system, AIR evaluators first conducted a literature review of early warning system indicators and a scan of state and district promising practices in the area of early warning systems, comparing these indicators and outcome variables with the existing EWII and other available data. This led to a set of recommendations for the Early Warning Indicator System (EWIS) design based on available research and best practice.

The team then developed a statistical model for the EWIS that shows low, moderate, and high levels of risk of falling off track to graduate at the earliest age possible. To do so, the team tested a series of multilevel models to identify variables that predict the likelihood of falling off track, based on predefined clusters and appropriate outcomes accounting for student- and school-level demographics. Evaluators also conducted research in the area of using early indicators of opportunity to identify qualified candidates for accelerated learning opportunities. In the final component of the analysis, evaluators conducted another literature review and best practices scan for early warning system indicators used to identify students falling off track in higher education institutions, to identify potential outcome data that could be used to develop the EWIS to include indicators for that purpose. The new system is being used to provide critical early warning systems information about individual students (prekindergarten through 12th grade) to local education agencies and schools. AIR is developing communications, presentations, and guides for state education agency staff, local education agencies, and schools to prepare them to use the system in planning, monitoring, and decision making toward improving outcomes for students.