American Institutes for Research

Here are excerpts from our new report, Putting Data into Practice, a case study of New York City's efforts to use data to improve instruction:

While there have been impressive advances in the collection and management of data used for purposes of accountability, a 2009 report from the U.S. Department of Education found that "even in districts with a reputation for leadership in using data, electronic data systems are barely influencing classroom-level decision-making.”  Many systems aren’t designed to provide data to teachers, let alone students. All too often, the preoccupation with data collection has overshadowed the ways in which data is — or isn't — used.

Few districts have embraced the use of data like New York City, the nation’s largest district with 1.1 million students and 90,000 educators. Data is a vital component of New York City's aggressive strategy to hold educators accountable for student performance and to make sure they have all the tools and support they need to succeed. The district’s "inquiry teams," groups of teachers who collaborate to help students based on shared information, rely heavily on data produced by an $80 million information storehouse called the Achievement Reporting and Innovation System, or ARIS, a repository of statistical information about students that enables classroom teachers to access data such as interim test scores, subject grades, attendance records, and English language learner status on a single computer screen.

After initial resistance, ARIS has won the cautious support of the local principals union, and more than 65 percent of the district’s teachers now participate in inquiry teams. But ARIS has been fraught with problems, as well. Developers have confronted a tangle of antiquated systems that can’t talk to each other – information silos that prevented any one person from getting a complete picture of a student. And they continue to struggle with making the data timely and accurate and giving educators the time and training they need to use it well. In the process, they have learned that technology holds little value unless it is flexible, relevant, and provides the fine-grained information that teachers really need. Above all, the district has realized that building a data system is only the first step – what educators do with the data is the critical second. Building the conditions and demand for data-based analysis is often more difficult than collecting the data itself.

Like all professionals, teachers need better tools to support complex work. Data informs judgment, but it does not replace it. And, different data is needed for different purposes:

The best teachers have always used information about their students to help them improve instruction -- and they know that more and better information can lead to even better results. Yet, unlike for almost all other professionals who perform complex, demanding work, the information tools available to teachers have been remarkably limited. Most teachers still work isolated in their classrooms, with only their own eyes and rudimentary assessment tools to guide them. For the most part, they aren’t benefiting from sophisticated information-gathering tools, from their colleagues’ knowledge, or from analyses of thousands of similar situations – the very kind of information that physicians, police officers, and even sports executives use on a daily basis. Concludes a recent article from SRI International’s Center for Technology and Learning: “Teachers do not have the data-rich, performance-support, and information-feedback work environment that virtually all other high-performance professionals … have at their disposal.”

...[data] systems often fail to give educators the information they need. While the amount of educational data collected continues to grow — Texas school districts alone respond to 104 data collections by the Texas Education Agency each year — the quality and utility of much of it remains questionable. Many systems have become de facto data morgues, used more often to perform autopsies of failed programs than to help educators and policymakers improve existing ones.

Other fields have tackled similar problems. Hospitals, for example, collect data on patient outcomes and mortality rates so policymakers, administrators, and consumers can use it to make judgments about entire institutions. But physicians require different types of information, such as measurements of vital signs and results of blood tests, to diagnose and treat individual patients. Likewise, in education, state and district officials want data that shows broad trends so they can assess a school’s or a district’s overall effectiveness. (This is accountability data.) Teachers want additional information, such as results from classroom assessments that may track weekly progress.

Health reformers have also demonstrated that electronic data systems will not improve performance on their own. Although they are essential, improved technology and better data are just the infrastructure for more substantive changes in the daily practices of providers. For example, officials at Central Pennsylvania’s Geisinger Health System, a model in its efforts to improve quality and control costs, quickly realized that the ability to share data across a variety of systems and contexts was not enough to improve care. What was needed was a cultural change among its employees — a shift from working in isolation on single tasks to working together on tasks that are aligned.

The report continues by describing New York City's approach, the rocky start of its ARIS system, examples from educators about how they want to use data, parent and community involvement, and more. I'll be posting additional excerpts over the next few days.