Meet the Expert: Joshua R. Polanin

Josh Polanin portraitJosh Polanin, Ph.D., has extensive experience in quantitative methodology, particularly systematic review and meta-analysis, which allows him to design and lead studies across the field of education research. He is the project director for AIR’s What Works Clearinghouse, a project of the Institute of Education Sciences (IES). He is also principal investigator of an National Science Foundation-funded grant building meta-analytic software, the co-principal investigator of an IES-funded grant to train other research in state-of-the-art meta-analysis techniques, and the co-principal investigator of two IES-funded systematic review and meta-analysis of (1) college aid programs and (2) English language learner programs. He won the Society for Research on Educational Effectiveness (SREE) Early Career Award in April 2020. He helps to lead an internal AIR initiative—the Methods of Synthesis and Integration Center (MOSAIC)—with Dr. Ryan Williams among many others, to bring together researchers who use and study systematic review and meta-analysis. 

POSITION: Principal Researcher

AREAS OF EXPERTISE: Quantitative Methodology, Systematic Review and Meta-Analysis, School Violence Prevention, K-12 Academic Achievement

YEARS OF EXPERIENCE: 7 (12, including Ph.D.)

Q: If you could tell the public one thing about research methods, and why they matter, what would it be?

Josh: We live in a world where people are constantly creating new and better data. But we aren’t always creating new and better ways to turn that data into evidence. The term “evidence-based practices” has gone from a niche term to one of the most over-used phrases in just about 10 years. When the term is used correctly, it means that researchers have utilized high-quality methods to produce a statement that they believe is as close to truth as you can get.

Underlying the term “evidence-based,” however, is the methods that researchers have chosen. As we’ve seen recently, when people want to discredit evidence, the first thing they attack is the methods. That’s why the methods are essential—they help the evidence withhold scrutiny from people who would attack it for personal or political reasons. 

Q:What sets meta-analysis apart from other research methods?

Josh: Different teams of researchers will produce lots of different types of results and evidence—even research on the same topic can produce different results. Systematic review and meta-analysis are techniques that locate and combine all the existing research, using predefined procedures and statistical techniques, to synthesize studies on the same topic. I sometimes call it “evidence synthesis,” which is a term that seems to resonate with people.

Essentially, by taking different researchers’ ideas and putting them into a single coherent narrative, meta-analysts try to provide clarity where the literature can be chaotic. Also, because meta-analysis is transparent and systematic, future researchers can use the same evidence and replicate results.

Q: What types of projects are best suited to meta-analysis?

Josh: There are two projects that really play to the strengths of meta-analysis. One is combining, or synthesizing, different evaluations of the same program. When you have authors or researchers who are independently studying and evaluating the same interventions or programs, meta-analysis is great at merging those independent evaluations together. Meta-analysis will tell them the on-average effectiveness of that program, as well as help them understand the discrepancies between the results of those different evaluations. That’s my personal favorite type of meta-analysis project.

The other type of project is synthesizing relationships between two variables. For example, there are lots of projects studying the relationship between socioeconomic status and academic achievement. Because there’s already lots of research on this relationship, meta-analysis lets you merge those disparate evaluations together, to see the relationship between those two factors.

Quote boxQ: What is the What Works Clearinghouse? As Project Director, what are your main goals for the WWC?

Josh: The What Works Clearinghouse provides researchers, practitioners, parents, administrators, and other stakeholders with useful information that will help them decide what kinds of education programs to implement or use. There’s so much different and sometimes contradictory evidence about educational programs and practices—and it can be challenging to make sense of all of it. In a way, the WWC is designed to act a bit like the Food and Drug Administration for educational programs—not in its regulatory capacity, but to advise on what is actually effective and evidence-based.

Now, we want to help move the WWC into its next iteration. In concert with IES, we want to maintain the rigor that the WWC is known for, while making it accessible and useful to an even broader audience.

Q: How has the WWC helped people gain access to high-quality evidence?

Josh: The U.S. Department of Education, in concert with congressional leaders, has tried to instill a culture of evidence through the Every Student Succeeds Act’s (ESSA) Tiers of Evidence. This operationalized how the Department of Education defined high-quality research. There are four tiers; for instance, a Tier 1 rating means that the research met a really high bar for both effectiveness and methodology. The tiers have a lot of practical significance, too, as they’re used to make determinations about funding. AIR has also helped IES develop a number of ways—infographics, videos, products, webinars—to get that information into the hands of decisionmakers.

Our highest-profile accomplishment to date is the release of the updated WWC Standards and Procedures handbook, version 4.1. This new version updated and streamlined several statistical and methodological issues that had hindered the WWC. The release affected every aspect of the WWC: it meant new training, new website features, new communication products, and updating all staff on the new procedures. It was a massive team effort—there are well over 100 people, across multiple subcontractors, involved in this work.

Q: What current education issues do you think will be most relevant in the next 10 years?

Josh: Research on distance education and remote learning will have a huge impact, of course. But what matters is how we’re studying them. As part of our WWC work, we’re conducting a systematic review—and eventually a meta-analysis—of distance education programming. We need research that’s going to help teachers work remotely, as well as programs that are helpful to parents and caregivers.

But it’s also important to consider context. In the past, distance education programs have almost always been compared to “business as usual” in a classroom. As we’ve seen in 2020, that is maybe not the perfect comparison. We need to compare them to other online programs, or maybe homeschooling methods, to get a useful understanding of their effectiveness.

Q: What book do you think everyone should read, and why?

Josh: I have two recommendations. The first is A People’s History of the United States by Howard Zinn. It’s a particularly salient read—it demonstrates the history of brutality in our country’s powerholders, and the power of protest.

The second is The Color of Law, by Richard Rothstein. I’m an urban planning nerd in my free time, but even for those who aren’t, this book shows how the American government designed cities in order to segregate and systematically oppress people of color. Decisions about where to put highways and railways, how big a sidewalk is or where a train station is located, where to zone factories, and which interstate runs through which neighborhoods—these all have huge impacts in every aspect of people’s lives. They have been used to systematically segregate and subjugate people of color and particularly Black people in this country. The decisions also impact students’ educational attainment and achievements. It’s a powerful—and oftentimes disheartening—read.