Challenges and Opportunities of Evaluating Whole-Child Approaches

Many school leaders and educators recognize that social and emotional skills are important for fostering supportive environments where children can learn and thrive. A wealth of research also confirms this. 

Image

“The science of learning and development is clear on the fact that social, emotional, and cognitive development are all important and interrelated,” AIR Principal Researcher Juliette Berg said. “A number of meta-analyses show that social and emotional skills are critical to learning and doing well—and they also show effects for academic outcomes both in and after school, for engagement in school, and for motivation for staying in school.”

Image

The science of learning and development also stresses that a learner’s individual context can affect the implementation of interventions and any outcomes related to interventions, AIR Institute Fellow and Vice President David Osher added. “In the case of schools, the context includes all the other interventions being implemented, as well as the interactions between and among members of classroom and school communities, and what is happening outside of the classroom and school,” he said.

Much of the evidence on approaches to improving social and emotional outcomes comes from stand-alone programs, and few studies evaluate whole-school and whole-child models. A five-year evaluation of City Year’s Whole School Whole Child model demonstrates both challenges and opportunities for studying an approach that addresses both individual student skills and the context.       
 

Intervention and Evaluation Details

City Year recruits and trains a diverse group of AmeriCorps Student Success Coaches, ages 18 to 25, as near peers to support students, teachers, administrators, and other school staff in classrooms and contribute to a positive school climate. Students look up to near peers for guidance and see them as accessible, because they are closer to their own age and have a better sense of what they may be going through than their teachers or other school staff. 

Under this whole-school model, all students receive a set of services (Tier 1), such as schoolwide events to celebrate positive behavior and student success, and support for students in English language arts and math. Students needing extra academic or behavioral support receive an additional level of services (Tier 2). City Year operates in 29 U.S. cities and partners with 250 schools to deliver this program. 

The five-year evaluation—funded by the U.S. Department of Education’s Institute of Education Sciences—involved a comparative interrupted time series design to examine whole-school effects and a randomized controlled trial to examine the effects of targeted supports. While AIR led the overall evaluation, MDRC led a quasi-experimental impact study to explore the effects of the whole-school model for all students. The analysis compared academic and behavioral outcomes in the 22 City Year middle schools and in 27 similar, matched middle schools in the same districts that did not implement the model. 

AIR also led a second study, a randomized controlled trial of the effects of Tier 2 services for individual students identified as being at heightened risk of dropping out of school. These students were randomly assigned to receive or not receive Tier 2 services.        
 

Select Evaluation Findings

• The randomized controlled trial did not find statistically significant effects on math, English language arts, behavioral outcomes, or social and emotional skills. The effect size for self-awareness for students in City Year schools receiving Tier 2 services in the first year of the study, although not statistically significant, was meaningful within the context of other whole-school social and emotional learning interventions. “Self-awareness is an incredibly important skill, particularly in middle school, for building other skills, such as strong relationship and academic skills,” Berg said. “For independence, identity development, and understanding where one’s strengths and weaknesses are, this is an important outcome.”   
   
• The comparative interrupted time series study did not find statistically significant differences in math, English language arts, or behavioral outcomes between City Year and comparison schools. “We found that 78 percent of comparison schools were partnering with organizations that offered student support services,” MDRC Senior Research Associate Marie-Andrée Somers said. “It’s not like they were doing nothing to support their students. These schools were probably doing some social and emotional interventions or skill building,” making it hard to differentiate the effects of the City Year intervention. The comparison school also may have been providing supplemental math and English language arts tutorial support, Osher added.   
   
• All City Year schools in the study implemented the elements of the Whole School Whole Child model as intended with high or moderate fidelity, including Student Success Coach training and professional development, how on-site teams worked together, and Tier 1 and Tier 2 services offered. This finding indicates that this is a viable model for schools that want to give their students a holistic set of academic and behavioral supports.        
   

The Challenges of Measuring the Effects Whole-Child Efforts

Researchers encountered several challenges while conducting the studies, illustrating the complexity of evaluating whole-school, whole-child models. 

• Intertwined interventions. Disentangling academic and social and emotional interventions that occur simultaneously is difficult. For example, a reading tutor might encourage a student to persist or coach them on study skills during academic tutoring sessions. “Years of convergent knowledge—synthesized as the science of learning and development and by distinguished scientists on the National Commission on Social, Emotional, and Academic Development—indicate the dynamic relationship between social, emotional, and cognitive processes,” Osher said. “The fact that methodology is finally addressing this more is long overdue.”  
   
• Prevalence of student supports. Isolating and measuring the effects of a holistic intervention like City Year’s model is challenging because students in the comparison group also had access to student services. In the comparative interrupted time series study, most comparison schools partnered with an outside organization to provide social and emotional and academic supports to their students. In the randomized controlled trial, Student Success Coaches worked with students in the control group when they asked for help. These choices, which are ethically and politically appropriate, make it more challenging to distinguish the effects of a model like City Year’s and the other supports students received.  
   
• Limited data. “School districts don’t measure or collect data on social and emotional learning in a regular or rigorous way,” Somers said. Beyond attendance, grades, and, in some cases, school climate surveys, “the data schools and districts have about students is pretty limited in terms of understanding whole-child development longitudinally,” Berg said. In addition, although research by Berg and Osher indicates how social and emotional competencies are expressed differently in different contexts, most available measures of social and emotional learning do not account for the changes in the educational, familial, community, and societal contexts in which students grow. There are also little existing data to capture how the context changes in response to an intervention and how the school community reacts or adapts to the intervention.   
   
• COVID-19. The pandemic curtailed the study design and scope by shortening the duration of the interventions when schools closed and reducing the planned evaluation of two cohorts of students to one. “Given the powerful disruptions caused by the pandemic, the evaluators had an ethical responsibility to ensure that their activities did not create additional stress on City Year and school staff,” Osher said. “The AIR/MDRC team honored this obligation, which the Institute of Education Sciences staff also understood.”       
   

Implications for Future Studies

AIR and MDRC experts believe this evaluation points the way toward the next frontier in studying whole-child development interventions, in three particular areas:

Evaluate both fidelity of implementation and schoolwide effects. Future studies should better capture implementation and how students, educators, and support providers experience these interventions. 

“The implementation question is big,” Berg said. “How is the intervention being implemented? Is it being implemented with fidelity? Is there buy-in for a whole-school model? Does it fit within the teaching mindset? Is there sustainability? I do think social and emotional learning programs suffer from a lack of fidelity in implementation either in the immediate or longer term.”

“I also would recommend looking at the individual components of the whole-school intervention, collecting data related to them, and collecting qualitative data from students themselves to learn how they experience the intervention to try to improve its reach,” Somers said. The research team recommended classroom observations, focus groups, interviews, and surveys with school leaders, teachers, and support staff. 

For future studies, the research team recommended bringing practitioner voices in early to inform implementation and the research design, collaborating with practitioners to design and plan the research methodologies and data collection methods, and maintaining communications with practitioners to monitor implementation and adapt to realities as needed. 

Take a holistic approach to measurement. Future studies should supplement the limited data that school districts collect with research methodologies and data collection methods about individual learners, learning settings, and broader contexts.  

"Context matters—and the more we know about it in both the schools receiving the intervention and the schools serving as comparison schools, the more we can discern effects and understand the impacts,” Osher said. “Qualitative data can help us understand what is happening on the ground and how participants experience it. This is valuable on its own and can inform our understanding of impacts, both descriptively and as mediators."

Evaluate programs for middle and high school students. “There are few studies of social and emotional learning for older students in middle and high school that reach What Works Clearinghouse standards for rigorous impact analyses,” Berg said. “There is a need for more studies around programs that seek to build up social and emotional skills in high schoolers and adolescents in general.”