The New Mexico Public Education Department developed the Kindergarten Observation Tool (KOT) as a multidimensional observational measure of students’ knowledge and skills at kindergarten entry. The primary purpose of the KOT is to inform instruction, so that kindergarten teachers can use the information about their students’ knowledge and skills from the KOT to inform their curricular and pedagogical decisions. Stakeholders also are interested in using data from the KOT for other purposes, such as assessing student readiness for school statewide and identifying disparities in students’ readiness for school across the state.

This study examined the construct validity of the KOT to determine whether data from a field test supported using the KOT to measure six school readiness domains and, if not, what domain structure the data best supported.

Key Findings

• Construct validity analyses supported an overall general school readiness score and two domain scores: a cognitive school readiness domain score and a noncognitive school readiness domain score. The analyses did not support the six domain scores identified by the developer.
• KOT domain scores were moderately correlated with scores from an established measure of early literacy skills, and the correlation patterns support the conclusion that the KOT domains measure distinct dimensions of school readiness.
• Rating categories were distinct (that is, no category was redundant) and ordered appropriately (that is, teachers used higher rating categories for students with higher overall ability).
• Substantial classroom-level variation was found for KOT domain scores and item ratings. Such variation is not uncommon among observational measures or indirect assessments, but it raises questions about the extent to which scores measure students’ true abilities.