Principle 1

Measure what students accomplish and what it takes to get them there.

Education quality should not be restricted to test scores or to any other set of student performance measures. Quality information should also include students’ actual learning experiences, the educational practices they encounter, the quality of the support their educators receive, and the system conditions required for their success.

This first principle tells us to map out the quality domains that we want to have data on. The basis for this decision rests in our core Theory of Change, a model that describes five key Elements of education quality and how these Elements are connected.

IRRE's Theory of Change

The Theory of Change does not pretend to capture all the contributors to student, school, or district success, much less the myriad possible connections among these contributors. Rather, it is designed to summarize in a clear fashion what research and experience have shown to be the most critical and central factors comprising the educational enterprise and how they affect each other.

Elements of Education Quality:

Student outcomes – how they perform academically and whether they attend, stay out of trouble, and successfully complete school – are most directly affected by the learning experiences they have in school: how they experience what they’re being taught, how it’s taught, their teachers’ commitment to them as students, and their own capacity, interest, and commitment to doing the work. Students’ learning experiences are shaped by what their teachers and their schools actually do: the instructional practices and school practices that characterize their everyday life in school. The actions of educators are influenced by what they’re expected and supported to do: norms and expectations, and the learning opportunities and ongoing supports they receive from peers, school and district administrators, and outside providers to engage in effective practices with their students. These supports for educators are affected by whether and how the system puts certain conditions in place – for example, focus, coherence and collective responsibility – to ensure these supports are present and persist. And these system conditions for success also impact whether or not supports for educators translate into effective practices.

Student Outcomes:

We include, when available, student outcomes typically available from district or state administrative records: state test scores, ACT/SAT scores, end-of-course examinations, completion of critical gateway courses toward graduation and college readiness, and graduation. We also include attendance, persistence year over year, and days of instruction missed because of behavioral issues. Measures of student outcomes are represented in almost all reports of education quality, but most of these do not adhere to the other principles included in this primer.

Students’ Learning Experiences:

Every educator knows the ways students experience their education – their beliefs, attitudes, and feelings – matter hugely to their success on every measure of academic performance and commitment. This is confirmed throughout the long history of survey and qualitative research. Recently we have begun to see limited efforts to incorporate students’ learning experiences into assessments of education quality.

Effective Practices:

It’s surprising how seldom reports of education quality include measures of what teachers are actually doing to create positive learning experiences for their students. Most reporting covers teachers’ credentials, years of experience, student-to-teacher ratios, and salaries. See Principles 2 and 7 for further discussion, but we do not prioritize these convenient measures of educational practice in our quality information systems; none has real authority to predict either students’ learning experiences or their academic performance and commitment. What counts is what students and teachers are actually doing.

Over the past 10 years we have developed and validated a measure we call the Engagement, Alignment, and Rigor Classroom Visit Protocol. This 20-minute classroom visit protocol focuses on essential learning opportunities for all students a) to be actively engaged, b) with instructional content that is aligned with academic standards, whether a set of common core state standards or their counterparts, and c) to experience rigorous academic content and pedagogy and be monitored and supported toward mastery of this content.

This measure of vital signs of quality instruction yields powerful and actionable information about what it takes to ensure students have positive learning experiences and strong academic outcomes.

Supports For Educators:

Also often missing are data representing the support educators receive to implement these effective practices. To help address this knowledge gap, we have developed and field-tested brief, targeted surveys that elicit illuminating responses from teachers and administrators about the supports they receive to initiate, strengthen, and expand effective practices in their schools. We ask about the frequency, utility, and impact of professional development activities and ongoing supports. We ask these questions about outside providers and about leaders and experts in their own schools and districts. From these measures we develop quality metrics shown by research to predict both higher quality implementation of effective practices and growth in student learning.

Conditions For Success:

Drawing from extensive study of the research literature and our own broad experience, we have identified important system conditions which, when present and effectively utilized, help predict improvements on all the other indicators in the core Theory of Change. To get at these conditions we interview key informants, observe organizational practices and examine records to discern how a system, whether a school, district, charter management organization, “improvement zone,” or state, makes its investments of time, people, policy, and political capital to maintain and improve education quality. We ask:

A Broader View of Education Quality:

According to Principle 1, education quality fully conceived can and should be measured in terms of what students accomplish, their learning experiences, the tools and practices educators use with their students, the supports those educators receive to improve, and conditions within the system that support quality defined in these ways.

We believe this broader view of education quality requires an expansive shift beyond current practice, away from near-exclusive focus on student achievement into a broader acknowledgment and appreciation of the context within which students and educators are asked to learn and perform. Our work with clients has shown that taking this broader view does not require massive increases in resources or time.

Many of the data collection, analysis, and reporting activities required to broaden and strengthen education quality assessments are already underway in districts across the nation, needing only to be redirected at these critical elements of education quality, then analyzed and reported according to these core principles.