Chapter 1
The Promise and Pitfalls of Data-Driven Decision Making

Data are ubiquitous in our lives. Using the latest technologies, we can now quickly calculate how many steps we took in a day, how many calories we consumed, and how much money we spent and on what. Knowing this information will ideally help us make better decisions that will improve the quality of our lives. Businesses, health and education organizations, and governments can now quickly crunch big data to help them understand phenomena in ways never before possible.1 The power of data use is simple: armed with data, people will make better choices and organizations will function more effectively. This is the thinking behind a hot topic in educational reform: data-driven decision making.

A decade ago, data-driven decision making wasn’t on the radar of most educators or policymakers. Now it is difficult to imagine an educational reform agenda that does not include data use as a key pillar. The use of data has the potential to change teaching and learning. Teachers now have wider access to information about students’ learning and can address learning gaps before students fall behind. Data use can also build collective responsibility for all students. As student achievement results and teaching strategies are shared among teachers within and across grades, school cultures and routines are changing as well. Transparency is increasing and the culture of individualism that used to characterize classroom teaching is decreasing.

But how do we find the time to incorporate data use into the already incredibly busy professional lives of leaders and teachers? What gets pushed aside if teachers focus on data? How much training and support do educators need to use data effectively? How does data use fit with other reform agendas? Are we in danger of chasing the numbers and forgetting the central purpose of data use, which is to improve teaching and learning?

Are we in danger of chasing the numbers and forgetting the central purpose of data use, which is to improve teaching and learning? Data-driven decision making is very popular in schools and districts across the United States, and there is also increasing emphasis on data use in other countries, including the Netherlands, Canada, Belgium, South Africa, Australia, and New Zealand.2 Although each place may take a different approach, the common idea is that when leaders and teachers become knowledgeable about how to use data in their work—when they collect and analyze data to guide educational decisions—they will become more effective in reviewing their existing capacities, identifying weaknesses, and charting plans for improvement. In the classroom, data can inform how teachers plan lessons, identify concepts for reteaching, and differentiate instruction.3

The push for educators to systematically gather and use data has brought with it a need to develop new competencies, skills, and cultures. But using data is not as straightforward as it seems. Leadership is essential in this endeavor. We can’t simply use data and expect good things to happen. Educational leaders play a critical role in shaping how and why data are used, what counts as data, and what people are aiming for when they push the use of data in schools. Although we titled this book Data-Driven Leadership, we strongly believe that data do not drive decisions by themselves. Individuals use data to engage in inquiry around current practices and inform courses of action. Data-informed leadership is thus a more appropriate term for what we’re asking leaders to do.4 And although the term data-driven decision making is commonly used in the field, from here onward we will refer to the practice as data-informed decision making to signal this important shift in thinking about data use. Leaders, we argue, should use data carefully to inform thoughtful decision making as part of an ongoing process of continuous improvement. Data use should not be seen as a passing fad or fancy. Leaders must take the initiative to assess what types of data are useful and for what purposes. Data-informed leadership aims to contribute to improving student achievement and teacher professionalism rather than threatening them.5

This book is written primarily for educational leaders at the school and district levels. It is geared toward leaders who are interested in becoming more data informed, as well as those who are well on the way and already feeling confident in their approach. Our aim is to provide a guide that will help build the reflective skills of leaders rather than offer a set of prescriptions about putting data use into practice. In order to help leaders get smarter about data use, we share research-based lessons learned from educators about how they have approached data use in their school systems. We examine how district and school leaders can create structures and cultures that support thoughtful engagement with data for continuous improvement. We also expose some of the potential land mines on the road to productive use of data for continuous learning and equity. Our intention is to help leaders avoid those problems and use data effectively and strategically in their decision making.

Perils and Perverse Incentives

All schools are already data informed in one sense or another. In the United States, the existing accountability system and its evaluation of schools based on student performance data expects and ensures it. Behind government accountability policies is the notion that educators need to know how to analyze, interpret, and use data so that they can make informed decisions about how to improve student achievement on state or national assessments. Within this, there is a strong policy emphasis on reducing the achievement gap, especially for historically underserved, low-income students of color.

Data can be very powerful, but they also have hazards.6 There are some perverse incentives inherent in using accountability data within a high-stakes, limited-resource environment, which have led to some perilous practices and pitfalls, including these:

• Cheating on state tests
• Implementing quick fixes
• Targeting resources to students just below accountability thresholds
• Narrowing the curriculum
• Data overload

Cheating on State Tests

Accountability policies “invest faith in ever-increasing and voluminous amounts of numerical data collection. They can create an evidence base for an Orwellian system that can see everything, know everyone, and judge just when and where to intervene with any student, school or classroom, at any time.”7 Continual surveillance and high-stakes tests have resulted in a great deal of fear among many educators struggling to help their students show progress on state tests. In some cases, teachers and administrators have resorted to outright cheating by giving students the answers or changing students’ answers after they have completed the tests, before they are sent for scoring.8

Implementing Quick Fixes

Critics argue that the term data-driven decision making implies an overly technical model of professional action in which educators diagnose weaknesses and implement solutions in a linear fashion, ignoring the complexity of the teaching and learning process.9 As Andy Hargreaves and Dennis Shirley claim, this kind of focus on data use through numerical test score data can impair and impede the improvement of learning for all students:

With AYP [Adequate Yearly Progress] deadlines looming and time running out, teachers have little chance to consider how best to respond to figures in front of them. They find themselves instead scrambling to apply instant solutions to all the students in the problematic cells—extra test-prep, new prescribed programs, or after-school and Saturday school sessions. There are few considered, professional developments here, just simplistic solutions driven by the scores and the political pressures behind them.10

This approach can lead to a focus on easy solutions rather than continuous development and substantive improvements.

Targeting Students Just below Accountability Thresholds

Accountability policies are meant to ensure that educators have high expectations for all students. They are intended to raise achievement across the board, regardless of students’ current ability levels. In what has been graphically termed “educational triage,”11 some schools separate students they see as “nonurgent” from those who are “suitable for treatment” and those who are seen as hopeless cases.12 This is done in order to target resources and attention more economically and address the increasing emphasis on accountability. Educators frequently report focusing their efforts on students who are hovering near the cutoff point for proficiency.13 Teachers target for remediation and additional tutoring those on the cusp—or “bubble”—of scoring in the proficient range while the students below this level are sometimes considered lost causes.14 Similarly, observations of teachers’ data reflection meetings in four schools revealed that discussions overwhelmingly centered on helping students who were below proficiency levels, with few discussions on raising students from proficient to advanced.15 Other research has also indicated that school-level administrators frequently feel pressure from their districts to use data reports to target “bubble...