Last week the Brown Center on Education Policy at the Brookings Institute published a new study, “School Superintendents: Vital or Irrelevant?” As the title implies, researchers wanted to gauge what—statistically speaking—superintendents get done. They analyzed data from across districts in North Carolina and Florida and measured the effects that different superintendents—of different tenures—had on student outcomes. The results are somewhat startling: superintendents account for only a small fraction of a percent (0.3 percent) of student differences in achievement. As the researchers explain, “this effect, while statistically significant, is orders of magnitude smaller than that associated with any other major component of the education system, including: measured and unmeasured student characteristics; teachers; schools; and districts.”

The study caught my eye because this week the Christensen Institute released a different piece on superintendents: Knocking down barriers: How California superintendents are implementing blended learning.” This policy brief, drawing from a much narrower sample, paints a very different picture of how superintendents are driving change. Our researchers convened a workshop of several California school district superintendents to answer the question: What are the barriers, real or perceived, to implementing blended learning in your district? After a morning of answering that question, we then asked: Have you found solutions to or ways around these barriers?

The superintendents in attendance described creative workaround solutions that have expanded the potential for blended learning to grow across their respective districts. Specifically, they identified a host of strategies they’d come up with to redesign teacher roles given state policy and teachers union contract provisions; to purchase and manage technology and infrastructure despite budget constraints; and to find ways to recognize credit from online classes as valid for the University of California and California State University systems. The piece is a must-read for California superintendents looking for good tactical guidance. It’s also an inspiring glimpse into strategies of creative leaders unwilling to let existing norms and policies stifle innovation.

In a nutshell, these two important but very different papers confirm why we need both quantitative and qualitative research to shed light on what’s actually happening in school systems on the ground, and how leaders can learn from what others are doing. The researchers for each project asked very different questions, and applied very different methods. But the two pieces overlap in the unit of change—district leadership—that they study. So how might we square the fact that statisticians are finding that superintendents show little impact on learning, but our work also illuminates how superintendents are proving vital to forging the path to new blended learning models that may drive student achievement down the line?

As the Brookings paper findings suggest, superintendents might indeed in many cases be cogs in a pre-fabricated machine hamstringed by policy, existing bureaucracy, and complex interdependent systems. We call such interdependent systems “value networks”—in any organizational context, the nature and parameters of the value network that the organization operates within tends to determine the resources, processes and priorities that people across the organization are all bought into. Changing the resources, processes and priorities of any institution are incredibly difficult: it essentially requires changing the business model, the incentives, and the culture. However, superintendents who are starting to implement blended learning across systems often have to manage major overhauls of this sort to bring new learning models to bear.

This type of superintendent-led transformation may be rare—in statistical terms. But we shouldn’t conclude that superintendents are definitely unable to reshape districts’ value networks. For example, as superintendents like Cary Matsuoka of Milpitas Unified demonstrates, leaders willing to buck many of these typical constraints may render whole new learning models possible.

The Brookings study findings should certainly discourage the simple assumption that changing leadership stands to flip a switch on student outcomes. That’s precisely what our theory of entrenched value networks suggests. But I think the study goes too far when the researchers conclude that leaders’ hands are tied. As the authors say, “The transformative school district superintendent who single-handedly raises student achievement through dent of will, instructional leadership, managerial talent, and political acumen may be a character of fiction rather than life. At the least, such individuals are very difficult to identify using quantitative tools.”

This gap in what quantitative tools can accomplish is precisely where an n of 1—or in Knocking Down Barriers, an n of 7—and further, detailed qualitative research can build nuance into what correlations and statistical research tells us. Once we understand averages like those in the School Superintendents study, we need to seek out anomalies to inch closer to a clear theory of what drives effective leadership in what circumstances. Every anomaly—that is, every superintendent managing to drive student outcomes by breaking out of pre-existing resources, processes and priorities—can teach us something about the causal mechanisms underlying statistical research. Knocking down barriers starts to do this at the level of policy reform, by taking a detailed look at some smart moves among California-based superintendents. More and deeper studies like these of leaders in particular contexts that manage to break the mold can continue to push us beyond average success or failure to identify specific circumstances for success.