Clay Christensen was known for, among other things, a hand-crafted sign that hung outside his door. It read: “Anomalies Wanted.”
Clay loved anomalies—things that defied a prevailing theory or conventional wisdom. Rather than brush them aside, he saw anomalies as chances to learn and sharpen his understanding of how the world worked.
If a theory couldn’t explain an anomaly, then something about the theory must be incomplete. Maybe the categorization scheme wasn’t quite right. Maybe it misunderstood causality. Or perhaps the anomaly occurred in a different circumstance that should refine the boundaries of the theory to create a series of “if-then” statements (as in—if you’re in this circumstance, then the following holds; but if you’re in this one, then something else applies).
Either way, the discovery of an anomaly—what Adam Grant might call discovering that you were wrong or that you didn’t know something—was an opportunity to learn and improve the predictive power of a practical theory.
I thought of all that when I read Marty Lueken’s recent piece for Informed Choice, “Test Scores Matter—But Not How You Think.”
The piece opened by citing the recent article in Education Next, “The Predictive Power of Standardized Tests,” which “highlights the strong relationship between middle school test scores and long-run educational outcomes such as college attendance, degree attainment, and earnings.”
But then Lueken points out an anomaly, or a paradox: “several studies of private school choice programs have found something different: students in choice programs sometimes score lower on standardized tests, but end up doing better in the long run.”
He notes that we might not understand causality—”just because early test scores predict later success doesn’t mean that raising test scores necessarily causes greater success”—and that we might not be categorizing the circumstances a child is in correctly. As in, test scores may matter more for a child in a public school than in a private school.
Why would that be?
The piece hypothesizes the following:
“Public schools are directly accountable to the state. The curriculum is aligned to state standards. The tests are aligned to those standards. Teachers are trained to teach to them. … In short, test scores are deeply embedded in the machinery of public education. They aren’t just a measure of student learning—they’re a tool of system-level accountability.”
By contrast, Lueken argues, a private school “might use a very different curriculum, such as classical, religious, or Montessori. It may have different sequencing. It may intentionally avoid teaching to the test. And the parents who choose it might care more about school culture, discipline, religious formation, or a safe environment than a state math test score. So when students at these schools take a test built for the public school system, it’s not surprising that the results might be lower—at least at first. That doesn’t mean the school isn’t working. It means it’s doing something different.”
The article concludes: “So yes, test scores matter—but not how you think. And not always where we’re looking.”
I’m not here to settle the debate—although Lueken’s suggestion seems to have merit. My point is simply that there’s an apparent anomaly here—and it’s worth studying further to sharpen our understanding.
The divide between short-term measures and the long-term outcomes we actually care about for individual students is a hard challenge to wade through in education. It’s among the reasons that a “build-measure-learn” ethos doesn’t always work as we might hope. Our theories and understandings of causality are, at best, incomplete.
How to improve them? Don’t exclude the anomalies. Study them.
