The inconvenient truth about personalized learning


May 4, 2016

Personalized learning is quickly gaining steam among educators, philanthropists, and policymakers. The promise of a personalized education system is enormous: we are witnessing an era when new school models and structures, often supported by technology, can tailor learning experiences to each student and allow students more choice in how they access and navigate those experiences.

But we’ve found that amidst the enthusiasm for personalized learning models, there’s a less talked-about aspect of the education system that will need to shift to make these models viable: education research.

Even though education receives paltry R&D dollars compared to other sectors, like defense and energy, there’s been a relatively steady stream of research over the past decades trying to look under the hood of “what works” in education. But if we are going to break open successfully the factory model of school, that research needs to go further.

In a new white paper out this week, “A blueprint for breakthroughs,” Michael Horn and I argue that simply asking what works stops short of the real question at the heart of a truly personalized system: what works, for which students, in what circumstances? Without this level of specificity and understanding of contextual factors, we’ll be stuck understanding only what works on average despite aspirations to reach each individual student (not to mention mounting evidence that “average” itself is a flawed construct). Moreover, we’ll fail to unearth theories of why certain interventions work in certain circumstances. And without that theoretical underpinning, scaling personalized learning approaches with predictable quality will remain challenging.

It bears noting that there is promising research on emerging personalized learning models, including, but not limited to, a RAND study published by the Bill & Melinda Gates Foundation last year and LEAP Innovations’ recently released personalized learning framework and corresponding survey instruments to analyze and refine personalized learning practices across schools over time. And more studies on topics not dubbed “personalized” continue to reveal crucial insights into how students learn best. For example, researchers, such as Daniel Willingham, have focused on describing and refining theories across the learning sciences (and, importantly, clarifying what those theories do and don’t explain).

More often than not, however, the traditional education research cycle stops short of surfacing the information we need to support high-quality personalized learning. For example, a recent federally funded randomized controlled trial (RCT) analyzing the efficacy of an adaptive math software program found that the software led to an average eight-percentile point gain among students. Although a thorough, well-funded study like this signals promise, findings like these do not reliably tell us why some portion of students or certain classes likely didn’t fare as well whereas others fared far better. A more complete cycle will move beyond finding what works best on average. Instead, those initial findings—often statements of correlation from A/B tests or RCTs—will be the starting point of a deeper inquiry into why certain approaches work for certain students. The graphic below illustrates the more complete research cycle that we lay out in the paper.

Life of the research cycle

Our hope is that the research community can begin to coordinate a more complete research cycle in order to surface the breadth and depth of information needed to support personalized learning environments.

Otherwise, as more schools embrace personalized learning, at best each school will have to go at it alone and figure out by trial and error what works for each student. Worse still, if we don’t support better research, “personalized” schools could end up looking radically different but yielding similar results to our traditional system. In other words, we risk rushing ahead with promising structural changes inherent to personalized learning—reorganizing space, integrating technology tools, freeing up seat-time—without arming educators with reliable and specific information about how to personalize to their particular students or what to do, for which students, in what circumstances.

Researchers will be quick to point out that this issue is as much about funding shortfalls as it is about refining our methodological approach. We agree. A more complete approach to education research will take more R&D dollars, both public and private. We think that this upfront investment, however, is bound to pay off. In the long run, these dollars stand to save us from the enormous inefficiencies of a system that for too long has only known how to deliver learning on average. As we seize the opportunity to personalize learning truly, let’s make sure we are building a solid base of research on how to reach each student.

Julia is the director of education research at the Clayton Christensen Institute. She leads a team that educates policymakers and community leaders on the power of disruptive innovation in the K-12 and higher education spheres.

  • Jay Loftus

    Very well stated! Education research, and in particular research related to educational technology or innovations is too easily abandoned when outcomes do not show promise. The next logical step, as is stated, would be to determine ‘why’. We also never examine the population of students or learners where improvements through innovations have not occurred.

  • Thank you for raising these important issues Julia. At Big Picture Learning we’ve partnered with a number of research teams to conduct a variety of research investigations that align in a number of ways with what you propose.

    see here for more info –

    That said it has been quite difficult to identify partners and perhaps even more importantly funding sources to enable us to conduct this vital research.

    We are actively seeking both researchers and funding sources and would welcome conversations about how we can better investigate, analyze and evaluate the efficacy of our personalized approach.

    One quick side note is that I feel that is FAR more than semantics to make sure that we’re talking about the same thing when we talk about “personalized learning”… I have appreciated the definitions laid out here in this piece from iNACOL by Natalie Abel – and I wonder about your thoughts?

    Even the Feds seem to be a bit confused as they might be using Personalized and Competency-Based interchangeably? –

  • PL schools will will yield similar results if the “growth” targets continue to be those based on attaining “grade level” scores without looking beyond to intellectual, academic, personal growth, etc. Where students are given choice, what meaning does “grade level” have in a non-traditional model?

  • Julie. Indeed, better research is needed on many of the personalized learning matters you surface — “rushing ahead with promising structural changes inherent to personalized learning” and not “arming” teachers and administrators with the right tools and the like.

    But you may be missing a very important point.

    Personalized learning for all students will not emerge until teachers lead their own learning.

    Check out our new CTQ papers on teacher leadership for deeper learning at:

    And another one on micro-credentials, in partnership with Digital Promise:

  • Laura H. Chapman

    Environments for teaching and learning are not static, nor are the conditions for student learning and teaching careers static, frozen in little modules and in etched in time such that preditions of best practices can be generalized without loss of attention to the subtleties that may promote or inhibit learning at a given time with a given student.

    Same gores for teachers,. They are not robots, fully capable of being programmed. A big problem with educational research it thinking it must be replicated and scaled up–like big box franchises.

    A big problem with computer-mediated instruction is the fallibility of the system is lost in the quest to make it the new normal, for reasons that do not go much beyond certain efficiencies in the “delivery system.”

    A big problem with contemporary research is the failure to learn from the history of educational research–issues of practice, financing, training, and practical uses of the knowledge.

  • I find it quite ironic that this article calls for creating “standardized metrics” to measure “personalized education”.

    Based on my 20+ years of experience, when education is truly personalized, young people are empowered to choose to take charge of their education which in turn fuels their passion and love of learning. Any attempt to create “standardized metrics” for “personalized education” is a complete contradiction. It ignores each child’s unique needs and diminishes a young person’s natural, innate love of learning.

    What if we compared Public Ed with Public Healthcare?
    With all the R&D spent on healthcare over the past 100 years, shouldn’t we all agree on how to specifically measure a “happy and healthy life” (mentally, emotionally, and physically)?

    Is it measured by your height/weight chart or cholesterol count or ability to run a mile or the absence of cancer cells? Is it defined by financial security or family type/size? Is it where you live or what work you do or what car you drive? Is it by how many hours of (voluntary) community service performed annually? If we can’t measure it in a standardized report, does it mean that “on average” no one knows how to lead a “happy and healthy life”? Or is it truly just an individualized, personalized answer?

    Our nation spends billions of public tax dollars on healthcare every year, but we don’t require doctor offices serving Medicaid users to report the blood pressure and height/weight stats of all 3rd, 8th, and 11th graders to somehow hold them “accountable” for spending public funds. Likewise, we don’t require dentists who get public Medicaid funds to report annually the number of cavities in all K-6 grade students as a way to “hold them accountable”.

    It’s time we stop our unhealthy obsession with trying to measure everything within public education with standardized tools. We should embrace the fact that asking the question “What works?” in public education can only be answered by “It depends.”

    Let parents and local communities work together with the primary goal of inspiring young people to “learn to love to learn”. Our world needs problem solvers and critical thinkers, not instruction followers and standardized test takers.

  • I agree with Matt Bowman’s thinking that “Any attempt to create “standardized metrics” for “personalized education” is a complete contradiction… and I’ll take his critique a step further: done well, personalized learning will necessarily do more than “reorganize space, integrate technology tools, and free up seat-time” – it will also force us to abandon the batching of students into age-based cohorts making any kind of “standardized metrics” impossible. Look at what Vermont is attempting in setting its standards for secondary students and tell me how any kind of “standardized metrics” will work… and tell me how they will deliver instruction that ISN’T personalized.

    The term “personalization”, like the term “reform”, has been expropriated by profiteers who want to focus first and foremost on data collection… and once data is collected there is a natural inclination to use the data to compare students to each other. Those comparisons are insidious and undercut the kind of personalized education that will compel teachers to gain a deep understanding of what motivates their students and will compel schools to ensure that every student is known well by at least one adult… and here’s the REAL inconvenient truth: that kind of outcome cannot be captured by a “standardized” assessment.

  • Virtually every academic area that studies various methods of assessment disciplines has largely been based on the same research method; assemble groups of individuals into an experiment scenario, determine their average response to the condition, then use the same averaging to formulate a general conclusion about all people. Our current didactic one-size-fits all, transactional digital learning solutions are designed inextricably around the concept of sameness.. All students are paced to a standardized syllabus where virtually all students consume the same course material within a fixed time frame all at the same time with the same assignments and assessments. Ever since grade school, our public education system has used grading systems that rank students by comparing student performance to norms of the average student. When applying to college, high school test scores are compared to the average applicant. When hired by an employer, your grades and test scores, SAT’s and other forms of skill measurement are compared to the average applicant. In the workforce, an employee’s annual review in most circumstances, will be compared to the average employee at your job rank. This is true from credit scores to personality assessments. For decades, the law of averages dictates our individual, personal value. Scientists in their research of practice uses the same core method of research; you put a group of people into an experimental scenario, determine their average response to the condition, then formulate a conclusion about all people based on the average. In the medicine and biology, professionals have for years embraced the theory of averages in research to medical treatment. If oncologists support treatments for the average cancer does a patient with pancreatic cancer does a patient want a treatment that’s designed for the average cancer? Not only was each person’s brain different from the average, they were all different from one another. Sadly, our evaluation systems are built around the theory of norms and averages. In a world where technological advancements are sweeping virtually every industry, It’s perplexing why that in an age where we can engineer stem cell cures through genetic coding or map the human genome, the education industry has been lethargic we have made little progress in developing learning environments that accurately map an individual’s learning potential.