Efforts to reform higher ed will founder on bad data


Sep 29, 2016

Despite the current unproductive political climate, a sensible and pragmatic bipartisan proposal to reform higher ed has been put forward in Congress. The bill, co-authored by Senators Chris Coons (D-DE) and Johnny Isakson (R-GA), would require elite schools to admit more low-income students; it would also require underperforming colleges to improve their graduation rates. But, as always, the devil is in the details, and it’s unclear if the tools to implement such a proposal even exist.

College graduation rates are often described as stubbornly low, with six-year completion rates hovering just below 60 percent. Private schools fare somewhat better than the average, with six-year completions slightly above 65 percent; for-profit schools are much worse, with six-year completion rates averaging only 27.8 percent for the cohort that started college in 2008. This measure, however, also varies widely by selectivity: non-selective schools are almost as bad as for-profit ones in terms of completion rates. Selective schools, on the other hand, have completion rates approaching 90 percent.

Six-year completion rates by institutional selectivity


From the National Center for Education Statistics.

Although these numbers may at first blush seem to be a sweeping indictment of U.S. higher ed, the picture is complicated by what the numbers measure—and what they don’t.

What they measure

  • First-time, full-time students who complete bachelor’s degree programs within six years at the institution at which they started.

What they don’t measure

Given these huge data gaps, it’s tough to know which colleges really aren’t performing well—and which ones are. Incorporating transfer students into the data would likely make completion rates look a lot higher; adding in part-time students could make them a lot worse. For example, if College A were to accept a large number of transfer students but never graduate any of them, it’s completion rate would be unaffected. But if College B were to enroll a lot of students who earned credits cheaply before transferring to College C to complete their degrees, College B would get dinged. Other schools that could get hurt? Those that try to enroll students that need additional remediation—graduation grates fall precipitously for students who need remedial courses. It’s possible that highly selective schools have high graduation rates because they are good at what they do—but it’s more likely that having the most academically prepared and well-resourced students has a lot to do with it.

In the Gates Foundation’s recent report, “Answering the Call: Institutions and States Lead the Way Toward Better Measures of Postsecondary Performance,” which examines data issues and solutions in higher ed, the authors note, “Currently, higher education’s data ‘infrastructure’ is a set of disconnected systems, all of which were created for their own purposes at distinct points in time, but none of which are presently able to fully provide the answers we need to pressing questions about key student outcomes.” Beyond information about students’ progress through institutions, including credit accumulation, transfers, and graduation, we also need information about learning, earnings, and employment. We also need to recognize that the diversity of our higher ed system is a good thing; different institutions are playing very different roles, and different metrics apply. A graduation rate below 90 percent at an Ivy League School would be a sign that something was wrong. But for most schools, 90 percent is so far from reality, it might as well be a dream.

Senators Coons and Isakson are going after a real problem in higher ed. But it’s hard to see how punishing “underperforming” schools will fix the problem when there still isn’t decent data on college performance. The first step in holding higher ed accountable is to build better data. Ironically, the bill is called “self-financing” because it assumes that the fees high-performing schools would pay for their lack of diversity would provide low-performing, high-access schools with $8 million over five years to improve their graduation rates. It’s hard to imagine how a school could substantially move its graduation rate with only $1.6 million—but it’s even harder to imagine how we could get data to measure those graduation rates without spending any money. Congress has identified the problem, but solutions will take a little more effort.

For more, see:

Alana Dunagan

Alana leads the Institute’s higher education research and works to find solutions for a more affordable system that better serves both students and employers. In this role, Alana analyzes disruptive forces changing the higher education landscape.

  • Absolutely to all of this. An even better drill down would be a reporting out on completion relative to Federal financial aid resources received. There most certainly are institutions and institutional cultures that do not bring any urgency to the table with regards to timely completion — and that too is a positive diversity in the system or is when it does not represent a mismanagement of public resources. In general, selectivity is a blunt tool in getting at the relationship between resource consumption and institutional performance. And, of course, in the realm of stating the obvious, completion for its own sake — without attention to the delivery of either marketable career skills or highly valued citizenship skills — is a narrow valuation. Richard Arum & co are certain they are on track to assert the magic algorithm, but I’m skeptical about the depth and richness of that data set either. Dan Chambliss’s read of long term return on investment remains compelling to me but it is admittedly uninterested in any comparative read on institutional performance.

    • Alana Dunagan

      Jim – Completely agree! Professor Astin from UCLA sent me an article he wrote in 2004 which makes essentially your point. http://www.chronicle.com/article/To-Use-Graduation-Rates-to/27636 His work found that variability in the student population explained a great deal of the variation between schools when it came to completion. From a policy perspective, I don’t think we can blame it all on the students and let it rest there – our economy needs more graduates then we are producing, so we have to find a way to improve the system. But I do think it makes efforts to ‘punish’ schools highly problematic. The other issue, as you point out, is that when we collapse our analysis to one metric – e.g. graduation rates – then the simple solution is to improve on that metric, at the expense of others. In this case, a rational response by institutions would be to graduate more students at the expense of academic quality and the very meaning of that degree.

      More work to be done before the carrot and stick solutions can solve all of our problems!

  • Alisa Chapman

    I believe you have a typo in the title. Should it be “flounder” instead of “founder”?

    • Alana Dunagan

      Both work!

  • Senator Coons and Isakson are right . Just close those underperforming school not just punish them .
    USA schools are great in the world . But only 200 or so HE schools are good in the USA . Rest is underperforming trying to make money and statue .

    • Alana Dunagan

      The US higher education system is incredibly diverse – and that creates huge value for this country. We are blessed that so many of our institutions are world class. But there is value in what other institutions are doing as well, and educating the populations they are serving. Harvard does a great job – but if all our schools were trying to be just like Harvard, we wouldn’t be serving our students or the labor force very well.