Malcolm Gladwell on the Rating Game
by Daniel Luzer
Ah, the college rankings. So much to rank, so much to criticize. In the latest issue of the New Yorker Malcolm Gladwell offers an interesting appraisal of the U.S. News rankings.
There are two kinds of rankings, Gladwell maintains. There are those that are heterogeneous, broad enough to cover a variety of characteristics and rank according to something subjective (“fun” or “exciting” or “sexy”). And then there are ranking systems that are comprehensive, narrow enough to compare similar things and rank along objective characteristics (“the most money,” “the quickest speed,” “the biggest space”). The trouble, Gladwell complains, is that U.S. News tries to be both of these things when it ranks colleges:
It’s an act of real audacity when a ranking system tries to be comprehensive and heterogeneous—which is the first thing to keep in mind in any consideration of U.S. News & World Report’s annual “Best Colleges” guide.
Later he writes that,
There’s no direct way to measure the quality of an institution—how well a college manages to inform, inspire, and challenge its students. So the U.S. News algorithm relies instead on proxies for quality—and the proxies for educational quality turn out to be flimsy at best.
It’s not that much of a revelation. Right, college rankings are an attempt to place a number and a position on something that is comprehensive, complicated, and highly personal. The trouble is that the ranking system applies to all schools. It’s like ranking pizzas, or hotels, or actresses.
This criticism of college rankings is familiar. Gladwell’s more interesting point here has to do with what goes into the rankings. It’s not just that the measures are flimsy; it’s that they’re really just based on someone’s untested assumptions.
The strangest thing about the country’s most prominent college ranking system is that it isn’t based on the one characteristic that most concerns parents: the price of the college. What’s the reason the ranking is determined by the formula U.S. News now uses? Why do the college rankings include a bunch of other characteristics but not college price (or student debt)?
Why? [U.S. News Director of Data Research Robert] Morse admitted that there was no formal reasoning for that position. It was just a feeling. “We’re not saying that we’re measuring educational outcomes,” he explained. “We’re not saying we’re social scientists, or we’re subjecting out rankings to some peer-review process. We’re just saying we’re interviewed a lot of experts, we’ve developed these academic indicators, and we think these measures measure quality schools.
In order words, affordability just isn’t that important. When an organization attempts to perform a ranking like that, one that’s both comprehensive and heterogeneous, some things are going to have to go.