lection

the tyranny of metrics

28 april 2018

I teach at a state university. We give students grades. But the state, suspecting that our grades don't mean anything, insists on conducting an extra layer of assessment called a Unit Effectiveness Plan. The UEP consists of extra grades (also given by faculty) that the students don't see, measuring course goals and outcomes that are unrelated to the course content. Suspicious (rightly) that the UEP is meaningless, the state recently added a third layer of assessment to some courses, called a Signature Assignment, which measures other things that have little to do with a course. Data from the UEP and the Signature Assignment gets sucked into vast databases somewhere, and we never see any analysis of it, for good or bad. For our part, we suspect that the myriad administrators hired to run the UEP and Signature Assignment assessments never look at any of the data they're collecting.

And the state has some reason to suspect grades in the first place. As I walk to my classroom in the morning, I pass students sitting in the hallways of buildings, waiting for their own next classes. They must know I'm a professor – I'm 59 years old and I wear a tie – but they know I'm not their professor, so they do not censor their conversation as I pass. Rarely do they talk about anything they're learning in class. Sometimes they complain about my colleagues, but most often their theme is neutrally about metrics: "She will take away ten points if you don't put your title in italics." "She gave everybody five points extra credit after that test we all bombed." "I don't see how I have a 77; what I really need going into the final is an 83."

Students, like employees and managers in a host of fields, care about the things they're incentivized to care about. Course grades are a classic example of a metric that gets in the way of a mission. As Jerry Z. Muller argues in his terrific, insightful new book The Tyranny of Metrics, when we're rewarded and punished via numerical measures, we focus on the measures and not on the reasons why people initially wanted to measure us.

Muller shows the despotic power of metrics in many fields: higher education (he's a humanities professor like me), medicine, K-12 education, law enforcement – and in business, too. Neoliberal faith dictates that non-profit sectors would run much better if they were run like businesses – but ironically, businesses themselves may run much worse when they are dominated by an obsession with meeting numerical targets.

"Those whom the gods want to destroy they first teach math," says Niall Ferguson, quoted by Muller in his chapter on business and finance (145). The initial impulse to measure and judge, like the first step of many a descent into madness, is pleasantly seductive. You see some complex endeavor – teaching, surgery, baseball – and you wonder how much more efficient it could be if you could pinpoint the best strategies for it objectively. Numbers are objective, right? Let's start keeping track of things we can measure, and analyze the data instead of the fluctuating phenomenon in front of us.

In baseball, this use of metrics – specifically, sabermetrics – works pretty well, or at least did for Brad Pitt in a movie once. But baseball is a metric activity to start with. You don't choose the winner of a ballgame by shifting political and social struggles, but mathematically. The criterion isn't negotiable: it's whoever scores more runs.

But how do we know when a school has won? Or a hospital? Or a charitable organization, or a city government? In a classic example, how do we know when an army has won a war? Robert McNamara believed that if we outdid the enemy in body count, we would win the Vietnam War. The enemy had a different definition of victory (35-36).

Metrics tyrants like McNamara not only made mistakes in not knowing what to measure. They also ran afoul of "Campbell's Law," a keynote of Muller's book. Donald T. Campbell observed that "the more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor" (19). If they're looking at your numbers and not at your work, you're tempted to manipulate the numbers. After a while, you're tempted not to do the work at all, just think of ways to produce numbers.

OK, you might object, guys like Muller and me are touchy-feely humanists, trying to avoid accountability while we blandish you with anecdotes. But Muller is not at all opposed to quantifying stuff. Metrics, he argues, are best used as diagnostics. One of his examples is CompStat (126), the crime-statistics system made infamous by the TV series The Wire. Keeping track of the amount and type of crime by precinct can help police administrators know where to devote resources. But when administrators get the bright idea of rewarding or punishing local captains according to CompStat figures, now seen as "results" instead of indicators, the focus gets shifted to the results themselves, not to preventing crime. Too many felonies on my watch? Maybe I'll get my people to report them as misdemeanors instead. And so goes Campbell's Law.

Doctors measured by survival rates avoid risky procedures and moribund patients. Hospitals punished for high readmission rates stop readmitting people. Colleges rewarded for high graduation rates make graduating easier. And everywhere, symptoms take priority over causes. Most perniciously, schools with low test scores get seen as "bad" and face withdrawal of resources. But such schools are "bad" because their communities are poor. Removing funding makes them worse – and despite the mantra, lots of poor children get left behind.

"Principal-agent" theory, explains Muller, is at the heart of much of the current tyranny of metrics. By "principal-agent" logic, the experts who actually work in an organization – teachers, doctors and nurses, cops, middle managers – are mere agents of the real principals: taxpayers and stockholders. The expertise of agents should be distrusted, because they are conservative, clannish, and rely on priestlike "peer review" which cloaks their irresponsibility. By relying on abstract, objective data instead of the narratives of agents, the principals can get greater return on their investment. One consequence is a rise of a caste of upper managers who can circulate freely among hospitals, universities, government, and executive offices, knowing little about the institutions they control except how to "run them like businesses."

But again, even businesses notoriously run terribly when run solely to meet abstract quantitative targets. Muller traces the collapse of several corporations and banks, and even of the whole financial system in 2008, to the replacement of field-specific expertise by a faith in one-size-fits-all analytics (145-47).

Muller is eloquent about the "psychic costs" of rule by metrics (and its handmaiden, pay for performance). Such complaints, again, might be assailable as mere sentimentality. But consider whose sentiments they are: they might be those of the man or woman doing your bypass operation. Muller quotes Donald M. Berwick:

Relieving pain, answering questions, exercising manual dexterity, being confided in, working on a professional team, solving puzzles, and experiencing the role of a trusted authority—these are not at all bad ways to spend part of one's day at work. … Unfortunately, neglecting intrinsic satisfiers in work can inadvertently diminish them. (119-20)

In other words, reward doctors and other professionals just for being professionals, and you might get way better work out of them than if you pit them against one another, force them into artificial competitions, and peg their compensation to an upward-ratcheting demand for greater "productivity."

Among the greatest of the psychic costs of the inappropriate use of metrics is the cost in morale that accompanies gathering the data itself – which is not just psychic, but a real opportunity cost as well. To go back to that UEP I mentioned earlier: the week after next, I will be grading final projects in a senior capstone course. In addition to giving students grades and feedback, I will also have to complete a checklist that asks whether students have achieved … something or other; the hell of it is that I don't even know what they were supposed to be achieving per UEP, if I was even inclined to teach it. This UEP data, as I said, will be collected, processed, and then forgotten forever. Think, as a taxpayer: what good will that metric do? Would it perhaps be better for me to spend five minutes more on each student paper instead of filling out some damn form that will disappear into the digital æther?

Of course, as a wise principal, you distrust Agent Me, and suspect that instead of spending another five minutes on each paper, I would take the 5m x N papers and devote it to watching ballgames, drinking beer, or writing book reviews on the Internet. But then again, maybe I'll spend five minutes less on each paper while I can comply with the metrics you have concluded will boost the ROI of state universities. You get what you pay for.

Muller, Jerry Z. The Tyranny of Metrics. Princeton: Princeton University Press, 2018.

top