Journalism prizes, such as the Pulitzers, seem like natural indicators of news quality for general audiences and for ranking and recommendation algorithms, but are they really?
Established in 1917 to reward the best in American journalism and letters, the Pulitzer Prizes are widely regarded as the pinnacle of journalistic achievement in the U.S. In this blog post, using the Pulitzer Prizes as an example, I’ll explore some reasons why such journalism awards might intuitively seem like clear measurements of quality, and why that should not be the case.
At First Glance, Prizes Seem Easily Quantifiable… and Comparable
Pulitzers and other similar awards may appear to be straightforward signals of news quality because these awards are easily quantifiable; we can count the prizes a news outlet receives exactly.
For example, Pulitzer recipients such as the New York Times might state the number of awards they have received:
The New York Times won three Pulitzer Prizes on Monday, bringing the publication’s total to 130 since Columbia University began presenting the annual journalism award in 1917.
Due to the sheer number of Pulitzers awarded to the New York Times, lay audiences may be tempted to think that an outlet that has won 130 Pulitzers is of higher quality than one that has won eight.
While we may know that we should qualify comparisons of the number of awards won with the details of the category in which a publication won, we need to give journalism prizes additional context if we want to compare outlets on that basis.
It’s Difficult to Objectively Compare Journalism Prizes
Using prizes to compare the quality of publications in an apples-to-apples way poses a problem. Take this case as an example: in 2018, journalists from The New Yorker and The New York Times shared the top Pulitzer Prize for public service journalism. One publication had been eligible to win the prize for a century. The other had not been considered before.
It was the first time that The New Yorker, a weekly magazine, won in this category, but the fifth time The New York Times, a daily newspaper, had won. The New Yorker, however, only became eligible for that category in late 2016 when the prizes opened up all the journalism categories to magazines. The New York Times had been eligible to submit entries since the founding of the prizes, and in fact won the public service award the first year it was given in 1918.
In order to really be able to compare the number of awards won by a publication, we need to know how many times that publication has entered the competition in a given category — and whether the publication was even eligible to enter.
Although the Pulitzer is becoming more wide-ranging in the type of journalism and journalists it recognizes, there are always parameters that limit what journalism can be considered for the prize. (Virtually every award operates like the Pulitzers in this regard of being confined to particular types of publication or journalistic media.)
The fact that there are different journalism prizes based on different parameters, for different kinds of media can create a challenge when making direct comparisons across awards, and therefore also when comparing the quality of outlets that have received different awards.
Moreover, many prizes are awarded to individual journalists or teams of journalists, and even when awarded to a publication, it is for a specific category. It’s not always the publishers exactly who are being awarded, making journalism awards imperfect signals of quality at the publisher level.
The Evaluation Process to Award Prizes is Opaque
Prizes may also appeal to us as a top indicator of journalistic quality because they can be more objective than an individual’s evaluation, as they are usually awarded by committees of expert practitioners. Prize juries generally consist of authoritative figures with professional expertise and renown in the field of journalism, so winning journalists and articles can also seem to represent objective journalistic principles as determined by the experts.
But in reality, it’s difficult to determine by what criteria such prizes are actually awarded. Let’s look again at the Pulitzers as an example: when evaluating prospective awardees, thirteen juries consisting of five to seven journalists and editors meet for three days to review more than 2,000 entries in 21 categories and nominate three finalists for each. The winners are later decided by the members of the Pulitzer board, typically from the nominated entries, although on occasion the board has overruled the juries’ nominations.
However, no one outside of these juries knows which publications entered for consideration, beyond the finalists. On top of that, the Pulitzer juries do not use a determined set of criteria to judge the entries.
Instead, the juries have the freedom “to determine exactly what makes a work ‘distinguished.’” At best, we can speculate about what the jury considered important in a given category based on the blurb used to announce the winner, but we really don’t know.
What Do We Value When We Value Prizes?
If we accept that all prizes are limited in scope when it comes to signifying news quality, we are better able to appreciate the values these prizes elevate without overstating and generalizing their importance. It may be that when we value prizes, what we actually value is professional expert judgment in determining quality rather than the winning work itself.
Prizes can also be a useful starting point for evaluating credibility in a media environment saturated with seemingly countless outlets and sources. But let’s not overestimate the significance of these prizes, either (especially for technological systems, like ranking and recommendation algorithms). Using prizes as signals of quality without context may reinforce the choices and implicit values of the prize-givers, which were intended for a specific purpose: to award a prize.
Data scientists may already be aware of the need to contextualize prizes and any other signals used to assess news quality in order to produce better results. But it bears repeating for journalists and others engaged in assessing the quality of news online that we need to be specific about the context of a given award when using them as a shorthand for quality.