News Ranking Review Panels Updates

Why That Story? Journalism Panel Recommendations on News Feeds

Why are you seeing that news story? The question of whether higher quality articles can be better ranked and recommended by news algorithms is central to the News Quality Initiative (NewsQ).

But what exactly is an article of higher news or journalistic “quality”? Do journalists even agree about what news quality means? To see if consensus was possible around issues like these, NewsQ facilitated discussions among three panels of journalists and media scholars. To make the discussions practical for news algorithms, panelists worked through samples of actual news feed results from eight different products in three different areas of journalism: local, opinion, and science/health. Technologists from platform news products also joined each panel to help explain the limitations and power of algorithm logic.

While project staff guessed there might be some areas of agreement, we were also prepared for a lack of deep consensus among participants. There are a number of ongoing debates regarding journalistic ethics and principles, about the role of objectivity in reporting, and the gatekeeping role of some publishers. Furthermore, we figured that the question of trade offs might prove sticky. For instance, when pressed to recommend a breaking news COVID-19 story for an event, which should rank higher: a local reporter’s version of events or an article by a journalist with a national science beat? 

The result? All three panels, in fact, had many areas of agreement and productive suggestions to consider. The reason for this agreement? One was that the news feed results were simply not good. Despite the small and illustrative nature of the sample size, panelists would not recommend many of the top five results reviewed as most worthy of reading. And a number of the prioritized articles were problematic. 

The disconnect between news product results and panel evaluations was large because they involved basic definitions about the purpose and mission of news. Put simply, our panelists thought that local news feeds should have stories from local journalists and meet the needs of local communities. Science and health news products should have stories that fit the missions of science and health journalism. Opinion journalism, which is fact-based, still needs to be distinguished from straight news reporting. And perhaps it’s worth saying overall that products positioning themselves as news products should enable readers to encounter news. You can find out more about why and what the panelists thought in technical papers released today here:

In making these recommendations and suggestions, panelists were sensitive to what they could comfortably speak to or support. They were very aware of their own limited perspectives, making the prospect of inviting more journalists to conversations like these critical. The panelists also sought to make suggestions that are practical, both for the technical complexities of algorithms and the grim business realities facing newsrooms. 

Algorithms constantly change, it’s true. This means that some suggestions and recommendations by our panels may potentially already be outdated; in the midst of our reviews, for example, Bing News changed its layout. Nevertheless, recognizing limits like these has not negated the overriding sense that urgent attention is needed. Journalists, technologists, and members of the public need better vocabularies and concepts to discuss the issues raised by the online distribution of news. We all also need to pursue the resolution of these issues in consistent, creative, and collaborative ways. 

The News Quality Initiative thanks our panel participants and the representatives from each platform who joined our conversations. We especially appreciate the commitment of everyone who faced the daunting task of reporting or weathering this year’s events and yet contributed to these discussions and papers. We look to advance these conversations in the year to come.