What challenges are there when attempting to incorporate fact-checking into news ranking systems that consider news quality?
In the ideal world of the NewsQ Initiative, accurate, informative, trustworthy and otherwise “quality” journalism would be promoted, recommended and ranked at the very top of our news feeds. So far, this goal has been challenging to accomplish, in part because the algorithms that prioritize what appears in our news feeds are bound by rules, and it is difficult to establish common rules for identifying quality.
To explore the challenges of designing algorithms based on individual signals of quality, and also the ecosystem around those signals, let’s consider the fact-checking initiative Check Your Fact and its relationship with The Daily Caller.
When Considering Online News Quality, Focusing on Facts Alone Can Risk Oversimplification
Incorporating fact-checking into news algorithms is attractive for platforms that need an efficient way to rank an enormous amount of information. This is because algorithms operate according to defined rules, and, in theory at least, standardized data inputs may be able to help machines decipher fact from fiction in the form of scorecards or grades, for example.
Unsurprisingly, platforms such as Google and Facebook are exploring different ways to enhance their ranking algorithms through the recognition of facts. For example, the International Fact Checking Network (IFCN) promotes professional journalistic standards through their commitment to a code of principles. Google and IFCN launched a partnership in 2017 that currently highlights the work of IFCN signatories through Google’s ”Fact check” tag. In addition, being an IFCN signatory is a prerequisite to becoming one of Facebook’s fact-checking partners.
Focusing on such facts alone in news algorithms could run the risk of oversimplifying the complex nature of journalism, which goes beyond simply presenting “both sides” of an issue. Emily Bell, director of the Tow Center for Digital Journalism at Columbia Journalism School, echoes this in an interview with journalism professor Mike Ananny:
Fact-checking is appealing to a company like Facebook because it addresses an immediate problem on their platform. […] But it also fits with a worldview which is common in technology companies that says: you have two contesting claims and you investigate them and then the one you deem correct wins.” That shrinks other aspects of journalism that go beyond “right” and “wrong” — things like empathy and context.
This is why it is challenging to develop an algorithm-friendly approach to identifying “signals of quality”: not only is there a massive number of signals that must be considered, but factors such as truth or trust are also frequently harder for machines, which cannot weigh nuance or context, to measure.
What is the Connection Between Truth, Public Trust, and Journalistic Standards?
However, truth and trust are central to the practice of journalism. “Out of necessity, citizens and societies need accurate and reliable accounts of events. They develop procedures and processes to arrive at what might be called ‘functional truth’,” argue journalists Bill Kovach and Tom Rosenstiel in the 2014 edition of their book The Elements of Journalism. And functional truth is important for trust.
By trusting the information we receive and consume, we are all more able to actively engage as citizens in our democracy, and to make informed decisions about how to interact with our communities. In a 2017 study by the Reuters Institute, survey responses indicated that public trust in news depends on journalistic standards such as proper sourcing and verification.
Cherilyn Ireton of the World Editors Forum also makes the connection between journalistic capacity and trust. In essence, there is a positive relationship between public trust and news organizations that adhere to journalistic standards regarding accurate information.
The effort to ensure there are clearly defined journalistic standards is not just useful for journalism professionals and readers. The rule-bound nature of these journalism standards also suggests potentially easier ways for machine algorithms to identify and incorporate quality systematically when serving content recommendations.
Case Study: Check Your Fact and The Daily Caller
The case of Check Your Fact shows the difficult relationship of facts-based standards and trust — both for humans and for rule-oriented algorithms.
Check Your Fact is a current International Fact Checking Network (IFCN) signatory and is also one of Facebook’s fact-checking partners. It is owned by The Daily Caller, a news outlet that has been accused of publishing conspiracy theories, as well as hiring authors that have made racist, homophobic, anti-Semitic, and misogynistic statements. While Check Your Fact claims editorial independence from The Daily Caller, critics are concerned that the editorial stance of its parent company leads to biased fact-checks and an even greater spread of misinformation.
The relationship among Check Your Fact, The Daily Caller, accuracy, and, ultimately, trust is fraught. While Check Your Fact calls itself the “Fact Check (sic) department” of The Daily Caller, a closer look at Check Your Fact’s professional practices as a fact-checker demonstrates that the initiative is in line with many of its peers. It has standardized processes for fact-checking that have been independently vetted by the IFCN for accuracy in February 2019 and again in February 2020.
However, despite this apparent “trust”, Check Your Fact’s professionalism has been undermined by its relationship to its parent The Daily Caller, which has itself been challenged about its accuracy and bias. The Daily Caller is clearly trusted by some, and is clearly mistrusted by others. The news outlet is not trusted by those who have pointed out The Daily Caller’s failings, yet trusted by others, at least enough to generate a high volume of traffic to its website. In April 2020, The Daily Caller site ranked #1,007 in the world and #170 in the United States (for comparison, The Atlantic site ranks #1278 in the world and #274 in the US).
These conundrums can be seen in a recent example where there were questions about potential political bias in a fact checked article by Check Your Fact. In February 2020, Check Your Fact labelled a Politico article as “false”, leading Facebook to post a disclaimer whenever the Politico article was shared on the social media platform. The ensuing controversy was not so much aimed at Check Your Fact’s fact-checking process, but about the lack of trust in The Daily Caller itself.
News Quality Relies on a Mix of Standards and Trust
The case of Check Your Fact and The Daily Caller shows us that trust is still important in public perceptions of news quality. So, even if trust is hard for news algorithms to measure, it is critical to figure out how to incorporate trust into algorithms in a way that also complements standards. For example, what does it mean that Check Your Fact uses The Daily Caller as a reliable source in its fact checks? When do connections like this have an implication for quality and perceptions of quality in the overall news ecosystem?
In the end, it seems like we need to clarify how standards — both factual and ethical — as well as how values like trust come together to make up the complicated notion of news quality. We will continue to explore this complexity as we reflect further upon the practice of fact-checking and news quality in subsequent blog posts.