In a previous blog post, in order to understand the complexity of accuracy and fact-checks as a set of signals that indicate news quality, I asked what fact-checking actually is. My previous blog post provided a short answer to this question and described ante fact-checking (before publication) and post fact-checking (after publication) that can be used as a proxy for accuracy and quality in journalism.
This second blog post will provide the “long answer” by explaining three levels of fact-checking.
Exploring the Different Levels of Fact-checking
When it comes to post or ante fact-checking, how to define a fact—and about what information the checker is empowered to check in the first place—is often unclear. Take this sentence from a Mother Jones investigation:
Today, dozens of people who were sentenced by [Judge] Reade while she owned prison stock remain behind bars.
During the fact-checking process, the “first level” of checking involves identifying independent facts, and then verifying each, one by one. Some facts are simple to verify: the date an event occurred, for example, the color of an object, or in this case, the existence of Judge Reade.
At the “second level” of fact checking, checkers review multiple facts together and fact-checking becomes more complicated. The above sentence from Mother Jones not only contains multiple facts, but also “strings” of facts that are dependent upon one another in ways that can complicate verification:
- First, a fact-checker encountering the sentence above needs to verify that Judge Reader sentenced dozens of people in this instance
- Second, she must verify that Judge Reade owns prison stock
- Finally, and most crucially, the fact checker must verify that the judge sentenced dozens of people while Judge Reade owned prison stock, and that these people remain behind bars
At this second level, the fact-checker must exercise judgement, verifying not only discrete facts, but also the accuracy of the overall assertion.
If, for example, the checker finds that it is untrue that the judge owned stock while she sentenced dozens of people, what is the next step? Would it be sufficient if the judge owned stock at one point in time, and sentenced dozens of people at some other point in time instead of contemporaneously?
The answer to any of these questions lies in context. What is the overarching assertion or framing of the story? In one context, that difference in timing may be essential; in another context, it may not matter at all.
The Final Level (and Limits) of Fact-Checking Before Publication
The need to consider context leads to a third, and final, level of checking: the ultimate claim that the story attempts to prove. It’s at this third level where, most often, fact-checking is insufficient, or fails altogether.
In an Oxford American essay, “Diary of a Mad Fact-Checker,” former fact-checker James Pogue outlines specific examples to help ensure that, in its discrete parts, a story is accurate. He also describes the limits of fact-checking through his encounter with two different histories of the Byzantium period, authored by the same person. The histories contradict one another, but an impartial fact-checker, Pogue argues, could justifiably sign off on either source.
“But it’s not really the province of fact-checking to point that kind of thing out, and a checker rarely gets anywhere close to dealing with the big-T-Truths of a piece,” he writes. “I’ve never once seen the argument advanced in a piece change because of something a checker discovered.”
As the fact-checker moves from discrete facts, to paragraphs, and then to published story, she makes innumerable judgements about the presentation and framing of the facts at hand. As the judgements start touching on the overall context or argument, the fact-checker’s influence starts encountering limits.
Fact-checkers can and do raise flags if they believe the overarching assertion, argument, or portrait rendered is unfair. But at this point, making a final decision about the overall accuracy of the article is usually out of the fact-checker’s hands.
The Limits of Fact-Checking After Publication
The examples discussed so far generally refer to pre-publication or ante fact-checking. With fact-checking after publication, the processes may be different, but similar challenges are still present. For example, when PolitiFact rates a claim “Half True,” it is often because the checkers are struggling with the space between the first and third levels of fact-checking.
Take a case from May 2020, when PolitiFact checked the claim by Presidential candidate Joe Biden that:
The U.S. had 44 CDC staffers ‘in China to observe what was going on [with the coronavirus]. (President Trump) brought home the vast majority of them, I think left only four in place.’
To assess the claim, the fact-checkers deconstructed Biden’s statement into the smallest possible bits of separate facts, and checked them, one by one. They then assessed the truth through a kind of sum-total math, by taking stock of all claims added together.
PolitiFact found that Biden “garbled” the numbers: Staffing was reduced from 47 to three, and not from 44 to four. Technically, Biden’s assertion is mistaken and therefore, in some ways, untrue.
PolitiFact also added that “It is unclear whether the presence of those Americans would have provided an early warning [about the virus].”
Yet PolitiFact found that the correct facts did “add weight to the thrust of his argument” and supported Biden’s overall claim that Trump “brought home” the vast majority of CDC staff in China. Politifact therefore rated the claim as “half-true.”
Accuracy as a Challenge of Writing Itself
Fact-checking deals well with technicalities. When it comes to precise claims of fact, and even tallying those per publication or domain, a rough, if limited and flawed, proxy for accuracy appears. But beyond individual, discrete facts, taking fact-checking as a straightforward signal for accuracy proves difficult because accuracy itself is challenging conceptually and practically.
Going beyond a precise fact to an accurate, truthful argument or portrayal is something the profession, in its hundreds of years of practice, still struggles with and frequently fails to do — a struggle that is perhaps inherent in the process of writing itself. As we consider the relationship of accuracy to news quality, it is critical that we understand that accuracy is one initial step in assessing the quality of news, overall.