Published: May 1, 2020
The NewsQ initiative researches, collects, and even develops potential signals of news quality, taking into consideration factors such as journalistic content and behavior, diversity, and more. Currently, our signals are categorized into seven signal classes.
Questions and challenges that NewsQ explores include:
- Can algorithms use signals like these to inform their classification and prioritization decisions as they curate the news content that we all consume?
- Would this lead to rewards for those outlets producing a higher level of quality?
NewsQ Orientation for Newcomers
For those participating in NewsQ news ranking review panels, this is an orientation to help newcomers to algorithms understand the challenge better.
The purpose of this orientation is also to propose common terminology that will form the basis of discussion. As a starting place for inquiries into online news quality, this guide is not intended to provide a full literature review on news ranking and recommendation. References can be found at the end for further reading.
Table of Contents
- What are News Aggregators?
- News Algorithms
- Algorithms and Online News Curation
- News Aggregators and News Ranking
- Algorithms in the Newsroom
- What Should a News Algorithm Look Like?
- References
What are News Aggregators?
News aggregators pull content from a variety of sources into one designated area [1]. While there are many different types of aggregators, we are going to focus more closely on feed aggregators for the purposes of the NewsQ pilot panel initiative[2].
Examples include Google News, Apple News, Bing News, Flipboard, and Yahoo News [3].
Thought Exercise: What is News?
Aggregators pull articles from sources all over the internet. How can aggregators ensure they are pulling from news sources? Is it possible to sort and classify news content without first defining news?
News Algorithms
Algorithms are defined as “a finite sequence of well-defined, computer-implementable instructions”[4]. To simplify this even more, an algorithm is a series of steps to solve a problem or perform a task [5].
Example: Everyday Algorithms Part 1
Algorithms can be compared to a set of instructions, like a recipe [6]:
- ingredients = inputs
- cooked food = outputs
- Follow set of instructions (recipe) to combine/transform ingredients (inputs) to produce cooked food (output).
The recipe metaphor works for an algorithm with a simple set of instructions. However, newsfeeds are based on more than one complex algorithm that utilize automated machine learning to make decisions without explicit human intervention [7].
“The process of [machine] learning,” according to one explanation, “begins with observations or data, such as examples, direct experience, or instruction, in order to look for patterns in data and make better decisions in the future based on the examples that we provide.” [8]
From providing recommendations to filtering hate speech, news aggregators like Facebook and Flipboard (to name a few) use machine-learned models to complete these tasks [9].
Algorithms and Online News Curation
The judgments algorithms make are often baked in via explicit rules, definitions, or procedures that designers and coders articulate when creating the algorithm. Algorithms are neither neutral nor objective — though they will apply whatever value laden rules they encode consistently
Nicholas Diakopoulos [10]
According to communications professor and data journalism expert Nicholas Diakopoulos’ book Automating the News, there are “at least four judging decisions” that algorithms make in combination with one another [10]:
- Prioritization
- Classification
- Association
- Filtering
This might mean prioritizing content that is marked as high quality, classifying articles by news topic, associating articles with other datasets, and filtering out content that contains hate speech. This involves layers upon layers of algorithms working simultaneously to pull, sort, classify and rank the millions of information sources that are published online at any given moment.
Thought Exercise: Algorithmic Judgments
Three articles in the example below have been pulled from the Top Story section of Bing News based on factors such as topic, keyword, and timeliness. Bing News has ranked the Reuters article in the most prominent place, followed by the Associated Press.
News Aggregators and News Ranking
While some news aggregators rely solely on algorithms to curate content on their newsfeeds, others employ a hybrid that relies on both algorithms and human-curated content.
Example: Apple News
Apple News is partly curated by an editorial team that chooses Top Stories for the app as well as the special features section [11]. Content is then updated throughout the day by the international team as news develops. The Trending Stories section is algorithmically-selected and chooses an average of 50 stories per day. Users can personalize what appears in their feed by following, unfollowing, or blocking channels [12].
According to a 2019 audit of the Apple News curation system by Nicholas Diakopoulos and Jack Bandy, “The editorially-selected Top Stories section exhibited more diverse and more equitable source distribution than the algorithmically-selected Trending Stories” [13].
As suggested in a New York Times article published in 2018, a blend of human and algorithmic curation may be helping Apple News avoid some of the “intense scrutiny” that other aggregators experienced at the time [11].
Example: Google News
At Google News, members of the Merchandising team choose “Featured” publications, and members of the News Product Experience team may add temporary topics [14]. Aside from these sections, content is chosen algorithmically based on characteristics such as: Relevance of content, Prominence, Authoritativeness, Freshness, Location, Usability, Language.
These are weighed against other factors to determine the ranking order in their newsfeed. Google News emphasizes that they do not rank content based on “political or ideological point of view” [15]
Google News also has a five-story “For You” news briefing which includes two stories that appear in every user’s feed and three based on a user’s interests and browsing habits [16].
Recently, Google announced that they will prioritize original reporting in their search algorithms, which they admitted previously focused more on the “latest and most comprehensive version of a story” [17, 18].
Thought Exercise: News Ranking Characteristics
For Google News, characteristics such as Relevance of content, Prominence, Authoritativeness, Freshness, Location, Usability, Language are weighed against other characteristics. How should these characteristics be defined?
Algorithms in the Newsroom
“Following predictable data patterns and programmed to ‘learn’ variations in these patterns over time, an algorithm can help reporters arrange, sort and produce content at a speed never thought possible. It can systematize data to find a missing link in an investigative story. It can identify trends and spot the outlier among millions of data points that could be the beginnings of a great scoop.”
Maria Teresa Ronderos [19]
Algorithms already play a role in the investigation, production, and distribution of content in the modern newsroom in a number of ways:
- Toxic Language Moderation at the NY Times
The New York Times uses Perspective API, a software developed by Google’s sister company Jigsaw, to flag toxic comments and moderate hateful speech in their comments section [20]. This is a hybrid effort between human content moderators and algorithms that sort and prioritize comments based on their likelihood to contain non-toxic speech.
- Automated News Leads
Newsworthy is a Swedish news service that identifies news leads with algorithms that sort through European datasets and identify interesting statistical patterns [21].
- Content curation at the New York Times
The New York Times uses algorithmic content curation to assist the work of the editorial teams. One recommendation algorithm uses geolocation to suggest articles from a pool of “30 noteworthy pieces of journalism” selected by editors [22].
What Should a News Algorithm Look Like?
When it comes to content curation, one of the many challenges of incorporating journalistic standards into algorithmic design is being able to distill abstract concepts into something a machine can process. Algorithms require a rules-based logic which inevitably complicates the incorporation of nuanced and complex concepts, such as diversity, into the newsfeed.
Example: Everyday Algorithms Part 2*
A seemingly simple search for local health news requires many decisions: from how to define news to which potential signals of quality should weigh more during prioritization.
Questions that may be helpful to keep in mind for discussion:
- When classifying news, health, and local: Which outlets qualify as news? What is the geographic scope of local coverage?
- When filtering: How are we defining hate speech? Should profanity be filtered out of searches? If so, are there certain examples of profanity that are acceptable?
- When prioritizing: What are potential signals of quality that should/could be taken into consideration during the ranking process? Which signals should ‘weigh’ more in the decision-making process?
If we were able to influence the decision-making process of news algorithms, what should those news algorithms look like?
References
Out of the sources that NewsQ cites for this packet, the project would like to acknowledge in particular Nicholas Diakopoulos and his recently published book, Automating the News: How Algorithms are Rewriting the Media, which informed key aspects of this orientation. We also thank Jack Bandy for his feedback on the guide.
[1] Isbell, Kimberley. “The Rise of the News Aggregator: Legal Implications and Best Practices.” SSRN Electronic Journal, 2010. https://doi.org/10.2139/ssrn.1670339.
[2] Isbell, Kimberly. “What’s the Law around Aggregating News Online? A Harvard Law Report on the Risks and the Best Practices.” Nieman Lab, September 8, 2010. https://www.niemanlab.org/2010/09/whats-the-law-around-aggregating-news-online-a-harvard-law-report-on-the-risks-and-the-best-practices/.
[3] Fogel, Stefanie. “The 10 Best News Aggregators of 2020.” Lifewire. Lifewire, January 3, 2020. https://www.lifewire.com/best-news-aggregators-4584410.
[4] “Algorithm.” Wikipedia. Wikimedia Foundation, April 20, 2020. https://en.wikipedia.org/wiki/Algorithm.
[5] Matlin, Chadwick. “Defining Algorithms-a Conversational Explainer.” Slate Magazine. Slate, February 2, 2016. https://slate.com/technology/2016/02/whats-the-deal-with-algorithms.html.
[6] Prelab 2: Everyday Algorithms. Accessed December 20, 2019. http://db.cs.duke.edu/courses/summer04/cps001/labs/plab2.html.
[7] Brogan, Jacob. “What’s the Deal with Algorithms?” Slate Magazine. Slate, February 2, 2016. https://slate.com/technology/2016/02/whats-the-deal-with-algorithms.html
[8] Mulonda, Yann. “What Is Machine Learning? ‘In Simple English.’” Medium. Medium, November 7, 2019. https://medium.com/@yannmjl/what-is-machine-learning-in-simple-english-b0aaa251cb60.
[9] Marr, Bernard. “27 Incredible Examples Of AI And Machine Learning In Practice.” Forbes. Forbes Magazine, December 12, 2018. https://www.forbes.com/sites/bernardmarr/2018/04/30/27-incredible-examples-of-ai-and-machine-learning-in-practice/#7ed4e5ea7502.
[10] Diakopoulos, N. (2019). Automating the News: How Algorithms Are Rewriting the Media. Cambridge, MA: Harvard University Press.
[11] Nicas, Jack. “Apple News’s Radical Approach: Humans Over Machines.” The New York Times. The New York Times, October 25, 2018. https://www.nytimes.com/2018/10/25/technology/apple-news-humans-algorithms.html.
[12] “View News Stories Chosen Just for You on IPhone.” Apple Support. Accessed April 27, 2020. https://support.apple.com/guide/iphone/see-news-stories-chosen-just-for-you-iph5b557ed3d/ios.
<[13] Bandy, Jack and Nick Diakopolous. “Auditing News Curation Systems: A Case Study Examining Algorithmic and Editorial Logic in Apple News.”ArXiv.org. August 1, 2019. https://arxiv.org/pdf/1908.00456.pdf
[14] “How Google News Stories Are Selected.” Google News Help. Google. Accessed April 27, 2020. https://support.google.com/googlenews/answer/9005749?hl=en.
[15] “Surfacing Useful and Relevant Content – How News Works.” Google. Google. Accessed April 27, 2020. https://newsinitiative.withgoogle.com/hownewsworks/approach/surfacing-useful-and-relevant-content/.
[16] Wang, Sean. “Google News Gets an Update, with More AI-Driven Curation, More Labeling, More Reader Controls (Fingers Crossed for No AI Flops).” Nieman Lab, May 9, 2018. https://www.niemanlab.org/2018/05/google-news-gets-an-update-with-more-ai-driven-curation-more-labeling-more-reader-controls-fingers-crossed-for-no-ai-flops/.
[17] Bohn, Dieter, and Sean Hollister. “Google Is Changing Its Search Algorithm to Prioritize Original News Reporting.” The Verge. The Verge, September 12, 2019. https://www.theverge.com/2019/9/12/20863305/google-change-search-algorithm-original-reporting-news-human-raters.
[18] Gingras, Richard. “Elevating Original Reporting in Search.” Google. Google, September 12, 2019. https://www.blog.google/products/search/original-reporting/.
[19] Ronderos, Maria Teresa. “How Innovative Newsrooms Are Using Artificial Intelligence.” Global Investigative Journalism Network, December 18, 2019. https://gijn.org/2019/01/22/artificial-intelligence-demands-genuine-journalism/.
[20] “The New York Times Is Teaming up with Alphabet’s Jigsaw to Expand Its Comments.” Poynter, June 13, 2017. https://www.poynter.org/tech-tools/2017/the-new-york-times-is-teaming-up-with-googles-jigsaw-to-expand-its-comments/.
[21] “Stats Contain Stories.We’ll Find Those Stories for You.” Newsworthy. Accessed April 27, 2020. https://www.newsworthy.se/en/.
[22] Coenen, Anna. “How The New York Times Is Experimenting with Recommendation Algorithms.” The New York Times. The New York Times, October 17, 2019. https://open.nytimes.com/how-the-new-york-times-is-experimenting-with-recommendation-algorithms-562f78624d26?gi=27c50b879025.