Investigative journalism is underprovided in the market, but new combinations of data and algorithms may make it easier for journalists to discover and tell the stories that hold institutions accountable.
By James T. Hamilton
Investigative journalism involves original work, about substantive issues, that someone wants to keep secret. This means it is costly, underprovided in the marketplace, and often opposed. It gets done when a media outlet has the resources to cover the costs, has an incentive to tell a new story, cares about impacts, and overcomes obstacles. Changes in media markets have put local investigative reporting particularly at risk. But new combinations of data and algorithms may make it easier for journalists to discover and tell the stories that hold institutions accountable.
CJ Lab co-founder Jay Hamilton writes all about this in his new book, published Fall 2016.Available from Harvard University Press
Winner of 2017 Goldsmith Book Prize for the best academic book on media, politics and public policy.
"The Goldsmith Book Prize is awarded to the academic and trade books that best fulfill the objective of improving democratic governance through an examination of the intersection between the media, politics and public policy. The Goldsmith Book Prize for best academic book will be awarded to James T. Hamilton for Democracy’s Detectives: The Economics of Investigative Journalism (Harvard University Press)."
Winner of Frank Luther Mott - Kappa Tau Alpha Journalism & Mass Communication Research Award for best research-based book about journalism or mass communication published in 2016.
"Hamilton provides a fresh and compelling look at the value of investigative journalism in our democracy," said Jeff Fruit, president of Kappa Tau Alpha and a contest judge. "He not only develops and analyzes a unique data set of investigations, but also clearly demonstrates the impact that investigative journalism can have through a case study of one Pulitzer Prize-winning reporter."
What is computational journalism?
Computational journalism is an evolving field.
“What is computational journalism? Ultimately, interactions among journalists, software developers, computer scientists and other scholars over the next few years will have to answer that question. For now though, we define computational journalism as the combination of algorithms, data, and knowledge from the social sciences to supplement the accountability function of journalism.” James T. Hamilton and Fred Turner, “Accountability through Algorithm: Developing the Field of Computational Journalism” (a report from Developing the Field of Computational Journalism, a Center for Advanced Study in the Behavioral Sciences Summer Workshop, July 27–31, 2009), 2. 514-
Computational journalism, “Broadly defined ... can involve changing how stories are discovered, presented, aggregated, monetized, and archived.” Sarah Cohen, James T. Hamilton, and Fred Turner, “Computational Journalism,” Communications of the ACM 54 (2011): 66
In analyzing the dimensions of computation that could advance sensemaking in journalism, Terry Flew and coauthors note:
“Automation alleviates activities such as data gathering and interpretation, number crunching, network analysis, sorting, and processing that would other wise need to be done manually; algorithms allow operators to follow predefined steps needed to accomplish certain goals, identify problems, find suitable solutions in a large set of alternatives, and verify information in a reliable, consistent and efficient manner; and abstraction enables the qualification of different levels or perspectives from which an idea may be presented or new directions that may be explored.” Terry Flew, Christina Spurgeon, Anna Daniel, and Adam Swift , “The Promise of Computational Journalism,” Journalism Practice 6 (2012): 159
Though computational journalism builds on and incorporates elements of computer-assisted reporting and data journalism, this new approach often involves larger data sets and more sophisticated algorithms. Recent advances in computational journalism center on reporting by algorithms, about algorithms, and through algorithms. Stories generated by computer algorithms include those by Narrative Science and Automated Insights, companies seen as leaders in “Automation in the Newsroom.” The Wall Street Journal and ProPublica have each done reporting that looks at the disparate impacts of private and government algorithms on different groups in society. Reporters are also using algorithms to mine for stories, as the Atlanta Journal Constitution did in using web scraping and machine learning to identify potential cases across the country of doctors involved in sexual misconduct.
The papers and video (to come) from the 2016 Computation + Journalism conference at Stanford show recent computational advances in story discovery, telling, and distribution.
The Stanford Computational Journalism lab is currently focused on two research questions:
How do you lower the costs of discovering stories through better use of data and algorithms?
How do you tell stories in more personalized and engaging ways?
Projects involving story discovery include:
- Tracking money in California politics
- Investigating possible racial bias in traffic stops across the U.S.
- Tutorials to discover stories in data
- Teaching Data and Computational Journalism Report
Projects involving telling stories in more personalized and engaging ways include:
- Stanford Journalism Program’s Guide to Using Virtual Reality for Storytelling
- Work by the Stanford Virtual Human Interaction Lab
For more on computational journalism’s history and potential, see Chapter 8, “Accountability and Algorithms,” in "Democracy’s Detectives."
Share on Twitter
What can be done?
Three changes in government policy would readily reduce the hassle costs involved with investigative work:
- (1) real reform of the implementation of FOIA laws;
- (2) adding journalism as a field to support in federal R & D competitions focused on algorithms, data, and tech and in IRS rules that would make it easier for online public affairs sites to gain nonprofit status; and
- (3) truer implementation of open government and transparency policies.
Advances in computational journalism can improve the economics of accountability journalism in two ways:
- On the supply side, new ways of combining data and algorithms could lower the costs of discovering stories.
- On the demand side, telling stories in more personalized and engaging ways for readers and viewers can increase the likelihood of revenues through subscription or advertising.
The set of people whose decisions could significantly expand the field of computational journalism include computer scientists, philanthropists, entrepreneurs, journalism educators, and reporters.
In the press:
- NIEMAN REPORTS: "Accountability Journalism: A Cost-Benefit Analysis" (July 22, 2016)
- POYNTER: "An economist makes the case for saving investigative journalism" (October 10, 2016)
- NONPROFIT QUARTERLY: "A New Book Attempts to Define the Value of Investigative Journalism" (October 11, 2016)
- HUFFINGTON POST: "Investigating Investigative Journalism" (October 18, 2016)
- AMERICAN PRESS INSTITUTE: "Measuring investigative journalism’s impact on society - 8 good questions with James T. Hamilton" (October 24, 2016)
- COLUMBIA JOURNALISM REVIEW: "A daily’s loss in court may cause journalists to rethink how they communicate" (October 25, 2016)
- TEXAS PUBLIC RADIO: "The Source - The Economics Of Investigative Journalism" (October 26, 2016)
- MEDIASHIFT: "How Much is Investigative Journalism Worth?" (October 27, 2016)
- THE NEWS & OBSERVER: "Lessons from a book and a trial - The N&O will keep digging" (October 28, 2016)
- THE CENTER FOR INNOVATION AND SUSTAINABILITY IN LOCAL MEDIA: "Five Questions with James T. Hamilton" (November 2, 2016)
- JOURNALISTEN: "En gavebod" (November 2, 2016)
- FIVE BOOKS: "James T. Hamilton recommends the best books on the Economics of News" (November 21, 2016)
- CORRECTIV: "What economic benefits good research can have" (November 30, 2016)
- THE ATLANTIC: "How the Economics of Journalism Explains 2016's Information Bubbles" (December 2, 2016)
- SVENSKA DAGBLADET: "Digging a bargain for society" (December 4, 2016)
- THE WIRE: "What is the Worth of Investigative Journalism?" (January 10, 2017)
- WASHINGTON MONTHLY: "How the Decline in Investigative Journalism Is Making Congress Dumb" (January 30, 2017)
- SHORENSTEIN CENTER AT HARVARD: "2017 Goldsmith Book Prizes" (January 30, 2017)
- THE NATION: "But who will cover the swill milk? A new book examines the dire future for investigative reporting in America." (February 1, 2017)
- THE SURABAYA REVIEW OF BOOKS: "Menakar Dampak Jurnalisme Investigatif" (February 11, 2017)
- REASON FOUNDATION: "Where Did All the Investigative Journalism Go?" (February 22, 2017)
- THE NEWS & OBSERVER: "How investigative reporting creates huge public benefits" (March 17, 2017)
- THE NEWS & OBSERVER: "Power of investigative journalism is celebrated in new book and Raleigh event" (March 24, 2017)
- COLUMBIA JOURNALISM REVIEW: "Book aims to pin down economic return on investigative reporting" (April 11, 2017)
- LOS ANGELES REVIEW OF BOOKS: "How Journalists Should Really Challenge Trumpism" (April 30, 2017)
- NEUE ZÜRCHER ZEITUNG: "Unprofitable detectives of democracy" (May 6, 2017)
- PERSOENLICH.COM: "U.S. citizens believe in the mission of the New York Times" (May 15, 2017)
- THE READING LIST: "James T. Hamilton - The Reading List Of An Academic Scribbler" (June 5, 2017)
“In riveting detail, Hamilton meticulously examines the storied history of investigative journalism in America, chronicles its current malaise, and makes a convincing case that pouring resources into gumshoe reporting makes economic sense for sclerotic news organizations. Why? Because readers hunger for more of it and are willing to pay to read it.” —Walter V. Robinson, Pulitzer Prize–winning investigative journalist and Editor-at-Large at the Boston Globe
“This is an outstanding book, the product of careful thinking, of remarkable and painstaking gathering of data on investigative reporting—past and present—that no one in academia or in journalism has ever undertaken before. It is a moving, evidence-based affirmation of the value of journalism to democracy.” —Michael Schudson, Columbia University
“Democracy’s Detectives is essential reading for anyone interested in the economics of news, and it is a master class in methodological creativity and ingenuity in conducting social science research.” —Philip M. Napoli, Duke University