“unstable and potentially unreliable”
By Sean Campbell, Daniel Nass & Mai Nguyen
The Trace (10/4/19)
For journalists, researchers and the general public, the Centers for Disease Control and Prevention serves as an authoritative source of information about Americans’ health, including estimates of how many people are killed or injured by guns. The agency’s most recent figures include a worrying uptick: Between 2015 and 2016, the number of Americans nonfatally injured by a firearm jumped by 37 percent, rising from about 85,000 to more than 116,000. It was the largest single-year increase recorded in more than 15 years.
But the gun injury estimate is one of several categories of CDC data flagged with an asterisk indicating that, according to the agency’s own standards, it should be treated as “unstable and potentially unreliable.” In fact, the agency’s 2016 estimate of gun injuries is more uncertain than nearly every other type of injury it tracks. Even its estimates of BB gun injuries are more reliable than its calculations for the number of Americans wounded by actual guns.
An analysis performed by FiveThirtyEight and The Trace found that the CDC’s report of a steady increase in nonfatal gun injuries is out of step with a downward trend we found using data from multiple independent public health and criminal justice databases. That casts doubt on the CDC’s figures and the narrative suggested by the way those numbers have changed over time.
In response to a detailed list of questions and an analysis memo showing that there may be issues with its gun injury data, a CDC spokesperson said in an email that the scientists involved in gathering and analyzing the data “are confident that the sampling and estimation methods are appropriate.”
To produce its estimates of gun injuries, the CDC uses data collected by the Consumer Product Safety Commission. We asked the commission about the single-year jump of 31,000 gun injuries, and a spokesperson replied, “Although visually, the [CDC] estimates for firearm-related assaults appear to be increasing from 2015 to 2016, there is not a statistically significant difference between the estimates.”
Over a dozen public health researchers reviewed The Trace and FiveThirtyEight’s analysis and said that the inaccuracy of the CDC gun injury data has serious implications for the national-level understanding of gun violence.
Hidden epidemic of firearm injury
“No one should trust the CDC’s nonfatal firearm injury point estimates,” said David Hemenway, director of the Harvard Injury Control Research Center.
But many researchers have trusted these numbers, or at least referenced them. Since 2010, at least 50 academic articles have cited the CDC’s gun injury estimates. Last year, for example, the American Journal of Epidemiologypublished a paper that used CDC data to conclude that there was a “hidden epidemic of firearm injury.”
“For those of us who are doing this kind of research, it’s disconcerting,” said Priscillia Hunt, a researcher at the RAND Corporation, a nonprofit research organization. “With the CDC, there’s this general assumption that they are reliable and have good data.”
Hunt herself cited the estimates in the introduction of a policy paper she published last year.
“It feels like a gotcha moment for people who’ve used it,” she said.
One likely reason that researchers frequently cite the data is that it’s easily accessible online through the CDC’s Web-based Injury Statistics Query and Reporting System, or WISQARS. Measures of the estimates’ uncertainty are also available through the system, but they are hidden by default.
Kristen Moore, a professor of mathematics at the University of Michigan, recently cited the CDC’s gun injury estimates in an article for an actuarial trade publication. She was surprised to see that trends from other sources of data pointed in the opposite direction from the trend in the CDC’s data. …