Stultus est sicut stultus facit*

Oh hum, another junk science study about guns. People are talking about a study (a few years old, actually) that purports to measure the "externalities" - costs to society - of gun ownership.  Philip Cook and Jens Ludwig write in the Journal of Public Economics:

This paper provides new estimates of the effect of household gun prevalence on homicide rates, and infers the marginal external cost of handgun ownership. The estimates utilize a superior proxy for gun prevalence, the percentage of suicides committed with a gun, which we validate. Using county- and state-level panels for 20 years, we estimate the elasticity of homicide with respect to gun prevalence as between +0.1 and + 0.3. All of the effect of gun prevalence is on gun homicide rates. Under certain reasonable assumptions, the average annual marginal social cost of household gun ownership is in the range $100 to $1800.

I read through this paper and it is entirely unconvincing.  The authors have chosen what they claim is a "superior proxy for gun prevalence" - the suicide rate.  The provide no discussion or justification other than a simple claim of superior technique, along with a couple of references to other articles.  OK, fine.

Here's why this is junk science: we have direct measurements of gun prevalence that have been gathered over the last 15 years or so.  These are not proxies (indirect measurements of something related to but not identical with what is being tracked), rather they are direct quantitative data measuring the actual prevalence of firearms:

• The number of background checks that gun stores run through the NCIS is a direct measurement.  These do not get run if someone is not purchasing a firearm.
• The number of Concealed Carry Permits issued by the States is a direct measurement.  People don't bother applying if they do not want a firearm handy (i.e. not just at home but carried with them).
• The number of firearms sold is a direct measure, as reported by publicly held manufacturers.  The SEC requires each public company to file annual 10-K reports which contain financial performance data for investors; these reports will list sales figures.

All of these are markedly up in the last 15 years, while the firearms murder rate is sharply down.  And yet the authors use an indirect measure (suicide rate) as a proxy for firearms prevalence.  Why?

The suspicion is that this is just the latest in a long line of academic studies purporting to "prove" that more guns result in more violence.  The "proofs" have been, shall we say, less than scientifically rigorous?

The scholar we have to thank most for this awakening is Michael Bellesiles, the author of Arming America: The Origins of a National Gun Culture (September 2000). In looking back on the public debate that led up to the Heller ruling, I can think of no other single person who did so much (even if inadvertently) to change the political climate around gun rights.

Bellesiles sought to to show that the bearing of civilian firearms had not been a central feature of life in the first century of the U.S’s history; that American gun culture had been founded on a myth, and the “truth” denied it political legitimacy. His book got a favorable reception (the Bancroft Prize, glowing reviews, near-unanimous praise in the national press) because it told the media-political elite what most of it wanted to hear, that the Second Amendment was an anachronism being defended by dupes and frauds.

Unless you’ve been living under a rock, you know the punchline: it was Bellesiles himself who was the fraud. His conclusions were unsupported; his ‘evidence’ was a tissue of deliberate misconstruction and outright fakery. His Bancroft prize was eventually withdrawn for “scholarly misconduct”

Raymond is only warming up, and continues with a Hall Of Shame from the academic community:

Noel Perrin’s 1979 book Giving Up The Gun. Perrin argued that Toyotomi Hideyoshi’s 1543 ban on firearms was successful and a key factor in the Tokugawa pacification of Japan. Implicitly he proposed this as a model for emulation. But, as an eminent historian of Japan once explained to me, it is known for certain that the great daimyos continued to equip the retainers with firearms after the ban. The truth is that the shogunate merely pretended to abolish firearms and the daimyos pretended to obey, a very Japanese face-saving maneuver. To support his conclusion, Perrin must either have ignored or outright suppressed the most obvious kind of primary documentary evidence — the actual weapons inventories from the Tokugawa period.

The 1986 Kellerman & Reay study Protection or peril? An analysis of firearm-related deaths in the home is the source of the widespread myth that home-defense weapons are 43 times more likely to kill or injure family members than a criminal. Dave Kopel’s refutation, devastating as it is, fails to mention that Kellerman has refused to disclose his entire primary data sets to peers so his statistical analysis could be checked. Kellerman was later a vocal defender of the Arming America fraud; perhaps his data sets are swimming with Bellesiles’ nonexistent probate records?

Kellerman and Reay are nothing if not consistent. Two years later, their 1988 Handgun Regulations, Crime, Assaults, and Homicide: A Tale of Two Cities blatantly manipulated and misrepresented data from Seattle and Vancouver in an attempt to argue for the effectiveness of Vancouver’s firearms restrictions. Among other failings, it omitted to control for socioeconomic and ethnic differences between the cities, and it ignored the actual 25% increase in Vancouver murder rates after the law.

Colin Loftin’s 1991 paper Effects of restrictive licensing of handguns on homicide and suicide in the District of Columbia failed to control for population changes between 1976 and 1987 and fraudulently ignored a doubling of D.C’s murder rate after 1978 that earned it the sobriquet “Murder Capital of the World” even as gun-ban advocates were citing the Loftin study as evidence of the success of their policy. Refutation here.

He continues with more examples and you should RTWT.  And so back to Cook and Ludwig.  Is it accidental that they chose to ignore direct measurement measurement of household gun prevalence and rather chose to rely on a proxy whose validity is of conjectural accuracy?  Presumably, the "scientists" who peer-reviewed their paper did not challenge them on this (we can't know because these comments are rarely published, but this issue is not addressed in the paper).  Presumably the editor of the journal had no objections to the suppression of actual data and its replacement by an inferior proxy data set.

This alone is enough to condemn the paper to the trash can as scientifically worthless.  It doesn't matter that the paper is larded with references to statistical significance and fancy regression mathematics.

If your data is junk, the output of your regression is junk, p > 0.95.

But back to Raymond's analysis, which suggests motive:

I described the errors as “systematic” before the jump because there is a pattern of distortions in the anti-gun literature that have been repeated over decades even though they violate known good practice in the social and medical sciences. These include but are not limited to:

• Failure to control for socioeconomic differences between star and control groups, even when the differences are known to correlate with large differences in per-capita rates of criminal deviance
  
  
• Choice of study periods that ignore well-documented trends that run contrary to the study’s conclusions immediately before or after the period.
  
  
• Selective use of suicide statistics, counting them only in star but not control groups and/or ignoring massive evidence that would-be suicides rapidly substitute other methods when firearms are not available.
  
  
• Tendentious misapplication of Uniform Crime Report data, for example by ignoring the fact that UCR reports of homicides are entered before trial and therefore fail to account for an unknown but significant percentage of findinga of misadventure and lawful self-defense.
  

And I described this pattern as “fraud” before the jump because the magnitude of these errors would be too great and their direction too consistent for honest error, even if we did not in several prominent cases have direct evidence that the fraud must have been intended.

The alert reader will note that at least two of his highlighted bullet points are present in Cook & Ludwig.  Raymond's prediction applies, at least close enough to suggest predictive value.  And so there are two conclusions that we can draw from our analysis of academic scholarship on the topic of gun violence:

1. The Academy is, as has been claimed, entirely biased in favor of liberal policy positions (including gun control), and researchers will knowingly ignore evidence that tends to debunk gun control initiatives, or

2. The Academy is incompetent to produce scientifically valid studies on the subject, as demonstrated by the long and sordid record described here and in Raymond's post.

The conclusion is inescapable, and it is entirely irrelevant as to which cause we take as most likely: all academic papers discussing firearms should be dismissed out of hand as almost certain to be junk.

It is not required to ascribe motives (knowing fraud) for this rule of thumb to be reliable.  As with climate science we see a continual advancement of weak evidence supporting Catastrophic Anthropogenic Global Warming that does not stand up to close scrutiny, and a rejection of strong evidence falsifying CAGW.  Did Cook & Ludwig do this knowingly?  Who knows?  But they did a splendid job of hiding the decline in gun violence during their study period.

The latest in a long line of junk is still junk.

* The common translation is "stupid is as stupid does".  I don't think that this is taught at the Academy anymore.  This actually is a recursive bit of snark in this bootnote, if you really think about it.