Grading PolitiFact: Joe Biden and the Flint crime rate

To assess the truth for a numbers claim, the biggest factor is the underlying message.
--PolitiFact editor Bill Adair

The issue:

(clipped from PolitiFact.com)

The fact checkers:

Angie Drobnic Holan:  writer, researcher
Sue Owen:  researcher
Martha Hamilton:  editor

Analysis:

This PolitiFact item very quickly blew up in their faces.  The story was published at about 6 p.m. on Oct. 20.  The CYA was published at about 2:30 p.m. on Oct. 21, after FactCheck.org and the Washington Post published parallel items very critical of Biden.  PolitiFact rated Biden "Mostly True."

First, the context:



(my portion of transcript in italics, portion of transcript used by PolitiFact highlighted in yellow):

BIDEN:

If anyone listening doubts whether there is a direct correlation between the reduction of cops and firefighters and the rise in concerns of public safety, they need look no further than your city, Mr. Mayor.  

In 2008--you know, Pat Moynihan said everyone's entitled to their own opinion, they're not entitled to their own facts.  Let's look at the facts.  In 2008 when Flint had 265 sworn officers on their police force, there were 35 murders and 91 rapes in this city.  In 2010, when Flint had only 144 police officers the murder rate climbed to 65 and rapes, just to pick two categories, climbed to 229.  In 2011 you now only have 125 shields.  

God only knows what the numbers will be this year for Flint if we don't rectify it.  And God only knows what the number would have been if we had not been able to get a little bit of help to you.

As we note from the standard Bill Adair epigraph, the most important thing about a numbers claim is the underlying message.  Writer Angie Drobnic Holan apparently has no trouble identifying Biden's underlying message (bold emphasis added):

If Congress doesn’t pass President Barack Obama’s jobs plan, crimes like rape and murder will go up as cops are laid off, says Vice President Joe Biden.

It’s a stark talking point. But Biden hasn’t backed down in the face of challenges during the past week, citing crime statistics and saying, "Look at the facts." In a confrontation with a conservative blogger on Oct. 19, Biden snapped, "Don’t screw around with me."

No doubt the Joe Biden of the good "Truth-O-Meter" rating is very admirable in refusing to back down.  The "conservative blogger" is Jason Mattera, editor of the long-running conservative periodical "Human Events."  You're a blogger, Mattera.  PolitiFact says so.

But back to shooting the bigger fish in this barrel.

PolitiFact:

We looked at Biden’s crime numbers and turned to the Federal Bureau of Investigation's uniform crime statistics to confirm them. But the federal numbers aren’t the same as the numbers Biden cited. (Several of our readers did the same thing; we received several requests to check Biden’s numbers.)

When we looked at the FBI’s crime statistics, we found that Flint reported 32 murders in 2008 and 53 murders in 2010. Biden said 35 and 65 -- not exactly the same but in the same ballpark.

Drobnic Holan initially emphasizes a fact check of the numbers.  Compared to the FBI numbers, Biden inflated the murder rate for both 2008 and 2010, and his inflated set of numbers in turn inflates the percentage increase by 45 percent (or 27 percentage points, going from 60 percent to 87 percent).  So it's a decent-sized ballpark.

PolitiFact:

For rapes, though, the numbers seemed seriously off. The FBI showed 103 rapes in 2008 and 92 rapes in 2010 -- a small decline. The numbers Biden cited were 91 rapes in 2008 and 229 in 2010 -- a dramatic increase.

If inflating the percentage increase in murders by 27 percentage points is not a problem for Biden then this at least sounds like a problem.

After going over some other reports on the numbers and a surprising discussion of how not much evidence suggests that Obama's jobs bill would address the number of police officers in Flint, PolitiFact returns to the discrepancy between the numbers:

(W)e found that discrepancies between the FBI and local agencies are not uncommon, and they happen for a number of reasons. Local numbers are usually more current and complete, and local police departments may have crime definitions that are more expansive than those of the FBI.

All this is very nice, but we're talking about the city of Flint, here.  We don't really need current stats for 2008 and 2010 because they're well past.  Perhaps that affects the completeness aspect of crime statistics also; PolitiFact's description is too thin to permit a judgment.  As for "expansive" definitions, well, there's a problem with that.  Biden's number of rapes in 2008 is lower than the number reported in the UCR (FBI) data.  That is a counterintuitive result for a more expansive definition of rape and ought to attract a journalist's attention.

In short, even with these proposed explanations it seems as though something isn't right.

PolitiFact:

Flint provided us with a statement from Police Chief Alvern Lock when we asked about the differences in the crime statistics, particularly the rape statistics.

"The City of Flint stands behind the crime statistics provided to the Office of The Vice President.  These numbers are an actual portrayal of the level of violent crime in our city and are the same numbers we have provided to our own community. This information is the most accurate data and demonstrates the rise in crime associated with the economic crisis and the reduced staffing levels.

"The discrepancies with the FBI and other sources reveal the differences in how crimes can be counted and categorized, based on different criteria." (Read the entire statement)

This is a city that's submitting clerical errors to the FBI, and we still have the odd problem with the rape statistics.  If the city can provide numbers to Joe Biden then why can't PolitiFact have the same set of numbers?   And maybe the city can include stats for crimes other than the ones Biden may have cherry-picked?  Not that PolitiFact cares about cherry-picked stats, of course.

Bottom line, why are we trusting the local Flint data sight unseen?

PolitiFact caps Biden's reward with a statement from criminologist and Obama campaign donor James Alan Fox of Northeastern University to the effect that Biden makes a legitimate point that "few police can translate to more violent crime" (PolitiFact's phrasing).  Fox affirms that point, by PolitiFact's account, though it's worth noting that on the record Biden asserted a "direct correlation" between crime and the size of a police force.  The change in wording seems strange for a fact check outfit that maintains that "words matter."

The conclusion gives us nothing new other than the "Mostly True" rating.  Biden was supposedly "largely in line" with the UCR murder data for Flint.  His claim about rape apparently did not drag down his rating much even though PolitiFact admittedly could not "fully" explain the discrepancies.  PolitiFact apparently gave Biden credit for the underlying argument that reductions in a police force "could result in increases in violent crime" despite Biden's rhetoric about a "direct correlation."

The grades:

Angie Drobnic Holan:  F
Sue Owen: N/A
Martha Hamilton:  F

This fact check was notable for its reliance on sources apparently predisposed toward the Obama administration and its relatively unquestioning acceptance of information from those sources.  The Washington Post version of this fact check, for comparison, contacted three experts to PolitiFact's one and none of the three had an FEC filing indicating a campaign contribution to Obama.

And no investigation of whether Biden cherry-picked Flint?  Seriously?  See the "Afters" section for more on that as well as commentary on PolitiFact's CYA attempt.

Afters:

Assuming Biden's numbers had escaped the competent fact checks unscathed, what about his underlying point?  Was he right, based on that presupposition, that we need not look beyond Flint to confirm the direct correlation between shrinking police forces and increasing crime rates?

My peek at the evidence did not seem to help Biden's case.

(chart from CityRating.com, chart uses FBI data)

Are we supposed to believe that Flint's police force was smaller back in 2006?


I also wanted to check Biden's claims about the number of policemen as well as population variance, so I took a series of screen captures including data from Flint as well as some of its alphabetical neighbors for comparison.


Annenberg Fact Check beat me to the employment numbers:

As for the city’s police force, it’s difficult to assess whether Biden was right. The FBI says Flint had 233 full-time law enforcement employees in 2008 and 155 in 2010, a decline of 78 employees or 33 percent. That’s a significant decline, for sure. Biden’s figures, however, show an even greater drop. In Flint, Biden said the number of “sworn officers” declined from 265 to 144, a drop of 46 percent.

The story went on to hypothesize that part-time employees might account for the difference since the FBI data only include full-time employees.

2006:

2007

2008

2009

2010

The data start including population data in 2008 (first column).  The next column (first column for 2006 and 2007) represents the total sworn employees, civilians included.  The next two columns represent officers and civilians, respectively.

The snippets don't seem to contain anything earth-shaking.  I mainly posted them to save the work if somebody's curiosity mirrored mine.  I find it plausible that Biden drew his employment figures from the prior year (2007), though factcheck.org may be right about the FBI not counting part-time employees.  For what it's worth, I saw nothing of that in the employment definitions.


PolitiFact's CYA

What to do when other liberal fact check sites are out of step with your rating?  CYA!

After reading the three reports closely, we began to understand how we arrived at different conclusions, even though there were no obvious errors of fact-checking to correct.

I fail to detect any mystery.  It's a plain case of disagreeing on the selection bias.  Jeff Dyberg put his finger on it very promptly in a comment at PolitiFact's Facebook page (bold emphasis replaces Dyberg's use of asterisks):

How convenient that they selected the statement that showed Biden in the best light.

How exactly does this work then when PF is constantly telling its readers to look at a persons profile to gauge that persons honesty? Had they selected Biden(')s other comments, it would have lowered him on the "Truth Index." Instead, he's given a pass, and PF would have left it that way except for the WaPo article.

And that my friends is what you call "Selection Bias."

And of course PolitiFact magnified the bias problem with its kid gloves treatment of Biden's underlying message--in contradiction of a principle described by PolitiFact editor Bill Adair. None of the three fact checks failed to correctly identify Biden's underlying message. PolitiFact is the one that gave Biden a pass.

And now PolitiFact's brilliant stroke:

Here, we want to highlight a few of the differences between our reports and ask what you think.

Hmmm.  Ask the predominantly liberal PolitiFact readership what it thinks about the situation.  What could possibly go wrong with that?

The good news is that even with the lame brained plan of surveying the readership for suggestions, PolitiFact pulls a few lessons from the debacle:

It matters what specific statement fact-checkers choose to check.

Yes!  And, as I have often pointed out, this aspect of selection bias warrants a strong disclaimer from PolitiFact about using its collected ratings about individuals or groups to draw conclusions.  Yet PolitiFact continues to do special stories encouraging readers to consider a public figure's Truth-O-Meter "report card" and no such disclaimer appears. 

Causal claims about complex social phenomena remain a fact-checking challenge.

Yes!  And if a researcher has an ideological bias (whether publicly known or not), it increases the chance of acceptance or rejection of a causal claim based on poor evidence.   As may be the case with the Biden claim, the researcher may feel it self-evident that more police equals less violent crime and not bother to test the claim while researching the story.  The editor ought to help prevent that occurrence, but if the editor shares the same or a similar ideology then a deeply flawed story may result.

Independent fact-checkers often agree -- but not always.

This one goes without saying, but the important thing is the source of the error.  Unfortunately PolitiFact's explanation seems to pin the blame on subjectivity.

Subjectivity doesn't really belong in a fact check, does it?

Perhaps one day it will occur to PolitiFact that methods exist to prevent this sort of thing from happening.  And maybe then we'll get a real mea culpa from them instead of CYA.



Oct. 26, 2011:  At the start of the "Afters" section replaced twin if's with an "Assuming."