PolitiFact's numbers racket

Lately I've used a quotation from PolitiFact's founder and editor Bill Adair to preface some of my individual grades of PolitiFact fact checks.

To assess the truth for a numbers claim, the biggest factor is the underlying message. 
--Bill Adair

I started using that quotation to help emphasize where it appears PolitiFact has fallen short of Adair's aspirations.

PolitiFact handles numbers claims in a ridiculously inconsistent manner.  In this post I'll contrast PolitiFact's handling of a claim involving Ron Paul with a recent claim made by Mitt Romney at CPAC.  I'll use Bill Adair's explanation of the Paul rating to establish a contrast with the method used to derive Romney's "Truth-O-Meter" rating.

Adair:

Our first encounter with a numbers problem came last September when Ron Paul cited the war death toll in a debate. "We've lost over 5,000 Americans over there in Afghanistan, in Iraq, and plus the civilians killed," Paul said.

Like Rep. Paul, Romney made a claim about numbers:

Today there are more men and women out of work in America than there are people working in Canada.

Granted, there's a difference between the claims.  Paul specifies his numbers while Romney pegs his to a statistic about Canada that probably isn't widely known but that people would imagine is high given the size and population of Canada.  Is that difference significant?  I wouldn't think so, but PolitiFact may have reasons for treating the two differently.  Just in case we'll ready an alternative quotation for Rep. Paul to use for the sake of argument:

We've lost more Americans over there in Afghanistan, in Iraq, and plus the civilians killed, than we could seat at Dublin Scioto Stadium.
--Paul quotation fabricated for the sake of argument

Dublin Scioto Stadium, for those reluctant to click the URL, has a capacity of 5000.

PolitiFact found that Paul was incorrect about the number of Americans killed under Paul's conditions, calculating the number at 4349 as of the date Paul made his claim.

PolitiFact found that Romney was incorrect about the number of Americans out of work in relation to the number working in Canada.  PolitiFact found that 16.7 million Americans were out of work compared to PolitiFact's figure of 17.2 million working Canadians.

Paul's figure inflated the actual figure by about 15 percent according to PolitiFact's figures, and Paul received a "Mostly True" rating from the "Truth-O-Meter."

Romney's figure inflated the actual figure by about 3 percent using PolitiFact's numbers.  The "Truth-O-Meter" found Romney "False."

The unenlightened reader may be wondering "How could that be?"

There are a number of possible explanations, not that I think any of them are necessarily any good.

1)  Paul would have been rated "False" if he had used Dublin Scioto Stadium as his benchmark.
2)  PolitiFact is grading Romney on his explanation rather than on the claim.
3)  Paul's underlying point was much more accurate than Romney's.

I assume out of charity that the first explanation is too ridiculous to consider.

As for the second explanation, it seems more than slightly misleading to grade Romney on the explanation rather than on the claim itself.  Especially when PolitiFact creates graphics such as the following to appear with the story:

click image to enlarge

Compare a corresponding blurb related to Paul's claim:

click image to enlarge

Using the second justification makes PolitiFact look inconsistent, inaccurate, or both.

But we have reason for hope.  Bill Adair, the creator of PolitiFact, has declared the underlying point the most important thing when fact checking number claims.   The third possible explanation.

What underlying point did PolitiFact identify in Paul's claim?  Here's how Adair expressed it:

To assess the truth for a numbers claim, the biggest factor is the underlying message. In Paul's case, his point was a simple one, that many people have died in the wars in Iraq and Afghanistan. He overstated the number, but not by all that many.

And Romney's underlying point?

Unfortunately, I was unable to identify any evidence in the story by Louis Jacobson that an effort was made to identify the underlying point.  If the point was that many Americans are unemployed then it's hard to see why Romney wasn't graded "Mostly True" or better, given that the figures PolitiFact provided put Romney's claim much closer to accuracy by percentage than Paul's.

But maybe there's a fourth explanation:

4)  PolitiFact has changed its standard (and forgot to tell us?).

What's the evidence in favor of the fourth explanation?  It's thin, but potentially compelling:  Bill Adair was Jacobson's editor for the Romney story.

Seriously, if the underlying point is the most important thing about claims involving numbers, then shouldn't we expect PolitiFact to always identify an underlying argument whenever possible?