PolitiFact and playing the percentages (Updated)

American Journalism Review's editor and senior vice president Rem Rieder is back at it again, writing about PolitiFact in a way that demands a response.

Last time, Rieder argued in effect that journalists' role as truth-tellers compelled them to call out lies.  Rieder blurred the line between a "lie" as an inaccurate statement and a "lie" as an attempt to deceive and wound up catching himself in his own criticism.  In my response I argued that objective journalism doesn't permit reporting motives based on misstatements of fact.

This time, Rieder broadcasts his disappointment with PolitiFact's ruling of "Mostly True" on Sen. Marco Rubio's claim that a majority of Americans are conservative.

As it assessed the Florida Republican's assertion, PolitiFact turned to the findings of the Gallup Poll, which regularly asks Americans about their political orientation. In 2011, Gallup found that 40 percent considered themselves conservative, 35 percent moderate and 21 percent liberal. It also found that the number calling themselves conservative has never been above 50 percent.

So this one should be easy. According to Merriam-Webster's Collegiate Dictionary, a majority is "a number or percentage equaling more than half of a total." Not even close. The number of conservatives would have to be more than 25 percent higher to be a majority. The verdict on the PolitiFact Truth-O-Meter is "Pants on Fire," right?

Can we blame Rieder for ignoring the context of Rubio's statement if PolitiFact did the same thing?  Of course we can, kids!  As I pointed out in an analysis over at the associated blog "PolitiFact Bias," Rubio spoke in the context of preferring conservative over liberal.  Here's part of the quotation, via PolitiFact:

"You know, somebody asked me: ‘How do you know that? How do you know Americans are majority conservative?’ Here’s why: How come liberals never admit that they are liberals? They never admit it. They’ve now come up with a new word called progressive, which I thought was an insurance company, but apparently it’s a label."

PolitiFact nearly hit on an approach to the fact check that might have tested what Rubio was claiming.  PolitiFact took into consideration the political leaning of Independents, reporting that "45 percent of Americans identified as Republicans or leaned that way."

Adding the conservative lean of moderates to the percentage of self-identified conservatives probably represents the best method of testing Rubio's claim.

A short illustration will help show why.

Suppose we used a self-identification poll with 50 choices.  Conservatives, for the sake of argument, represent only 4 percent of the total, with the other 36 percent who chose "conservative" out of only three choices picking some other label.

Do conservatives make up 40 percent of the total or just 4 percent?

The number of choices matters.

For that reason, limiting the choices to match the nature of Rubio's claim constitutes the best method of testing the claim.  Calculating the conservative lean of moderates added to self-identified conservatives is a fair measurement.  Gallup doesn't ask that question of moderates, so we can't use that stat to test Rubio's claim.  As a result we don't really know the answer.  My guess is that the conservative lean of moderates would exceed 10 percent of the total and put conservatives over the 50 percent bar.  Therefore I think PolitiFact's rating is reasonably accurate but flawed in its reasoning.  But worse still is the reasoning coming from critics on the left.  Can the statement be reasonably termed "False" based on self-identification surveys with three choices?

Rieder clearly thinks so.  And maybe he'd just as willingly accept a self-identification survey with 50 choices in deciding the question.

Oops, did I lump Rieder in with critics on the left?  But he's an objective journalist, isn't he?

I don't think so.  If Rieder was objective he might have taken the opportunity to criticize PolitiFact's "Mostly True" rating for White House representative Cecilia Muñoz's claim that 98 percent of Catholic women have used contraception.  That claim is off by at least the 10 percentage points representing women who have never had sex at all; it simply couldn't be a true claim at all in contrast to Rubio's, and the degree of error is at least similar to that Rieder finds an obvious "Pants on Fire" for Rubio.

So where's Rieder's objection?

I'm not holding my breath.


Update 3:40 p.m.

Kevin Drum of the liberal rag Mother Jones points to survey data supportive of Marco Rubio's statement:

In this poll, 61% of the country identified as conservative. That's a majority!

Now, this is a poll of "likely" voters. It forces a choice between liberal and conservative. And even though we all know that "independent" voters mostly lean left or right pretty reliably, I imagine this poll still understates the number of true centrists.

Nonetheless, it's a poll. And Rubio could quite reasonably point to it as evidence that a majority of the country is conservative.

The Politico/George Washington University survey samples likely voters, so it isn't quite the perfect support for Rubio.  Of significance, however, Democrats and Democrat-leaners constitute a plurality in the sample population.  Drum fails to emphasize that key point while mentioning a few less significant complications.