Yet more bogus evidence from truth-hustler Chris Mooney

We should expect Chris Mooney to know something about science and epistemology.

Get used to disappointment.

I've classed Mooney as a "truth hustler."  Mooney, posing as a voice of reason, misleads people.  An article published today at "Desmogblog" serves as the latest example.

In the article, titled "More Evidence That Republicans Are More Factually Challenged Than Democrats," Mooney introduces us to the latest evidence:  Republicans fare worse than Democrats according to the past year's statistics from The Washington Post Fact Checker (Glenn Kessler).

What's wrong with that evidence?  Plenty.  And we'll let Mooney help explain it via contrast with a series of wrongheaded statements.

Mooney:

I ... needed evidence from the real world showing that, you know, conservatives or Republicans are more factually incorrect.

That’s where all the fact-checker data came in.

You see, we have paid professionals whose job it is to track just how wrong Democrats and Republicans are.

Mooney, intentionally or not, misses an important distinction.  Journalists are not paid to determine whether Democrats are differentially more wrong than Republicans or vice versa.  Fact-checking journalists are paid to produce stories that interest readers.  Toward that end, fact-checking journalists use editorial judgment to find potentially popular stories.

In scientific terms, fact checking journalists purposely employ a form of selection bias in choosing the data set they choose to fact check.

If Mooney's bio isn't filled with lies then he has to know about selection bias and its implications for would-be scientific conclusions.  A study that fails to control for selection bias provides poor evidence for any conclusion it attempts to support.

Mooney:

I find these fact-checker data particularly compelling, by the way, for the following reason: Neither PolitiFact nor Glenn Kessler (who writes the Post’s column), think of themselves as liberal partisans.

Lovely.  We might as well favor the findings of scientists who do not perceive their own bias.

If Kessler and the PolitiFact folks thought of themselves as partisan, maybe it would increase the chances they would institute controls on selection bias.  Mooney's self-confessed reassurance is misplaced.  In truth, the self-perception of bias is irrelevant.  A properly done study controls for bias so that the point of view of the researcher is irrelevant.  But Kessler and PolitiFact apparently do absolutely nothing to control for selection bias.

I would argue that both go too far in trying to ding Democrats and liberals, just to make themselves appear balanced (and, presumably, to keep getting their calls returned by the other side of the aisle). Therefore, if their data shows Republicans fare worse, that really says something.

As evidence, Mooney first presented his prediction that Kessler would react to the last half year's findings with a resulting change in the numbers indicating Kessler's attempt to correct for the potential perception of bias.

Mooney was wrong.  The new data looked pretty much like the old data.

Mooney, undaunted by the failure of his prediction, found even more reason to cleave to his theory.  After all, if the numbers look the same this year six months as last, doesn't this suggest that Kessler actually treated Democrats unfairly in order to obtain that result?

Mooney:

(T)he data are all the more striking in that I believe Kessler inflates the Democrat total, intentionally or otherwise.

At least Mooney tries to support his statement with evidence.  Unfortunately the evidence remains pathetically weak.

1)  Kessler gave President Obama four "Pinocchios" for misattributing a quotation to Rutherford B. Hayes.

2)  Kessler didn't even bother to rate Sarah Palin's statement that Paul Revere "warned the British."

Two anecdotes making up one comparison?  Seriously, that's all we get?

Mooney does make a late-breaking feint toward reason:

(G)iven that this difference does not appear to be a fluke, there has to be a reason for it to exist. And I agree, one could posit many explanations: Maybe there is a selection bias with respect to which items are being rated. Maybe Kessler himself is biased against Republicans.

Maybe there is a selection bias?

In science we expect researchers to take affirmative steps to eliminate selection bias.  Lacking evidence of such steps we assume that the evidence is tainted by selection bias, lest we draw unfounded conclusions.  Or if you're Chris Mooney just go ahead and draw the unfounded conclusion while allowing that maybe there's selection bias.

Mooney:

But the point is, there has to be an explanation. It just won’t do to say, as Kessler does, that since both Dems and Republicans score above Two Pinocchios, both ought to be ashamed of themselves. The data are telling us something more than that—something stronger than that.

The answer to the question isn't that tough.  The data are not designed to give us a dependable differential breakdown for the claims of Democrats against those of Republicans.  No serious researcher would try to answer that question while relying on left-leaning journalists with no control for selection bias.  But the data do have an obvious use.  The data directly measure how fact checkers choose subject matter.  Kessler and PolitiFact give us an excellent tool measuring their selection bias.  Add to that the ratings each fact checker provides and we end up with data that can help us measure media bias.

As for Chris Mooney, I can only wonder what cognitive processes he uses to either miss or de-emphasize the obvious.


Correction 7/10/2012:  The totals for Glenn Kessler's Pinocchio count apparent came from a six-month period rather than from a full year. The text was altered to reflect the shorter period of time.