Yet more bogus evidence from truth-hustler Chris Mooney

We should expect Chris Mooney to know something about science and epistemology.

Get used to disappointment.

I've classed Mooney as a "truth hustler."  Mooney, posing as a voice of reason, misleads people.  An article published today at "Desmogblog" serves as the latest example.

In the article, titled "More Evidence That Republicans Are More Factually Challenged Than Democrats," Mooney introduces us to the latest evidence:  Republicans fare worse than Democrats according to the past year's statistics from The Washington Post Fact Checker (Glenn Kessler).

What's wrong with that evidence?  Plenty.  And we'll let Mooney help explain it via contrast with a series of wrongheaded statements.

Mooney:

I ... needed evidence from the real world showing that, you know, conservatives or Republicans are more factually incorrect.

That’s where all the fact-checker data came in.

You see, we have paid professionals whose job it is to track just how wrong Democrats and Republicans are.

Mooney, intentionally or not, misses an important distinction.  Journalists are not paid to determine whether Democrats are differentially more wrong than Republicans or vice versa.  Fact-checking journalists are paid to produce stories that interest readers.  Toward that end, fact-checking journalists use editorial judgment to find potentially popular stories.

In scientific terms, fact checking journalists purposely employ a form of selection bias in choosing the data set they choose to fact check.

If Mooney's bio isn't filled with lies then he has to know about selection bias and its implications for would-be scientific conclusions.  A study that fails to control for selection bias provides poor evidence for any conclusion it attempts to support.

Mooney:

I find these fact-checker data particularly compelling, by the way, for the following reason: Neither PolitiFact nor Glenn Kessler (who writes the Post’s column), think of themselves as liberal partisans.

Lovely.  We might as well favor the findings of scientists who do not perceive their own bias.

If Kessler and the PolitiFact folks thought of themselves as partisan, maybe it would increase the chances they would institute controls on selection bias.  Mooney's self-confessed reassurance is misplaced.  In truth, the self-perception of bias is irrelevant.  A properly done study controls for bias so that the point of view of the researcher is irrelevant.  But Kessler and PolitiFact apparently do absolutely nothing to control for selection bias.

I would argue that both go too far in trying to ding Democrats and liberals, just to make themselves appear balanced (and, presumably, to keep getting their calls returned by the other side of the aisle). Therefore, if their data shows Republicans fare worse, that really says something.

As evidence, Mooney first presented his prediction that Kessler would react to the last half year's findings with a resulting change in the numbers indicating Kessler's attempt to correct for the potential perception of bias.

Mooney was wrong.  The new data looked pretty much like the old data.

Mooney, undaunted by the failure of his prediction, found even more reason to cleave to his theory.  After all, if the numbers look the same this year six months as last, doesn't this suggest that Kessler actually treated Democrats unfairly in order to obtain that result?

Mooney:

(T)he data are all the more striking in that I believe Kessler inflates the Democrat total, intentionally or otherwise.

At least Mooney tries to support his statement with evidence.  Unfortunately the evidence remains pathetically weak.

1)  Kessler gave President Obama four "Pinocchios" for misattributing a quotation to Rutherford B. Hayes.

2)  Kessler didn't even bother to rate Sarah Palin's statement that Paul Revere "warned the British."

Two anecdotes making up one comparison?  Seriously, that's all we get?

Mooney does make a late-breaking feint toward reason:

(G)iven that this difference does not appear to be a fluke, there has to be a reason for it to exist. And I agree, one could posit many explanations: Maybe there is a selection bias with respect to which items are being rated. Maybe Kessler himself is biased against Republicans.

Maybe there is a selection bias?

In science we expect researchers to take affirmative steps to eliminate selection bias.  Lacking evidence of such steps we assume that the evidence is tainted by selection bias, lest we draw unfounded conclusions.  Or if you're Chris Mooney just go ahead and draw the unfounded conclusion while allowing that maybe there's selection bias.

Mooney:

But the point is, there has to be an explanation. It just won’t do to say, as Kessler does, that since both Dems and Republicans score above Two Pinocchios, both ought to be ashamed of themselves. The data are telling us something more than that—something stronger than that.

The answer to the question isn't that tough.  The data are not designed to give us a dependable differential breakdown for the claims of Democrats against those of Republicans.  No serious researcher would try to answer that question while relying on left-leaning journalists with no control for selection bias.  But the data do have an obvious use.  The data directly measure how fact checkers choose subject matter.  Kessler and PolitiFact give us an excellent tool measuring their selection bias.  Add to that the ratings each fact checker provides and we end up with data that can help us measure media bias.

As for Chris Mooney, I can only wonder what cognitive processes he uses to either miss or de-emphasize the obvious.


Correction 7/10/2012:  The totals for Glenn Kessler's Pinocchio count apparent came from a six-month period rather than from a full year. The text was altered to reflect the shorter period of time.

Another truth-hustler: Eric Alterman

What is a "truth-hustler"?

I've defined it as one who writes on the issue of truth while spinning on behalf of a partisan agenda.

Eric Alterman, formerly part of the left's media influence machine Media Matters for America, comes to our attention as another example of a truth-hustler via his "As a Matter of Fact" entry in his "Think Again" series at the Center for American Progress.

I'm skipping the portion dedicated to Alterman's obsession with Koch Industries.  That leaves us plenty to work on.

Reality-biased?

Alterman presents us with the proposition that "reality has a liberal bias" may serve as a valuable device in explaining (or at least describing) a supposed tendency for conservatives to favor ideology over fact.  The trick, of course, is for Alterman to make his case without favoring his ideology over the facts (hat tip to Dr. Sanity, who is among those to beat me to coining the term "reality-biased" as a play on Left's "reality-based" mantra). 

Does he have good evidence?

Working the refs?

Without seeming self-conscious about either the aim of his column or his history with Media Matters, Alterman tells us that conservatives are effectively working the media referees.  Instead of providing any example in support of his claim, Alterman offers up this odd portrayal of Mark Hemingway's critique of media fact checking:

As The Weekly Standard online editor Mark Hemingway complains, when such fact-checking organizations tend to find many more conservative lies than liberal ones, rather than respond that conservatives tell far more lies, the fact-checkers go looking for liberal fabrications and find them whether they exist or not. How else could the priests of false equivalence maintain their law that "both sides do it" -- what I call "on-the-one-handism" -- which has proven to be a fundamental tenet of Beltway faith?

Somebody please read Hemingway's article and find the complaint described by Alterman, for I can find no evidence of it, unless Alterman only intended the "such fact-checking organizations tend to find more conservative lies than liberal ones" to represent the entirety of Hemingway's relevant complaint.


The refs want to look fair!

Before we know it, Alterman has drifted from conservatives supposedly working the refs to the fact-checkers' attempts to appear fair (Alterman, of course, isn't working the refs!):

The American Prospect's Paul Waldman discerned this dynamic at work in 2011, when he bravely predicted that yet another of these fact-checking organizations, PolitiFact, would choose as its (gimmicky) lie of the year a "'lie' told by Democrats, even if the one it picks is far from the most egregious lie told this year, or even really a lie at all" -- owing to the fact that the previous two years, Republicans had been caught fibbing. Three times in a row is enough to cry partisanship whether it exists or not, so Waldman was on pretty strong ground despite the damage that such a strategy might do to the organization's reputation for intellectual integrity.

I'll partially grant Alterman's point with this example, except it doesn't appear to serve as an example of  "working the refs."  And if predicting the award would go to a Democrat indicates bravery then pin a medal on my co-editor at PolitiFact Bias, Jeff Dyberg, who confidently predicted an Obama win in 2011 for the "Lie of the Year."  On the other hand, the "Lie of the Year" isn't a typical fact check so it doesn't serve Alterman as a particularly strong example, particularly when every past "Lie of the Year" was prone to the same types of criticisms about its legitimacy.

Alterman's subsequent example, Jon Stewart's claim about Fox News viewers being the most misinformed, fares worse if anything.

Alterman's examples actually suggest that some on the Left take poorly-supported propositions as truth and count against others the failure to equally subscribe to those dubious propositions.

The truth, no spin

Alterman's right that fact checkers want to appear fair, and I would grant that PolitiFact's 2011 "Lie of the Year" may reflect an influence from that motivation.  But studies show that referees are biased.  For example, referees working home games tend to favor the home team.  Is that relevant when audience feedback for fact check stories comes primarily from the political left?  And when journalists as a group tend well left of the general population?

Simply in terms of science, we can't accept without good evidence a narrative from liberals that liberals have a better handle on the truth.  We clearly can't just take their word for it.  We shouldn't accept as evidence studies with rampant selection bias problems.  And we should view with suspicion writers such as Alterman who trumpet the high value of dubious evidence.

The delusions of truth-hustler Chris Mooney

I call the new breed of writers who focus on issues of truth while applying their own spin to truth by the term "truth hustlers."

Chris Mooney is one such.

Mooney writes on a number of subjects concerning science and recently published a book called "The Republican Brain."  I'm focused on Mooney's recent article for the The Nation, since I'm satisfied with the review Mooney's book received from one of his erstwhile colleagues at the Center for Inquiry.

In "Reality Bites Republicans" Mooney continues pushing the meme that liberals are somehow more attuned to the truth than conservatives, with most of his fodder coming from the stats connected to the liberal fact checkers at PolitiFact and the Washington Post.

Mooney:

The Smart Politics blog went on to suggest, based on these numbers, that PolitiFact is biased against the right—precisely the type of knee-jerk centrism that Mann and Ornstein have called into question.

Knee-jerk centrism is hardly worse than knee-jerk liberalism.  So what knee-jerk, if any, are we looking at, here?

I've written a number of times about the Smart Politics study by Eric Ostermeier.  People, apparently including Chris Mooney, don't appear to understand the basis for Ostermeier's criticism.

Ostermeier criticized PolitiFact on the basis of a selection bias problem.  Mooney, cloaked as he is in an ostentatious mantle of science, ought to have thorough familiarity with the concept.

Ostermeier looked into the method PolitiFact uses to choose which claims it would check, and found PolitiFact editor Bill Adair saying "We choose to check things we are curious about. If we look at something and we think that an elected official or talk show host is wrong, then we will fact-check it."  Ostermeier questioned how the above method could explain the numbers:  a roughly even number of ratings from the Republican and Democrat groups with one group faring significantly worse in the ratings.  Ostermeier's making the point that the PolitiFact data do not constitute anything like a fair scientific experiment designed to figure out which party does a better job as a steward of the truth.

Mooney, if he has the type of familiarity with science that he tries to project in his writing, surely knows that the numbers from PolitiFact do not lend themselves to scientific conclusions.  Yet somehow he manages to produce that impression.

Mooney:

After all, there is another possibility: the left just might be right more often (or the right, wrong more often), and the fact-checkers simply too competent not to reflect this—at least over long periods.

Ostermeier allowed for that possibility despite the "knee-jerk" characterization from Mooney.

Mooney:

So which interpretation is correct? While certainly not definitive, a study undertaken for my book The Republican Brain and updated for this article—with dedicated data-gathering and statistical analysis from an assistant, Aviva Meyer—lends additional credence to the latter possibility.

Mooney's study doesn't lend any "additional credence to the latter possibility."  His study of the Washington Post "Fact Checker" column in essence duplicates the Ostermeier study and leaves exactly the same questions about selection bias.  Even if one assumes that the fact checks are free from error and bias, the breakdown of the findings is entirely consistent with the hypothesis that liberal media bias accounts for the result.

Mooney's entire argument in support of liberal truthiness appears to stem from an appeal to the principle of parsimony (Occam's razor):

As Kessler’s words suggest, interpreting these data—or, for that matter, the aforementioned data on PolitiFact—as evidence of a liberal bias among the fact-checkers discussed here would be shortsighted and simplistic. All indications are that both outlets try (or even bend over backward) to be as balanced as possible.

A potentially simpler explanation for these results, then, is that the fact-checkers are simply doing their job—and Republicans today just happen to be more egregiously wrong. Democrats, meanwhile, are certainly not innocent when it comes to making misleading statements, but their pants are not on fire.

Apparently since Kessler professes unconcern about which party he fact checks Mooney believes this indicates something akin to a lack of selection bias.  In truth, it indicates nothing of the kind and we are left to wonder at how Kessler's words are supposed to connect to the conclusion the liberal bias hypothesis is "shortsighted and simplistic."

Mooney says the fact checkers try to be as balanced as possible.  Because they say so?  That's silly.  There is nothing in either Kessler's or PolitiFact's selection process, so far as we know, that would keep the data from manifesting a selection bias problem even if we assume that each individual fact is checked in a neutral manner.

In science, one does not accept a researcher's word that his data is free of selection bias.  One expects a check on selection bias built into the method of selection, often accomplished via a simulation of randomness.

Occam's razor?

Mooney suggests that it is "potentially simpler"  to explain the data by assuming that the fact checkers are neutral while Republicans fib more than Democrats.  Mooney apparently thinks it simpler to suppose that a large group of Republicans lies more compared to Democrats than to suppose bias in a small group of fact checkers.  On the surface, that is the opposite of the way the principle of parsimony is supposed to work.

I challenge Mooney to describe his understanding of Occam's razor as it applies to this issue.

Until he deals with that problem as well as others, he's properly classed as a truth hustler.