Yet more bogus evidence from truth-hustler Chris Mooney

We should expect Chris Mooney to know something about science and epistemology.

Get used to disappointment.

I've classed Mooney as a "truth hustler."  Mooney, posing as a voice of reason, misleads people.  An article published today at "Desmogblog" serves as the latest example.

In the article, titled "More Evidence That Republicans Are More Factually Challenged Than Democrats," Mooney introduces us to the latest evidence:  Republicans fare worse than Democrats according to the past year's statistics from The Washington Post Fact Checker (Glenn Kessler).

What's wrong with that evidence?  Plenty.  And we'll let Mooney help explain it via contrast with a series of wrongheaded statements.

Mooney:

I ... needed evidence from the real world showing that, you know, conservatives or Republicans are more factually incorrect.

That’s where all the fact-checker data came in.

You see, we have paid professionals whose job it is to track just how wrong Democrats and Republicans are.

Mooney, intentionally or not, misses an important distinction.  Journalists are not paid to determine whether Democrats are differentially more wrong than Republicans or vice versa.  Fact-checking journalists are paid to produce stories that interest readers.  Toward that end, fact-checking journalists use editorial judgment to find potentially popular stories.

In scientific terms, fact checking journalists purposely employ a form of selection bias in choosing the data set they choose to fact check.

If Mooney's bio isn't filled with lies then he has to know about selection bias and its implications for would-be scientific conclusions.  A study that fails to control for selection bias provides poor evidence for any conclusion it attempts to support.

Mooney:

I find these fact-checker data particularly compelling, by the way, for the following reason: Neither PolitiFact nor Glenn Kessler (who writes the Post’s column), think of themselves as liberal partisans.

Lovely.  We might as well favor the findings of scientists who do not perceive their own bias.

If Kessler and the PolitiFact folks thought of themselves as partisan, maybe it would increase the chances they would institute controls on selection bias.  Mooney's self-confessed reassurance is misplaced.  In truth, the self-perception of bias is irrelevant.  A properly done study controls for bias so that the point of view of the researcher is irrelevant.  But Kessler and PolitiFact apparently do absolutely nothing to control for selection bias.

I would argue that both go too far in trying to ding Democrats and liberals, just to make themselves appear balanced (and, presumably, to keep getting their calls returned by the other side of the aisle). Therefore, if their data shows Republicans fare worse, that really says something.

As evidence, Mooney first presented his prediction that Kessler would react to the last half year's findings with a resulting change in the numbers indicating Kessler's attempt to correct for the potential perception of bias.

Mooney was wrong.  The new data looked pretty much like the old data.

Mooney, undaunted by the failure of his prediction, found even more reason to cleave to his theory.  After all, if the numbers look the same this year six months as last, doesn't this suggest that Kessler actually treated Democrats unfairly in order to obtain that result?

Mooney:

(T)he data are all the more striking in that I believe Kessler inflates the Democrat total, intentionally or otherwise.

At least Mooney tries to support his statement with evidence.  Unfortunately the evidence remains pathetically weak.

1)  Kessler gave President Obama four "Pinocchios" for misattributing a quotation to Rutherford B. Hayes.

2)  Kessler didn't even bother to rate Sarah Palin's statement that Paul Revere "warned the British."

Two anecdotes making up one comparison?  Seriously, that's all we get?

Mooney does make a late-breaking feint toward reason:

(G)iven that this difference does not appear to be a fluke, there has to be a reason for it to exist. And I agree, one could posit many explanations: Maybe there is a selection bias with respect to which items are being rated. Maybe Kessler himself is biased against Republicans.

Maybe there is a selection bias?

In science we expect researchers to take affirmative steps to eliminate selection bias.  Lacking evidence of such steps we assume that the evidence is tainted by selection bias, lest we draw unfounded conclusions.  Or if you're Chris Mooney just go ahead and draw the unfounded conclusion while allowing that maybe there's selection bias.

Mooney:

But the point is, there has to be an explanation. It just won’t do to say, as Kessler does, that since both Dems and Republicans score above Two Pinocchios, both ought to be ashamed of themselves. The data are telling us something more than that—something stronger than that.

The answer to the question isn't that tough.  The data are not designed to give us a dependable differential breakdown for the claims of Democrats against those of Republicans.  No serious researcher would try to answer that question while relying on left-leaning journalists with no control for selection bias.  But the data do have an obvious use.  The data directly measure how fact checkers choose subject matter.  Kessler and PolitiFact give us an excellent tool measuring their selection bias.  Add to that the ratings each fact checker provides and we end up with data that can help us measure media bias.

As for Chris Mooney, I can only wonder what cognitive processes he uses to either miss or de-emphasize the obvious.


Correction 7/10/2012:  The totals for Glenn Kessler's Pinocchio count apparent came from a six-month period rather than from a full year. The text was altered to reflect the shorter period of time.

(for Glenn Kessler and PolitiFact) How to fact check the job recovery numbers

A valuable media watchdog watchdog post at the new blog "counterirritant" pointed out a problem with Glenn Kessler's fact check of a claim from Republican presidential candidate Mitt Romney.

Kessler writes the fact checker feature for the Washington Post.  Romney claimed that in a normal recovery from recession the country should be adding something like 500,000 jobs per month.

counterirritant:

Kessler decided that the best way to “check” this was determine how frequently 500,000 jobs were created in a month in the last 65 year.

The post goes on to very effectively criticize Kessler's methodology throughout.

By a funny coincidence (general leftward lean of the mainstream media, maybe?), PolitiFact used very similar reasoning on the same claim:

Is 500,000 jobs created per month normal for a recovery?

The short answer is "no."

We arrived at this conclusion by looking at the net monthly change in jobs all the way back to 1970. Since Romney was referring to total jobs, rather than private-sector jobs only, we used total jobs as our measurement. And since Romney was talking about job creation patterns during a recovery, we looked only at job creation figures for non-recessionary periods, as defined by the National Bureau of Economic Research. Finally, we excluded the current recovery.

The Kessler/PolitiFact method is entirely wrongheaded.

Romney didn't claim that 500,000 jobs created per month was a normal figure during a recovery.  I can imagine the furrowed brows of Kessler, Louis Jacobson and other mainstream media fact checkers.  Aren't they the experts?  What am I talking about?

It's actually pretty simple.

The size of the economy changes.  If country A enters a recession losing 1 million jobs  then it takes two months to regain the lost jobs at a rate of 500,000 per month.  If country B experiences a recession losing 10 million jobs then it takes 20 months to regain them at a rate of 500,000 per month.

Not only does the size of the economy vary, but so does the depth of the recession.  The rate of recovery for lost jobs needs  to account for both factors.  Neither Kessler nor PolitiFact gave any apparent consideration to those critical criteria.  It's like comparing prices between now and the 1950s without adjusting for inflation.

The Romney campaign has made a number of statements like the 500,000 jobs claim, and they probably relate to the following chart or one like it from the Bureau of Labor Statistics:

Clipped from nytimes.com

Romney's claim almost certainly derives from the fact that post-war recoveries usually replace lost jobs much faster than the present recovery. 

So how does one check that claim?  It's not that hard.  Take the bottom point of employment, then count the number of months that it takes to get employment back up to the peak level.  Divide the number of jobs lost by the number of months it took to return to return to the employment peak at the start of the recession.  Do the same for each of the post-war recessions, then average the numbers to obtain the average job recovery time.  After all of that, divide the number of jobs lost from the 2007 recession by the averaged job recovery time.

Why didn't the Washington Post or PolitiFact do anything remotely resembling the fact check I just described?  It could be gross incompetence.  It could be ideological bias.  Or it could be both.





Correction 6/7/2012  Removed a redundant division operation from the next-to-last paragraph

WaPo's Fact Checker weighs in: Did harsh interrogation help nab bin Laden?

The press treatment of waterboarding has interested me for some time, so naturally the fact check to determine whether waterboarding helped pin down bin Laden's location piqued my curiosity.

The Washington Post's Glenn Kessler picked up with a minor war of words between former Bush administration attorney general Michael Mucasey and Sen. John McCain.  I'll start with Kessler's conclusion and work back from there:

We do not have enough information to make a definitive judgment. But it appears that Mukasey is straining to make a connection between the killing of bin Laden and the harsh interrogation techniques that appears, at best, tangential. Otherwise, he would not have had to resort to verbal sleight of hand to make his case. McCain, by contrast, appears to clearly connect the dots from the courier to bin Laden, citing information derived from conventional techniques.

At the same time, while the enhanced techniques may not have provided the Rosetta stone to bin Laden’s whereabouts, Mukasey may be right when he asserts that valuable leads in the broader war against al-Qaeda were derived through these techniques.

We probably will never know whether the same information -– or more accurate information -- could have been obtained through conventional interrogation. The use of these techniques also harmed the U.S. image overseas — another question U.S, policymakers will have to balance in the future.

Kessler did a better job than I thought at first.  The first paragraph of his concluding section made it appear that he had largely bought McCain's view and discounted Mukasey's.

Marc Thiessen's book "Courting Disaster" makes clear the role of enhanced interrogation.  It is not used to elicit information from a detainee but rather used to make detainees cooperate with "conventional techniques."  Thus every piece of information will occur as a result of using "conventional techniques."  But not every piece of information gathered using conventional techniques would necessarily obtain all the information accrued after using enhanced techniques.  In particular, the detainee might delay in revealing information minus the use of enhanced interrogation.  But it's hard to say if such delays might have obscured the trail to bin Laden.

Kessler might have been clearer about the role of enhanced interrogation, but his conclusion is substantially correct.

Journalists reporting badly: The Clinton "Reset" button (Udated)

The news is out for some time that the State Department and Secretary of State Hillary Clinton produced a bit of a flub during a meeting with the Russians.

The incident was amusing enough, but the mainstream media have compounded the amusement.

The St. Petersburg Times published two associated stories, one regarding the reactions to the meeting and the other about the gaffe. The first was apparently based on AP material. The second drew content from Glenn Kessler's story for The Washington Post. The story seems to underemphasize the degree of embarrassment the State Department suffered after having "worked hard" on the translation.

And while going a bit soft on the gaffe, reporters including the estimable Robert Burns described the incident incorrectly. The video clearly shows that Clinton opened the box and handed the button mock-up to Russian Foreign Minister Sergey Lavrov, yet Burns' story reads thus:

Clinton handed Lavrov a green box tied with a green bow. He opened it to reveal a "reset button," a reminder of Vice President Joe Biden's recent remark that the Obama administration hopes to reset U.S. relations with Moscow.

Kessler:

Clinton began the meal by presenting Lavrov with a palm-size box wrapped in a green ribbon. Lavrov opened it up and pulled out a small plastic box with a red button that clicked — a symbol of the Obama administration's determination to "reset" the relationship.

Roll the video:



Note to the press: Get basic facts right, including incidental ones like this, and you have a better chance of getting people to trust your reporting.


Update:

Note to readers: Content providers have been more aggressive lately in getting material removed from YouTube, including the video I linked. However, the remaining still image shows Clinton with the box already opened in one hand the and the button held in the other. Lavrov did not remove the button from the box as reported in the press.