PolitiFact 2011: A review

PolitiFact Bias has now spent approximately a full year highlighting criticisms of the PolitiFact fact checking brand.

Our hopes that PolitiFact would improve its performance in light of outside criticism have gone largely unfulfilled.  Perhaps the biggest improvement was the reconciliation of two differing definitions of the "Half True" rating, but that modest accomplishment occurred without any announcement or acknowledgment at all from PolitiFact.  By contrast, PolitiFact wrote extensively about its momentous change in calling its fourth rating from the top "Mostly False" rather than "Mostly True" even though the definition remained the same.

Here's a rundown of the issues that should keep discerning readers from trusting PolitiFact:

1)  PolitiFact persistently ignores the effects of selection bias.  It simply isn't plausible that editors who are very probably predominantly liberal will choose stories of interest on a neutral basis without some systematic check on ideological bias.  PolitiFact, for example, continues to publish candidate report cards as though selection bias has no effect on the report card data.

2)  PolitiFact continues to publish obviously non-objective stories without employing the journalistic custom of using labels like "commentary," "opinion" or even "news analysis."  Readers are implicitly led to believe that stories like an editorial "Lie of the Year" selection are objective news stories.

3)  PolitiFact continues to routinely apply its principles of analysis unevenly, as with its interpretation of job creation claims (are the claims assumed to refer to gross job creation or net job creation?).

4)  PolitiFact has yet to shake its penchant for snark.  Snark has no place in objective reporting (see #2 above).  Unfortunately, PolitiFact treats it like a selling point instead of a weakness, and PolitiFact's intentional use of it has apparently influenced Annenberg Fact Check to follow suit.

There is a silver lining.  PolitiFact's methods produce perhaps the best opportunity yet to objectively measure mainstream media bias.  Some of those projects will be published at PolitiFact Bias over the coming year, with the study specifics available through Google Docs.