The Good, Bad and the Ugly in New Hampshire

PolitiFact returns to New Hampshire with a local affiliate, led by the Nashua Telegraph.

The Telegraph published an editorial about the renewed fact-checking venture.

A few points deserve special emphasis.

The Good

Now, it’s true fact-checking has its imperfections. One major criticism is that fact-checkers get to decide which statements to check, which is a subjective exercise.

Granted.

 I have yet to see PolitiFact editor Bill Adair make a comparably clear statement admitting to the non-objective nature of story selection.  Compare the above with Adair's response when the latter was asked how PolitiFact avoids the selection bias problem:

There are many things that go into deciding what we are going to choose. We try to be timely, we try to stay on top of the news and we try to have balance so we check people from both parties. That can be challenging though, because if you have eight voices speaking up in a Republican primary and only one Democratic incumbent – naturally you have eight times the number of statements being made on the Republican side than on the Democratic side. We try to check roughly the same number of claims by Democrats as we do for Republicans, but we have to go where the claims are and lately there have been more made by Republicans. In terms of avoiding selection bias, I think the key is to be guided by what serves the reader. Once you get past claims selections, our fact-check process is entirely driven by journalistic and independent assessment.

Note to Adair:  The Telegraph editors have the right of it.  Next time you're asked how you avoid selection bias, just admit that you don't avoid it.  If you want to bolster that answer with the assurance that PolitiFact tries to treat statements fairly after that, then go for it.  But don't go around thinking that the attempt mitigates selection bias.  It doesn't.  Your organization is also very poor at applying its standards fairly, though that's a separate issue.

The Bad

Yet every other article we choose to publish in the newspaper or online also holds a degree of subjectivity. Anticipating the interests of readers is at the foundation of good journalism.

This bizarre rationalization amounts to a self-referential application of the tu quoque ("You also") fallacy.  The tu quoque fallacy rationalizes one action based on the fact another (typically the accuser) performs the same action.  In this case the message amounts to "Yes, it's subjective to choose which stories we fact check, but we do that all the time in "objective" newspapers."  It's a "Me, too!" version of the "You, too!" fallacy.

Readers are willing to forgive non-objective story selection because there really isn't much of an alternative.   The response from the Telegraph misses the point.  Nobody criticizes Factcheck.org on the issue of selection bias.  Why is that?  It's because Factcheck.org does not use a silly "Truth-O-Meter" that misleads readers into thinking that they have a useful guide as to the truthfulness of individual politicians based on a selected sampling of fact check stories.  As researcher Eric Ostermeier of the University of Minnesota and the Humphrey School of Public Affairs put it (bold emphasis added):

The question is not whether PolitiFact will ultimately convert skeptics on the right that they do not have ulterior motives in the selection of what statements are rated, but whether the organization can give a convincing argument that either a) Republicans in fact do lie much more than Democrats, or b) if they do not, that it is immaterial that PolitiFact covers political discourse with a frame that suggests this is the case.

The Telegraph needs no excuse for selection bias if it does not join PolitiFact in using "Truth-O-Meter" stats to produce the impression that such statistic represent reliable information about a pattern of truth-telling by the individuals rated.  The excuse does not mitigate the sins of the Truth-O-Meter.

The Ugly

Another criticism is that journalists themselves – Republicans, Democrats or something else along the political spectrum – can’t resist tilting the scales toward the ideology they agree with most.

The same criticism is made in professional sports. Referees are accused of making calls that are overly generous for one team and overly critical of the other. There’s no doubt it happens.

Adair has used this same referee analogy recently:

"When you're a loyal fan of a team, you're going to think the referee is biased against your team."

The Telegraph deserves some credit for taking the analogy to a point Adair tends to avoid, specifically the reality that referees make mistakes and allow bias to affect their rulings.  That's one example showing why the referee analogy serves PolitiFact poorly by coming too close to the truth.

Other examples show the ugliness of the analogy.

Why do sports leagues employ referees?

Because somebody has to make a final determination on matters where the application of the rules has an effect on the outcome of a game.  The sports league chooses the referees, and the league has an interest in choosing quality referees in order to protect the image of its product.

In contrast, journalistic fact checkers appoint themselves to look over a "game" (American politics) that lacks an owner.  Without an owner with an interest in maintaining the integrity of the game, what is supposed to motivate fact checkers to provide neutral judgment? 

The Telegraph provides an answer of sorts.  There are many referees, so somebody will make the right call.

The Telegraph offers no guidance as to which referee we should trust.  Lacking that, it's hard to see how the answer helps us.

The ugliness of the analogy reveals itself as we realize that the game of politics needs no referee other than the voter.  Fact checking is not a referee function.  Ideally, it is an information resource the real referees--the voters--can use to help fulfill their role in the political process.

When fact checkers raise their status to that of refereeing the political process they become nothing other than another elite voice trying to sway voters.

That is a partisan place.