A frequent complaint -- from Democrats and Republicans alike -- is that we’re biased toward the other side. Sometimes, readers ask for a breakdown of our ratings by political party and are surprised to learn that we did not keep count. That’s because we’re focusing on the action in front of us, trying to make our most objective call. A good umpire shouldn’t keep track of the number of close plays at home plate he or she has called in favor of the home team or visitors.I'll suggest there's a problem here.
What if the record shows the umpire consistently favoring the Yankees? Does the umpire wish to remain ignorant of what the record shows about his judgments?
The comparison shows PolitiFact Virginia rating more Republican statements than Democrat statements, 79 to 58. PolitiFact Virginia says that's fair considering the Republicans control the governor's office and the legislature, along with a spirited Republican primary for a Senate seat running against the Democratic former governor of Virginia, Tim Kaine.
There's a ready-made study topic.
Democrats scored slightly higher on a numerical basis (2.78 to 2.66), but the striking thing is PolitiFact's continued failure to note that such numbers mean very little without some control for selection bias. Trusting editorial selection to pass as a random determinant simply doesn't cut it.
The numbers do contain some potential evidence showing a degree of even handedness at PolitiFact Virginia: The percentage of "Pants on Fire" ratings is nearly the same for both parties, with the Republicans actually scoring slightly lower by that measure. I suggest the number of "Pants on Fire" ratings is significant because PolitiFact offers no objective measure for using "Pants on Fire" as opposed to simply "False."
That's not to say that the, uh, Subjectivometer cannot be fooled, of course. It always matters what particular rulings the editors choose. In primaries, for example, one can easily rule favorably on a claim by a Republican that makes it look bad for other Republicans. So conservatives suffer the harm while looking pretty on the "Truth-O-Meter" scale.
Bottom line, the numbers PolitiFact Virginia offers us don't mean much. We're left to wonder whether the birthday story appears this way if Republicans score a perfect 5.0 on the Truth-O-Meter while Democrats score below a 2, or any result that seems questionable in terms of fairness. It's hard to see why PolitiFact Virginia publishes the story unless it is designed to make the operation look fair. The numbers don't say that without a control for selection bias.
A tally of the aggregate ratings has another aspect that makes them useless, regardless of the selection bias. The fact checks themselves aren't accurate.
ReplyDeleteThe Ingraham/RomneyCare rating comes to mind. They rated Ingraham False for her statement that RomneyCare was unpopular in Massachusetts. That would have been an accurate rating to give her if she had actually said that. Instead, they flubbed the quote and judged her against a fictional statement (as Sublime Bloviation readers already know).
As we've seen, PF makes so many errors in the fact checks (that even if they could adjust for selection bias) any "Truth Index" they create is still dubious.
Good point, Jeff, though I'm inclined to give PolitiFact the benefit of the doubt that its editors believe its ratings are accurate. It's a concern that PolitiFact ignores the obvious while promoting its report cards, "Truth Index" and the like.
ReplyDelete