Bill Adair wrote a story about Glenn Beck's history on PolitFact's cheesy "Truth-O-Meter."
Perhaps that has Karen Street beaming in partial satisfaction.
Most of Adair's story is relatively inoffensive, since much of it merely offers brief mentions of past stories on Beck. The problem comes from portions like this:
In the meantime, we thought it would be timely to look at Beck's record on the Truth-O-Meter. As you can see from the running tally in his PolitiFact file, we've rated 17 statements by the Fox News talk show host. It's fair to say that record skews toward the False end of the Truth-O-Meter.Folks like Karen Street are thinking "So, what's the problem? Glenn Beck tends to fudge the truth."
His record (as of Aug. 27, 2010):
Mostly True 1
Half True 3
Barely True 4
Pants on Fire 3
The problem is the type of generalization that stems from Adair's presentation. The "Truth-O-Meter" ratings do not come from a random sample of Beck's remarks. They are selected by the staffers at PolitiFact based, it is said, on their own editorial judgment with input via the suggestions of readers. The readers, if the comments appearing at FaceBook offer any indication, trend left.
When sampling is the result of specific choices (such as editorial judgment), the resulting sample contains selection bias. Because of selection bias, the "Truth-O-Meter" stats like the ones PolitiFact published for Beck mean virtually nothing with respect to Beck's veracity. Assuming the accuracy of PolitiFact ratings--which, unfortunately, we can't--the only reliable information we get from the story is via anecdote. In other words, if the ratings are accurate then we get a set of examples of things said by Beck reflecting various shades of truth.
Individually, the examples say something significant about Beck in each case. In the aggregate, however, they say little to nothing about Beck. They tell you more about PolitiFact than about Beck because the examples pattern PolitiFact's selection bias.
That, in a nutshell, is why PolitiFact ought to be wary of suggestions to aggregate its scoring for individuals and parties. PolitiFact likely receives any resulting black eye.
While browsing some comments at PolitiFact's FaceBook page, it occurred to me that I could easily add some support to my key claim above, that people easily accept the idea that aggregated PolitiFact ratings for an individual roughly translate to a measure of that individual's truthfulness. The comments all occur in response to Adair's story on Beck.