Selection bias? What selection bias? (Updated)

PolitiFact published a new type of story on Aug. 27. At least it was new to me.

Bill Adair wrote a story about Glenn Beck's history on PolitFact's cheesy "Truth-O-Meter."

Perhaps that has Karen Street beaming in partial satisfaction.

Most of Adair's story is relatively inoffensive, since much of it merely offers brief mentions of past stories on Beck. The problem comes from portions like this:

In the meantime, we thought it would be timely to look at Beck's record on the Truth-O-Meter. As you can see from the running tally in his PolitiFact file, we've rated 17 statements by the Fox News talk show host. It's fair to say that record skews toward the False end of the Truth-O-Meter.

His record (as of Aug. 27, 2010):

True                1
Mostly True     1
Half True         3
Barely True     4
False               5
Pants on Fire   3

Folks like Karen Street are thinking "So, what's the problem?  Glenn Beck tends to fudge the truth."

The problem is the type of generalization that stems from Adair's presentation.  The "Truth-O-Meter" ratings do not come from a random sample of Beck's remarks.  They are selected by the staffers at PolitiFact based, it is said, on their own editorial judgment with input via the suggestions of readers.  The readers, if the comments appearing at FaceBook offer any indication, trend left.

When sampling is the result of specific choices (such as editorial judgment), the resulting sample contains selection bias.  Because of selection bias, the "Truth-O-Meter" stats like the ones PolitiFact published for Beck mean virtually nothing with respect to Beck's veracity.  Assuming the accuracy of PolitiFact ratings--which, unfortunately, we can't--the only reliable information we get from the story is via anecdote.  In other words, if the ratings are accurate then we get a set of examples of things said by Beck reflecting various shades of truth.

Individually, the examples say something significant about Beck in each case.  In the aggregate, however, they say little to nothing about Beck.  They tell you more about PolitiFact than about Beck because the examples pattern PolitiFact's selection bias.

That, in a nutshell, is why PolitiFact ought to be wary of suggestions to aggregate its scoring for individuals and parties.  PolitiFact likely receives any resulting black eye.


Update:

While browsing some comments at PolitiFact's FaceBook page, it occurred to me that I could easily add some support to my key claim above, that people easily accept the idea that aggregated PolitiFact ratings for an individual roughly translate to a measure of that individual's truthfulness.  The comments all occur in response to Adair's story on Beck.

Jean Sutherland
No surprise here. The man couldn't tell the truth if he tried.
Monday at 12:34pm

Valerie Neufeld Gebroe
Just one question...Why does he have so many followers if he is so dishonest? It's scary.
Monday at 12:44pm

John Samuel
Were one to seriously and dispassionately to compare the veracity of Mr Obama's statements and those of Mr Beck then the victory is clearly Obama's.  
Monday at 12:49pm 


Valerie Neufeld Gebroe
‎(...) On PF if you look at the honesty scale for POTUS you will see that his % of false statements is far below that of Beck. 
Monday at 12:53pm 


Monica N Thadius Whittle
Not surprised. This man is full of lies! Thanks for the info, anyway.  
Monday at 12:54pm 


Bill Fant
Gee, what a surprise... Beck lies more than he tells the truth and he holds the record on PF for the most False ratings.

How people fall for this charlatan is beyond me. 
Monday at 2:56p


Shari Stowell
I guess the truth-o-meter must be a liberal shill ... 
Tuesday at 12:07am