Tuesday, August 31, 2010

Selection bias? What selection bias? (Updated)

PolitiFact published a new type of story on Aug. 27. At least it was new to me.

Bill Adair wrote a story about Glenn Beck's history on PolitFact's cheesy "Truth-O-Meter."

Perhaps that has Karen Street beaming in partial satisfaction.

Most of Adair's story is relatively inoffensive, since much of it merely offers brief mentions of past stories on Beck. The problem comes from portions like this:
In the meantime, we thought it would be timely to look at Beck's record on the Truth-O-Meter. As you can see from the running tally in his PolitiFact file, we've rated 17 statements by the Fox News talk show host. It's fair to say that record skews toward the False end of the Truth-O-Meter.

His record (as of Aug. 27, 2010):

True                1
Mostly True     1
Half True         3
Barely True     4
False               5
Pants on Fire   3
Folks like Karen Street are thinking "So, what's the problem?  Glenn Beck tends to fudge the truth."

The problem is the type of generalization that stems from Adair's presentation.  The "Truth-O-Meter" ratings do not come from a random sample of Beck's remarks.  They are selected by the staffers at PolitiFact based, it is said, on their own editorial judgment with input via the suggestions of readers.  The readers, if the comments appearing at FaceBook offer any indication, trend left.

When sampling is the result of specific choices (such as editorial judgment), the resulting sample contains selection bias.  Because of selection bias, the "Truth-O-Meter" stats like the ones PolitiFact published for Beck mean virtually nothing with respect to Beck's veracity.  Assuming the accuracy of PolitiFact ratings--which, unfortunately, we can't--the only reliable information we get from the story is via anecdote.  In other words, if the ratings are accurate then we get a set of examples of things said by Beck reflecting various shades of truth.

Individually, the examples say something significant about Beck in each case.  In the aggregate, however, they say little to nothing about Beck.  They tell you more about PolitiFact than about Beck because the examples pattern PolitiFact's selection bias.

That, in a nutshell, is why PolitiFact ought to be wary of suggestions to aggregate its scoring for individuals and parties.  PolitiFact likely receives any resulting black eye.


Update:

While browsing some comments at PolitiFact's FaceBook page, it occurred to me that I could easily add some support to my key claim above, that people easily accept the idea that aggregated PolitiFact ratings for an individual roughly translate to a measure of that individual's truthfulness.  The comments all occur in response to Adair's story on Beck.

MRAP training at Camp Shelby (video)



The video provides a brief explanation of the MRAP program and looks at the training soldiers now receive before using MRAPs in the field.

Find an associated story here.

Monday, August 30, 2010

CNN Political Ticker: Crist looking to "eek out a victory."

Enough said.  I think "eek" is a good way to put it.

John Bartosek responds

Yesterday I sent an e-mail query to PolitiFact, intended for the e-mail boxes of editor John Bartosek and writer-researcher Lukas Pleva.

I asked for the reason why PolitiFact, given its non-partisan nature, would focus on Rep. Alan Grayson's claim that the U.S. ranks No. 50 in life expectancy worldwide rather than the adjacent claims that the U.S. ranks "just below" Albania and ranks "dead last" in math test scores (background and text of my message).

Bartosek's prompt reply arrived in my e-mail box at 9:39 a.m. EDT.

Fascinating, to borrow a line from Mr. Spock:
Thank you for your recent note about PolitiFact. I'm happy to give you some background on how we work. We focus on a single statement, as often as we can, so that we're making a ruling on only one comment at a time. We think that helps keep us focused and makes things clear. We have, in the past, tackled various education issues, including the state's high school graduation rate being below the national average (True), the state being 50th in education spending (Half True), and Florida teachers being paid $5,000 below national average (Mostly True). It wasn't bias in not picking the comment about math ranking, it was simply the chance to look at another topic (health care) that we hadn't touched on lately.
Bartosek wasn't done, but I need to cut in.  Deciding to make the story about one part of Grayson's comment instead of about another part of the same comment is selection bias by definition.  Whether or not it represents a political bias in addition remains an open question, though we now have Bartosek on record denying the latter type of bias occurred.

Bartosek's assertion that PolitiFact deals with one claim at a time was ridiculous on its face.  I've run across numerous counterexamples, and I sent a couple of them to him in my e-mail reply.

Note that through this point Bartosek has sought to excuse ignoring Grayson's comment about that math ranking.  What about "just below Albania"?  That claim is part of Grayson's statement about U.S. life expectancy and thus, according to Bartosek, on the subject of "health care."  Bartosek tackles that next:
I think Rep. Grayson's comment about Albania was an example of his bigger point about the U.S. ranking worldwide, and we chose to focus on that broader point, rather than writing just about Albania. (Just FYI, the U.S. came in just above Albania in both the 2009 and 2010 CIA Factbook rankings, meaning our life expectancy is slightly higher than that country. The difference is very small.)
This paragraph from Bartosek is funnier every time I read it.

Undoubtedly, Grayson's comment about Albania was an example of his bigger point about the U.S. ranking worldwide.  But it was wrong!  And that fact got swept under the rug!  Bartosek then says that the decision was made to focus on the broader point rather than write "just about Albania."  But what prevents focusing on the broader point while also setting the record straight about Albania falling consistently below the United States in the rankings? 

As for Bartosek's "FYI," the CIA Factbook provides the best case for Albania ranking close to the U.S. in life expectancy.  The other two sources cited in the story told a different story, and Bartosek's claim about the 2009 CIA Factbook remains open to question.  The writer, Lukas Pleva used Wikipedia as a secondary source rather than going to the CIA Factbook itself.  This alternative source tells a different story and which is supported by the published version of the CIA Factbook remains an open question.

"The difference is very small."  Right, but the difference is very small in both directions.  Take the difference in the rankings published by the Population Reference Bureau (a source cited in the PolitiFact story) between Albania (No. 59, avg. 75 yrs) and the U.S. (No. 45, avg. 78 years) and add it to the U.S. total.  Average life expectancy of 81 years would put the U.S. up in the neighborhood of Canada (No. 15, avg. 81 years).

Even for the Wikipedia version of the 2009 CIA Factbook figures most favorable to Grayson, the U.S. could hop from No. 50 to No. 47 based merely on the difference (.15 years) with Albania:


Though it appears possible on its face that the United States could make up the difference with No. 1 simply by a change in the national diet, PolitiFact justifies the focus of the story because it supposedly concerns health care.

Sure, not much separates the U.S. from Albania in years of life expectancy.  But not much more than what separates the U.S. from Albania separates the U.S. from a top 10 ranking, all the cited sources considered.

Bartosek's response does little to dispel the appearance of political bias.  On the contrary, it offers evidence of a considered selection bias with possible roots in political bias.  In addition, the claim that PolitiFact sticks with one subject "as often as we can" is hard to swallow given the contradictory evidence.  "(A)s often as we can" apparently boils down to subjective news judgment and provides an avenue for the influence of political bias.

As I've said often in the past, I do not believe that PolitiFact staffers intentionally inject their political bias into their work.  They simply take inadequate precautions against that occurrence.


Afters

I decided early not to use the full text of my e-mail reply to Bartosek in this post.  That decision had nothing to do with my leaving out a "the" near the end of my message:
The average reader has no conception that a fact checking outfit will ignore the truth values for dubious adjacent claims merely to help ensure that [the] topic of health care (?) gets its due.
That's not to rule out the existence of other typos I may have missed on review.

Sunday, August 29, 2010

Transparently opaque

Content has been reexamined, new sources have been sought, and more reporting assigned. The result has been outstanding journalism, which you will see in the coming days, weeks and months. The work will be as transparent as possible. The writing will be lively.
So wrote Providence Journal editor Thomas E. Heslin regarding his paper's experience upon entering its partnership with PolitiFact, the fact checking enterprise starting at the St. Petersburg Times.

How transparent is "as transparent as possible"?

Well, PolitiFact provides a list of sources used for the story.  Many of them permit the reader to double check and verify with their own eyes.  Others do not, such as e-mail correspondence and telephone interviews.  In the latter cases the reader must rely on the fairness of the reporter.  As for the reporters, they probably have their own biases--but most news organizations strongly discourage reporters from making their political leanings known to the public.  That type of transparency, it is thought, erodes confidence in the objectivity of news reporting.

Another aspect of reporting remains almost always hidden from the public:  the choice of which story to cover and what portion of the story deserves focus.

I recently performed a critical review of a fact check story on Rep. Alan Grayson (D-Fla.).  Grayson, during a television appearance, made three questionable claims in the space of a few sentences.  Grayson claimed that the U.S. ranks No. 50 worldwide in terms of life expectancy, the U.S. ranks just below Albania in life expectancy and  the U.S. ranks "dead last" in math test scores.

All three of Grayson's claims were on thin ice, though PolitiFact justified rating Grayson "Mostly True" by ignoring the fact that the ranking Grayson claimed came from a list of well over 200 territories and sovereign nations.  Those aware of the approximate number of sovereign nations (about 195) were likely to take Grayson's claim to mean that the U.S. fell in the fourth quintile.  The U.S. belongs in the top group instead of the runnerup group when compared to other sovereign nations rather than competing with itself (Puerto Rico) or with other non-sovereign territories like Guernsey.

My fact check found that Grayson's latter two statements were assuredly false.  So what accounted for the editorial decision to ignore those claims in favor of the chosen one?  There's really only one way to find out, and that is to ask PolitiFact.  I sent the following e-mail to the editor and writer/researcher of the story:
Dear Mr. Bartosek, Mr. Pleva,

Are you able to shed any light at all on the editorial decision to focus on the portion of Rep. Alan Grayson's statement regarding the No. 50 rank in life expectancy rather than the portion placing the U.S. "just behind" Albania or his statement about the U.S. ranking "dead last" in math?  It's very easy to imagine political bias behind the decision as to which part of the statement to rate on the "Truth-O-Meter" since the latter two statements appear exceptionally difficult to justify.  But PolitiFact is non-partisan.  So what was the real reason for the focus on the No. 50 ranking, please?
Do I expect an answer?  No, I do not.  It seems that in a best-case scenario the PolitiFact staff can simply claim that the two false claims did not seem like an interesting story nor relevant to the story that was written.  That answer does not admit to bias, but it lends itself to the interpretation of selection bias.  PolitiFact will not want to put forth any plain or even implicit admission of selection bias, particularly in a case like this one.

But it's only fair to give PolitiFact a chance to exhibit an unexpected degree of transparency.

Saturday, August 28, 2010

Grading PolitiFact: Alan Grayson announces the rankings

The issue:



The fact checkers:

Lukas Pleva:  writer, researcher
John Bartosek:  editor


Analysis

Rep. Alan Grayson (D-Fla) appeared on The Ed Show on MSNBC.  Grayson drew PolitiFact's attention with this:
UYGUR: I swear to you that my next question was if they accused him of being a Martian, do you have to come out every once in a while and say the president is not Martian?

(LAUGHTER)

GRAYSON: It‘s sad. Look, we really are cheating ourselves.  This can be a heaven on Earth. This could be an outstanding place to live, the very top in the entire world.

Instead, we‘re 50th in the world -- 50th in life expectancy, just below Albania. We are dead last in math test scores. And believe me, the math is the same in Seoul, Korea, as it is in here. It‘s the same math.

We have to work harder to solve our problems. And we can‘t get distracted all the time by the people who want to distract us for their own selfish purposes.

(yellow highlights added to emphasize portion quoted by PolitiFact)
Grayson made many dubious statements during his television appearance.  Just in the quoted portion he made the claim the U.S. was ranked No. 50 worldwide in life expectancy, ranked below Albania for life expectancy and "dead last" in math test scores.  PolitiFact apparently found the latter two claims uninteresting.

Picking a focus from among those three statements represents a case of framing.  Such framing may make a story potentially difficult to distinguish from political spin.

Take it away, Lukas Pleva:

Wednesday, August 25, 2010

Scott bests McCollum for GOP governor bid

Outsider Rick Scott defeated former congressman Bill McCollum in the Republican primary yesterday, securing a chance to run against Democratic winner Alex Sink for governor of Florida.

I voted for McCollum, thinking he had the better shot at beating Sink in the general election.  Scott's conservative message, combined with a personal charisma somewhat greater than McCollum's, gave him the victory.  It didn't hurt that Scott spent a great deal of his own money in the primary, but if spending money was the secret to winning then Jeff Greene would have beaten rival Kendrick Meek for the chance to challenge Marco Rubio for one of Florida's two seats in the Senate.

***

I'm amused by an ad Google placed on my blog:


There must be some specialized definition of "proven business experience."  One that allows mid-level banking executives (I'm counting CEO of Bank of American in Florida as the top of mid level) to claim "proven business experience" while also preserving plausible denial for all the things banks do that their customers and employees resent, such as layoffs in the wake of mergers, high transaction fees and cascading overdraft charges.

Meanwhile, Rick Scott's creation of Columbia Healthcare and his administrative role in Columbia/HCA are somehow discounted.

Tuesday, August 24, 2010

Fanblog: The Return

In a recent post I noted the appearance and disappearance of Politi-Psychotics, a blog by Karen Street (see sidebar under "Testimonials") loosely dedicated to undermining my sublime bloviations.

The blog is back under the same name, but with a wisely decreased emphasis on debunking my undebunkable bunkum.  Just kidding about the "bunkum" part--I just could help typing it with "debunking" and "undebunkable" in such proximity.  The subtitle now reads "Because in politics, the road to Truth can be an obstacle course!"

Street took to heart my criticisms of the post formatting:
I do agree on Bryan's criticism of the formatting with regard to the margins.  The text editor does not publish the same as it looks in the "compose" stage.
As it turned out, Street made her blog public before she intended, which accounted perhaps entirely for the appearance of the original subtitle and at least in part for the ungainly overall appearance.  Unfortunately the arguments are just as bad as ever. 

That Street is the author of the blog is now confirmed beyond all reasonable doubt.  For example, this author stamp from before the time Street settled on "PolitiFact Commenter" as her blogging nom de plume:
This may indicate a pattern in Bryan’s bias as well!
Posted by Karen S
(bold emphasis added)
Hat tip to Google cache for catching that one.

***

Street was a tad surprised that I found her blog.  That apparently means that she has not yet explored the use of tools such as StatCounter.    No charge for the tip.

Monday, August 23, 2010

Florida 2010: A triumph of private election financing?

Remember a few weeks ago when Robyn Blumner was wringing her hands over the pernicious influence of private fortunes in Florida politics?

Today, the day before Florida's primary election, the relatively cash poor campaigns of Kendrick Meek and Bill McCollum stand poised to knock off privately-funded powerhouses Jeff Greene and Rick Scott, respectively.

How did that happen, considering that Meek and McCollum are essentially unexciting candidates for office during a year ripe for outsider challenges?

In a word (using a Bidenesque count), issues matter.

OK, so the critical issue in both cases was the fact that both Greene and Scott carry huge negatives.  Voters figured that out and gravitated toward the candidate with the best chance to win, at least in sufficient numbers to give Meek and McCollum an advantage in the polls headed into Tuesday.

Perhaps the St. Petersburg Times gives itself some credit for editorializing against the big-spending candidates.  But the impact of those editorials was modest at best and almost certainly not the deciding factor.  The people figured it out.  And that's important.

Why is it important?  Because our electoral system depends on the voters making rational decisions.  And that ought to include voting for a rich candidate who comes out of nowhere if the voters believe that candidate best represents their interests.  The day public financing is necessary to keep voters from voting non-
rationally is the day we might as well let a single party pick the one candidate voters are allowed to select.

Greene and Scott haven't lost yet.  But their names are very close to being added to the long list of failed self-financed candidates.  Elections can't be bought with money alone.  If a candidate has an attractive message then there's an opportunity to fund the campaign without initial public support, however.  That's it.

Saturday, August 21, 2010

Grading PolitiFact (Georgia): Nathan Deal and birther dabbling

Apologies to those who check regularly for "Grading PolitiFact" updates.  I have a PolitiFact-related research project in the works, and it's labor intensive.  PolitiFact Georgia snagged my attention, however.


The issue:




The fact checkers


Willoughby Mariano:  writer, researcher
Jim Denery:  editor
Jim Tharpe:  editor


Analysis:

The headline and the deck both place quotation marks around "dabbled."  Apparently the quotation marks are used to show that a Huffington Post reporter ("liberal bloggers") used the term in making the claim as he prefaced a question to White House Press Secretary Robert Gibbs.

The term ends up the key to the entire story.

 Willoughby Mariano:
Dabble generally means to take part in something casually or superficially -- to not take it seriously. We will focus on claims that he did that with birther conspiracies.

Mariano's explanation of "dabble" suffers from ambiguity. A person who dabbles in black magic, for example, may practice the art casually in the sense of spending little time in the pursuit. But we understand the person to be practicing black magic with the expectation that it and its effects are real. By analogy, "dabbling" in birther theories means spending some casual time believing they are serious theories. In that sense they are taken seriously.  Dabbling in birther conspiracies ought to mean taking the conspiracy itself seriously, but not seriously enough to spend a serious amount of time or effort on it.

What is the evidence, PolitiFact?

Friday, August 20, 2010

The first "Sublime Bloviations" fanblog ... of sorts (Updated)

A new blog appeared recently by name of "Politi-Psychotics" and with the subtitle of "Because Bloviations aren't that Sublime."

The author, presumably Karen Street (based on the blatant similarity to her comments at PolitiFact's FaceBook page), has begun a series of criticisms of my criticisms of PolitFact.  Despite the ghastly formatting of her blog posts, I'll probably bother to explain the problems with her criticisms from time to time.

Why anyone would bother to target the little-known commentary I publish is beyond me ... unless it's love ...


Update:

Closed for remodeling?


Remodeling would be a good idea.  As mentioned above, the formatting was so atrocious that reading the blog was a chore--not that I count Sublime Bloviations as a shining counterexample.  The arguments presented there at the time I visited were horrible, so the lousy formatting perhaps served a useful purpose by protecting them from scrutiny.  Though allowing readers only by invitation is arguably an even more effective method.

Friday, August 13, 2010

Weddings? What weddings?

I find one of the threads for the recent reporting on the Prop 8 decision intriguing.  News reports are circulating regarding the resumption of same-sex weddings.
SAN FRANCISCO-- The federal judge who struck down California's gay-marriage ban said Thursday that same-sex weddings can resume next week unless an appeals court intervenes before then.
This issue is state recognition of same-sex marriage, isn't it?  Any two people can have a wedding.  Indeed, I don't see why one couldn't hold a wedding ceremony between a man and his pet tarantula.

To be fair, it's easy enough to imagine a same-sex couple only considering a wedding where the state recognizes the resulting union as a marriage.  And it's conceivable that persons who traditionally officiate at weddings insist on paperwork reflecting the blessing of the state.

That said, the reporting tends to create a misleading impression that a wedding ceremony between members of the same sex can't proceed so long as Prop 8 proponents or their ilk (gasp) have their way.

***

As for the recently completed trial, the results have stimulated some great commentary.  Judge Vaughan Walker's handling of the case makes it look as though he granted standing primarily for the purpose of giving him the opportunity of presiding over a show trial.  The show trial would grant him power over the outcome of a case in which he may hold a palpable interest.  Walker has since offered that the courtroom defenders of Prop 8 probably do not have standing to appeal his decision.  How convenient!  Even Time took note:
(A)t least one constitutional-law scholar in California is suggesting that by trumpeting the issue of standing, Walker has opened a hornet's nest he may have been better off leaving undisturbed. "If the proponents don't have standing to appeal, then it's entirely plausible that the courts will rule that they did not properly have standing to go to trial," Vikram Amar, a law professor at the University of California at Davis, told TIME Thursday evening. "This is an issue he glossed over when he allowed them to intervene in the trial.


Hmm.

Ouch! Nile Gardiner counts Obama's liabilities

Please read the whole thing, but the following paragraph beautifully summarizes the whole:
There is a distinctly Titanic-like feel to the Obama presidency and it’s not hard to see why. The most left-wing president in modern American history has tried to force a highly interventionist, government-driven agenda that runs counter to the principles of free enterprise, individual freedom, and limited government that have made the United States the greatest power in the world, and the freest nation on earth.
That disconnect between Obama and the nation he leads, along with the cooperation he received from Democrats, signals the electoral backlash in November.

The problem, of course, is that Republicans may not achieve the type of majorities the Democrats enjoyed while they remade society over the objections of their constituents.  Much of the federal government creep may be difficult to lop off.

Thursday, August 05, 2010

That about sums it up

Bob Rathgeber nailed it with his assessment of PolitiFact.  Sort of.

Rathgeber, who writes for the Fort Myers News-Press, penned a little story about PolitiFact and its coverage of the Florida gubernatorial election.  The story was forgettable.  The title of the story was priceless:  "PolitiFact ferrets out the truth ... and fibs."

The title may be taken either of two ways.  "Fibs" may be taken as a noun, with the meaning that PolitiFact ferrets out the truth and also ferrets out fibs.  "Fibs" may also be taken as a verb, in which case PolitiFact fibs in addition to ferreting out the truth.

Both are "Mostly True," I'd say.