PolitiFact & the expert interview

Journalists keep secrets.

No, not secrets like a deal by the U.S. government to get banks to help find terrorists via their transaction records.  And not secrets like the existence of secret CIA prisons.  They keep other secrets.

They keep secrets such as their political leanings.  And secrets like the context of the quotations they use in their stories.  The latter secret accounts for this post, combined with my suspicion that PolitiFact reports don't always follow the best practices when obtaining comment from expert sources.

My first confrontation with PolitiFact's shortcomings in this area occurred while researching their story about John McCain's supposed earmarks.  I expressed the suspicion that Pete Sepp of the National Taxpayers Union may been taken a bit out of context.  Eventually I contacted Sepp and confirmed that suspicion to my satisfaction.

Often since I have wondered what questions PolitiFact reporters used to obtain statements from expert sources.  The reporter must take care not to ask leading questions.  Moreover, the reporter needs to tailor the question to ensure that the response reflects the expert's area of expertise.


This week I grasped an opportunity to obtain a peek behind the scenes at PolitiFact's methods.  I did a short phone interview with Todd Harrison, senior fellow with the Center for Strategic and Budgetary Assessments in the area of defense budget studies.

Harrison told me that PolitiFact sent him an e-mail asking for his opinion of two statements by Sarah Palin, and that PolitiFact intended primarily for him to evaluate the second of the two statements.  I asked if PolitiFact provided the context of the two statements from Palin.  He said no, but he was familiar with the context of her remarks prior to hearing from PolitiFact.

The good:  Asking an expert their opinion on something is an open-ended and obviously non-leading question.

The bad:  PolitiFact's reported failure to provide the context may have hampered the expert's ability to apply his expertise to the issue.

The ugly:  PolitiFact may have ended up quoting Harrison on a matter outside his realm of expertise.

Palin:
"Did you know the US actually only ranks 25th worldwide on defense spending as a percentage of GDP?"

Harrison's comment, via PolitiFact:  "In absolute dollars, we spend almost as much as all other countries combined," Harrison said. "So saying we are 25th is a bit misleading and a selective use of facts."

The first portion of the statement is right up Harrison's alley.  He is well qualified to authoritatively state that the U.S. spends almost as much as all other countries combined on defense.  But what about the latter part?

During my conversation with Harrison he more than once emphasized that measuring defense spending as a percentage of GDP was not "useful."  While considering his remarks, I wondered why experts would compile lists of non-useful statistics.  I sent Harrison the following e-mail shortly after our phone conversation:

Dear Mr. Harrison,

Thanks again for taking the time to talk to me about your interview with Louis Jacobson.

A follow up question occurred to me.  I'm not on a deadline, so reply at your leisure if you choose to reply.

You stated that comparing defense spending as a percentage of GDP was not useful.  Yet such comparisons are at least trivially useful or else nobody would keep track of the statistic for all nations.  On its face, it appears that the statistic provides a fair approximation for the priority governments place on defense.  Certainly a nation with a very poor GDP might push its spending as a percentage of GDP way up the chart just by starting up an air force with the purchase of a used Piper Cub.  So clearly the statistic is no measure of military strength.

The question is this:  With respect to your statement appearing in the PolitiFact story, what type of misdirection did you have in mind?  Put another way, what falsehood would people be inclined to believe because of Palin's statement?

(No reply yet, but I am equipped with patience and will update the story if and when I receive a reply; this version of the e-mail omits some concluding pleasantries, otherwise it is complete)

 As reflected in my e-mail, I judged that ranking nations according to the percentage of GDP spent on defense does have a use.  It provides a rough approximation of the importance a government places on defense.  Even for a country with a very low GDP, the nation's leaders need to consider the economic effects of spending a considerable percentage of the GDP on defense.

Unquestionably, the usefulness of the GDP figure falls within Harrison's scope of expertise.  For that reason, I devised a question that might encourage Harrison to explain his statement and help make clear to what degree his judgment relied on his heightened knowledge in the area of defense spending.


Find my grading of PolitiFact's rating of Palin's statement here.



Aug. 2, 2010:  Removed an extra "their" in the third paragraph. 
Aug. 13, 2010:  Added a helpful "of" in the final sentence.