Grading PolitiFact: Sarah Palin & defense spending (Updated)

"There is ... a deep anti-military bias in the media. One that begins from the premise that the military must be lying, and that American projection of power around the world must be wrong."
--Terry Moran of ABC News, broadcast on the Hugh Hewitt Show May 18, 2005

Who knew?

Sarah Palin submitted a new post to her FaceBook page.  That's news, and time again for PolitiFact to fact check.


The issue:

PolitiFact did its work for two of Palin's claims.  I'm posting the main page blurbs together as they appeared there:

I'm including both images because the second one accounts for some of the context of Palin's statement about the 25th ranking for the U.S.

The fact checkers:

Louis Jacobson:  writer, researcher
Morris Kennedy:  editor

Analysis:

PolitiFact often deals with the literal statement as well as the underlying argument.

Louis Jacobson begins:

In a June 30, 2010, Facebook post, former Alaska Gov. Sarah Palin posted excerpts from a speech she gave in Norfolk, Va., primarily on national security. At one point, she said, "Did you know the U.S. actually only ranks 25th worldwide on defense spending as a percentage of GDP?"

PolitiFact claims that it grades claims while keeping them in context.  Perhaps revealing the her overall subject, national security, adequately fulfills that goal.  Just in case, here's more:

Our Defense Secretary recently stated the “gusher” of defense spending was over and that it was time for the Department of Defense to tighten its belt. There’s a gusher of spending alright, but it’s not on defense. Did you know the US actually only ranks 25th worldwide on defense spending as a percentage of GDP? We spend three times more on entitlements and debt services than we do on defense.
(yellow highlights added)

Note that the third sentence represents this fact check.  The following sentence is the one also receiving a fact check.  What is the underlying argument?  Most likely Palin sought to support her claim that there is no "gusher" of defense spending compared to other spending and chose two stats she expected to surprise her audience in order to underline that judgment.  That Palin presents one of the factoids as a question supports that interpretation.

Assuming I am correct, PolitiFact's decision to fact check the two claims at least partially validates her underlying argument.  PolitiFact found the claim surprising, as was made clear by Jacobson statement after he had quoted Palin:

At one point, she said, "Did you know the U.S. actually only ranks 25th worldwide on defense spending as a percentage of GDP?"

We didn't, so we decided to check up on her statistic.

We quickly tracked down the chart from which we suspect she pulled her factoid. (Her staff didn't return our e-mail query.) It's a credible source -- the CIA World Factbook -- and, as Palin said, the U.S. does rank 25th in the world, spending an estimated 4.06 percent of GDP on defense in 2005.

With the "False" and "Pants On Fire" ratings apparently ruled out, let's review the descriptions for the remaining "Truth-O-Meter" possibilities:

TRUE – The statement is accurate and there’s nothing significant missing.
MOSTLY TRUE – The statement is accurate but needs clarification or additional information.
HALF TRUE – The statement is accurate but leaves out important details or takes things out of context.
BARELY TRUE – The statement contains some element of truth but ignores critical facts that would give a different impression.

Review completed, back to Jacobson:

Case closed? Not really.

The list includes all countries, regardless of size, so some tiny countries outrank the United States on the CIA list. There's Eritrea at number 9 (with an economy about 1/1000th of the size of the U.S. economy); Burundi at number 11 (with an economy that's even less than 1/1000th the size of the U.S. economy); and Maldives at 13th (with an economy roughly the same size as Burundi's).

Though Palin specifically stated that the U.S. was 25th "worldwide," apparently she misled in Jacobson's eyes.

Jacobson:

All told, only four nations on the CIA list could be described as either industrialized democracies or major world players. They are Israel (6th), Turkey (16th), China (23rd) and Greece (24th).

Probably Jacobson meant that only four of the nations rated above the U.S. on the list were major world players.  If the list "includes all countries" as Jacobson claimed earlier, then we ought to find every industrialized democracy and major world player on the list.  It's OK for PolitiFact to misstate things like that, however.  The important thing is the other guy is not accurate.  Apparently Palin should have asked "Did you know that U.S. actually only ranks 5th among major world players on defense spending as a percentage of GDP?"  That would have made a huge difference or something.

Then Jacobson drops what probably accounts for the bigger complaint:

Is there a better yardstick? We think there is -- using rankings of members of the Organization for Economic Cooperation and Development. The OECD is a group of 31 nations that are generally large, industrialized democracies. This list makes the comparison closer to one of peers.

Hmm.  How many of those "large, industrialized democracies" are members of NATO and thus substantially subsidized in their defense by the U.S.?  It's easier to list the world players not part of NATO:

• Australia
• Chile
• Korea
• Mexico
• New Zealand
• Sweden
• Austria
• Finland
• Ireland
• Japan
• Switzerland

We can add Japan and Korea immediately to the list of nations substantially subsidized in their national defense by the U.S.

Should any others qualify as resting under U.S. protection? I would suggest Australia (by treaty), Austria (nestled in the midst of NATO), Mexico (by proximity) and Switzerland (like Austria surrounded by NATO).

That leaves us only with Chile, New Zealand, Sweden and Finland as nations not unambiguously subsidized in their defense by the U.S. military. PolitiFact wants us to compare our military spending as a percentage of GDP with nations whose defense we subsidize plus two nations, Finland and Sweden, with an official policy of neutrality. New Zealand is geographically isolated and has a defense treaty with Australia. Chile has no history of using its military away from its home continent of South America. That's the best comparison? Ridiculous.

Additional sifting through the data revealed more problems. The OECD did not originate the data from which its list was derived. The data came from the Stockholm International Peace Research Institute.  And the following is from their description of their methodology:

SIPRI data reflects the official data reported by governments. As a general rule, SIPRI takes national data to be accurate until there is convincing information to the contrary. Estimates are made primarily when the coverage of official data does not correspond to the SIPRI definition or when there are no consistent time series available.

What makes SIPRI data better than CIA data?  Clearly superiority is not established by concentrating the comparison between the U.S. and other nations whose defense the U.S. subsidizes.  Just as clearly, giving non-transparent regimes (such as China) the benefit of the doubt minus strong evidence to the contrary tilts the board in favor of secretive nations.  To be sure, PolitiFact grants the caveat that secrecy is a problem--but then why prefer SIPRI as a data source in the first place?  Minus a recommendation from a consensus of neutral experts, PolitiFact's preference cannot count as an objective judgment.  It is opinion.

Speaking of expert opinions, Jacobson includes but one in his story:

Todd Harrison, a fellow with the Center for Strategic and Budgetary Assessments, said other factors set the U.S. apart.

"In absolute dollars, we spend almost as much as all other countries combined," Harrison said. "So saying we are 25th is a bit misleading and a selective use of facts."

OK, then, let's take the test.  It consists of one question:


Which of the following best matches "a bit misleading and a selective use of facts"?

A)  The statement is accurate and there’s nothing significant missing.
B)  The statement is accurate but needs clarification or additional information.
C)  The statement is accurate but leaves out important details or takes things out of context. 
D)  The statement contains some element of truth but ignores critical facts that would give a different    impression.

If you chose D then you have something in common with Louis Jacobson and PolitiFact:

Although she's technically correct, the numbers are wildly skewed by tiny, non-industrialized countries. We find her claim Barely True.

If, on the other hand, you thought Harrison's statement fit B or C just about as well if not better, I sympathize.

The fact is that Palin provided additional context that touched on the great expense involved in projecting power throughout the world:

It takes a lot of resources to maintain the best fighting force in the world – especially at a time when we face financial uncertainty and a mountain of debt that threatens all of our futures.

***

If we lose wars, if we lose the ability to deter adversaries, if we lose the ability to provide security for ourselves and for our allies, we risk losing all that makes America great! That is a price we cannot afford to pay.

***

Secretary Gates recently spoke about the future of the US Navy. He said we have to “ask whether the nation can really afford a Navy that relies on $3 to $6 billion destroyers, $7 billion submarines, and $11 billion carriers.” He went on to ask, “Do we really need... more strike groups for another 30 years when no other country has more than one?”

Well, my answer is pretty simple: Yes, we can and, yes, we do because we must. Our Navy has global responsibilities.

Palin was "technically correct."  So what was misleading in the underlying argument?  What numbers were supposedly "wildly skewed" by Palin's use of the CIA fact book?  Jacobson never identifies the target he claims Palin missed other than to claim that U.S. defense spending ought to be compared to nations either under our protective umbrella or with other reasons for low defense spending that do not allow for a fair comparison.  And he uses a dubious source to make the critical comparison with the nations who might tend to cause defense concerns for the U.S.

The grades:

Louis Jacobson:  F
Morris Kennedy:  F

PolitiFact left important data hidden and inserted opinion, if not error, by preferring a different measure based on incoherent grounds.

The fact check is not so much horrible for its rating of Palin but for the fact that PolitiFact does the same types of things for which Palin is faulted.  If it is wrong for Palin to leave out relevant data then it is wrong for PolitiFact to leave out relevant data.

Afters:

 I played with the numbers a little bit.  I went to a site that ranks the military strength of nations (globalfirepower.com).  Of those that PolitiFact counted as a "major player," the lowest rated was Greece at 22.  Number 24 was Saudi Arabia, which PolitiFact did not count as a major player.  We'll give Canada the benefit of the doubt at 23 and call her a major player. Take 25 as a percentage out of 195 nations--the total from the CIA fact page including the mighty nation known as "World," and the U.S falls in the top 13 percent.  Take five as a percentage out of 23 "major players" and the U.S. ends up in the top 22 percent. In short, Palin's method of counting all nations moved the U.S. up toward the head of the pack as ranked by GDP percentage.

Final After:  Jacobson's use of the term "technically correct" has inspired an idea for a research project.


Update:

I followed up on my mistrust of PolitiFact interview methods by contacting Todd Harrison.  See the results of my interview with Harrison with this link to the jump break of a post on PolitiFact's interview methods.