Grading PolitiFact (Florida): Alan Grayson and global wealth inequality

To assess the truth for a numbers claim, the biggest factor is the underlying message.
--PolitiFact editor Bill Adair

The issue:

(clipped from PolitiFact.com)

The fact checkers:

Louis Jacobson: writer, researcher
Aaron Sharockman: editor

Analysis:

We have here another specimen where PolitiFact makes a clear exception to its principle that the most important aspect of claims involving numbers is the underlying message.  There is no attempt in the story to either identify or evaluate the underlying argument of Democrat Alan Grayson.  One could imagine that PolitiFact did not extend the effort because Grayson's point was opinion--that unequal wealth distribution is unfair at least in some sense--but without any explanation to that effect we're left with PolitiFact relaxing its principles without explanation.

Before we start evaluating the Grayson fact check, have a look at how PolitiFact pans Sarah Palin for an accurate report of the U.S. ranking of defense spending as a percentage of the gross domestic product:

"In absolute dollars, we spend almost as much as all other countries combined," (Todd) Harrison said. "So saying we are 25th is a bit misleading and a selective use of facts."

We agree. Although she's technically correct, the numbers are wildly skewed by tiny, non-industrialized countries. We find her claim Barely True. 

The Palin rating adequately demonstrates that PolitiFact will take an underlying message into account for a number ranking claim such as Grayson's, at least on a selective basis.

Grayson, responding to MSNBC host Rachel Maddow's question about why the "Occupy Wall Street" protests resonate with Americans (yellow highlights indicate portion used by PolitiFact):

The second thing is that they`ve created a system that is enormously unequal. And the result of that is people are struggling to find a job to pay their bills, to pay their rent, to pay their credit card bills.  According to Wikipedia, there are only five countries in the entire planet that are more unequal than the United States in the distribution of our wealth. That`s a system that Wall Street created, that Wall Street maintains, and that Wall Street enforces.

PolitiFact puts immediate and exclusive focus on the numbers claim:

We should note here that we’re glad that Grayson cited his sources -- in the midst of a national TV interview no less! -- but we’re also disappointed that his source was Wikipedia, which is open to editing by anyone.

Luckily for Grayson, the Wikipedia page he cited had sourced its numbers to a peer-reviewed paper. The paper was coauthored by four academics, James B. Davies, Susanna Sandstrom, Anthony Shorrocks and Edward N. Wolff and published in The Economic Journal in 2010.

Grayson was correct that the United States had the fifth-highest Gini coefficient for wealth in the world, trailing only Denmark, Namibia, Switzerland and Zimbabwe. In other words, the U.S. distribution of wealth was more unequal than all but four other nations.

If the PolitiFact team had paid closer attention, they would have noticed that the numbers on which Grayson relies were not published in any peer-reviewed journal.  The numbers Grayson used ultimately came from an appendix (Excel file) to a working paper that preceded the version published in The Economic Journal. 

PolitiFact reports falsely in claiming that PolitiFact sourced its numbers to the linked "a peer-reviewed paper."  Sift through the linked work (see Update below) from now until doomsday and you won't find "Namibia" mentioned at all nor any comprehensive list of Gini coefficients (identified by PolitiFact as the primary statistic used to identify inequality) for all the nations of the earth.

Wikipedia apparently made no secret of the source of the information.  PolitiFact's research simply missed it (click image for enlarged view):

(clipped from Wikipedia.org)

As we continue evaluating PolitiFact's story we'll obtain some potential clues as to why the appendix did not appear with the published version of the paper.
PolitiFact:

There are a few technical caveats to note.

First, international comparisons such as these are always a bit dicey, because the statistics available for each country are not always perfectly aligned.

The first caveat is a massive understatement.

From the published paper:

The core data for this exercise are provided by national wealth distribution data that are available for 20 countries. These countries include the largest and richest countries in the world and together account for 59% of the world's population and, we estimate, 75% of its wealth. While it is interesting to look at the distribution of wealth just for these countries, and we do provide those results, our main focus is on an estimate of the full global distribution of wealth. This requires imputation of both wealth levels and distribution to the countries with missing data.

For nations outside the 20 represented in the "core data," it is not a matter of statistics "not always perfectly aligned" but a matter of the researchers approximating the statistics because of a lack of data.

Perhaps the group of researchers did an absolutely fantastic job with their approximations, but for purposes of the published article they provided cautions aplenty:

The purpose of the empirical exercise reported in this Section is to predict wealth levels in countries where wealth data are missing. This imposes certain limitations.  Most importantly, the independent variables in our regressions need to be available not only for the countries with wealth data, but also for most of those without such data – otherwise we would not be able to impute wealth to the missing countries. Fortunately, this limitation does not prevent us from estimating a sensible empirical model of personal wealth levels across countries.

The researchers' methodology represents the key review aspect of the paper, and simply publishing the paper does not serve as an endorsement of the methodology.  It is plausible that the appendix did not appear with the published version because it provides an appearance of concrete reliability unmatched by the content of the rest of the paper (see Update below).

But as much as this oversight embarrasses PolitiFact, it represents the least of the problems with Grayson's claim.

PolitiFact:

Second, most of the figures are about a decade old. These are the most recent comparisons available -- and as a result, we don’t fault Grayson for using them -- but they do represent a snapshot of time before the recession that began in late 2007, a recession notable for the implosion of the U.S. real estate market as well as the values of many other financial holdings. (To be fair, it’s unclear whether current numbers would show a higher or lower Gini coefficient for the United States, since many Americans of modest incomes lost significant wealth when home prices tanked. It’s also unclear what new data would show for other nations.)

 At least PolitiFact recognizes that the figures are over a decade old, but the spin here is ridiculous.  Does one credit the wealth inequality for the economic growth and low unemployment rate during the bulk of the Bush administration or for the economic crisis occurring in 2007?  PolitiFact opts for the latter, for no apparent reason other than to give Grayson the benefit of the doubt.

Seriously, should we register a high degree of concern that we're ranked right up there with Switzerland and Denmark in wealth inequality?  Which nations should we emulate?  Those with the lowest Gini coefficients such as Spain (0.565)?  Oh, wait.  Spain's unemployment is worse than ours now and in 2000.

PolitiFact:

Third and most important, these figures refer to wealth (that is, accumulated holdings) rather than income (the funds earned on an annual basis). To his credit, Grayson described the statistics correctly -- he very clearly said "wealth," not income.

However, to understand the full picture, it’s worth a look at income figures as well. A paper by the United Nations Development Program -- the Human Development Report 2010 -- shows that the Gini measurement for income places the U.S. more in the middle of the international pack. (Wikipedia has its own page for international income inequality, as well.)

Among the nations ranking as more unequal in income than the U.S. were Hong Kong, Singapore and Qatar among richer nations and a large proportion of middle- and lower-income nations, ranging from Niger and Mozambique in Africa to Nicaragua and Honduras in Central America to Cambodia and Thailand in Asia to Russia and Turkey in Europe.

If the income distribution rankings have us closer to the middle of the pack then what are the implications for the high ranking in wealth distribution?  What are the implications for Grayson's underlying message?

PolitiFact lets it drop and again gives Grayson the benefit of the doubt:

The peer-reviewed data backs up Grayson’s claim that "there are only five countries in the entire planet that are more unequal than the United States in the distribution of our wealth." Had Grayson used "income" rather than "wealth," the answer would have been much more mixed.

But Grayson was admirably careful in his phrasing. So we rate this claim True.

It's funny that PolitiFact credits Grayson with being "admirably careful in his phrasing" right after printing the quotation from Grayson that "there is overwhelming, staggering inequality in America, however it is measured, and that inequality substantially exceeds the inequality in many other countries."  It's neither careful nor admirable to characterize income inequality measuring near the middle of the pack as "overwhelming" or "staggering," especially with such thin (if any) correlation to general prosperity.

In short, Grayson's underlying message was obviously spurious and PolitiFact did him a profound favor by ignoring it.  Considering the underlying messages his claim was less true than Palin's claim about defense spending.  Without considering the underlying message for either the claims were equally true.  Yet Grayson ended with a "True" while Palin ended with a "Barely True."

PolitiFact's standards vary and follow a trend of partisanship.


The grades:

Louis Jacobson:  F
Aaron Sharockman:  F

Reporting inaccurately that Grayson's source material was peer-reviewed and published in The Economic Journal warrants a failing grade.  Likewise, ignoring PolitiFact's standard of stressing the importance of the underlying message for a numbers claim warrants a failing grade.

Together, they warrant the tag "journalists reporting badly," and that's without considering the journalists' failure to fully consider the meaning of the journal article and the very tenuous support Grayson could claim even from the appendix that the publishers omitted.  Jacobson and Sharockman produced a big fat mess unless their purpose was simply to undergird Grayson.

Toward that purpose the story was a rousing success.


Update (10/20/2011):  

While going back to add the Excel file I neglected to link, I noticed that PolitiFact linked to two different versions of the paper in the same paragraph (bold emphasis added):

Luckily for Grayson, the Wikipedia page he cited had sourced its numbers to a peer-reviewed paper. The paper was coauthored by four academics, James B. Davies, Susanna Sandstrom, Anthony Shorrocks and Edward N. Wolff and published in The Economic Journal in 2010.

The a peer-reviewed paper link goes to the published version that does not include the comprehensive list of nations with their Gini coefficients.  The "paper" link leads to the earlier "working paper" version that does include the appendix.  In my view, this "bait and switch" paragraph represents the type of writing an author would do if the intent was to mislead the audience.  Using two different link destinations provides the appearance of honesty (plausible deniability) while the vast majority of readers will assume that the "paper" is the same as "a peer-reviewed paper."


With luck, nobody would look into PolitiFact's work to the point of discovering the inaccuracy and the misleading language that serves to hide it from the reader. 


Oops.


I should add that, because none of the appendices associated with the working paper appear in the version published in The Economics Journal, the journal may simply have a policy of not publishing appendices.  Either way, no professional reading the journal would have the opportunity of peer-reviewing the numbers supporting Grayson's claim without seeking out the working paper.  That working paper is no more "peer-reviewed" than this blog except to the extent that a portion of it was later published in a peer-reviewed journal.  The portion not published in that journal is not peer-reviewed.