PolitiFact and lessons unlearned: the "Truth Index"

PolitiFact continues to focus on marketing at the expense of quality, with the rollout of the "Truth Index" serving as the latest example.

PolitiFact's description (by editor Bill Adair):

An average of all Truth-O-Meter rulings

To calculate the Index, we use the Truth-O-Meter ratings from our national and state sites for the prior seven days. We chose seven days to get a larger sample and avoid the dramatic swings that would occur if it were based on one day.

It's hard to imagine that even if PolitiFact controlled for selection bias that the "Truth Index" would carry any value at all other than entertainment.  Is the index supposed to show trends in the truthfulness of politicians and others who receive the PolitiFact treatment?  Again, even given appropriate controls on the selection of statements, the index probably means absolutely nothing in terms of daily trends.  We might hypothesize that as politicians increase their awareness of the popularity of fact checking sites that they would develop a less cavalier attitude toward their political speech.  But that isn't something that one would measure from day to day with a running average, even one softened using a period of seven days.

In terms of fact checking, the "Truth Index" is nonsense.  In terms of marketing perhaps it makes sense.

This is why I call PolitiFact the Fox News of fact checking:  They favor style over substance in search of ratings.

PolitiFlub: Obama & tax rates on hedge fund managers, Pt. 2

The flub (yellow highlights added):

We realize that Obama offered two different standards -- the lowest rates ever, and the lowest since the 1950s. We’ve decided to use the latter standard -- "since the 1950s" -- because the data is more consistent and easily accessible. (With one exception to fund the Civil War, there was no federal taxation prior to 1913.)

Yes, it's snarky to turn PolitiFact's own poor approach to fact checking against it.  PolitiFact earned such treatment, however.

There was, in fact, federal taxation prior to 1913 apart from a wartime exception during the Civil War:

From 1791 to 1802, the United States government was supported by internal taxes on distilled spirits, carriages, refined sugar, tobacco and snuff, property sold at auction, corporate bonds, and slaves. The high cost of the War of 1812 brought about the nation's first sales taxes on gold, silverware, jewelry, and watches. In 1817, however, Congress did away with all internal taxes, relying on tariffs on imported goods to provide sufficient funds for running the government.
infoplease.com

Of course, folks at PolitiFact almost certainly meant "federal tax on income" rather than "federal taxation," but that's not what they said.  And even if we give PolitiFact the charitable interpretation it so often denies to others, they're leaving out a few things:

[8] In 1894, another federal income tax was enacted, but it was declared unconstitutional by the Supreme Court in 1895, on the ground that it was a direct tax not apportioned among the states by population.

[9] Excise tax rates were increased and new excises ² imposed along with the imposition of an inheritance tax to help finance the Spanish-American War. These excise taxes were mostly repealed by 1902. The low revenue-yielding inheritance tax continued a few years longer.

[10] The Act of 1909 imposed a corporate excise tax of 1% on corporate net income over $5,000. Four years later, the Sixteenth Amendment (ratified on February 25, 1913) provided that "Congress shall have the power to lay and collect taxes on incomes, from whatever source derived without apportionment among the several states and without regard to any census or enumeration." 

Congressional Research Service report, hosted at taxhistory.org

PolitiFlub: Obama & tax rates on hedge fund managers

Just a little reminder regarding the way PolitiFact defines its grading system:

True – The statement is accurate and there’s nothing significant missing.
Mostly True – The statement is accurate but needs clarification or additional information.
Half True – The statement is accurate but leaves out important details or takes things out of context.
Barely True – The statement contains some element of truth but ignores critical facts that would give a different impression.
False – The statement is not accurate.
Pants on Fire – The statement is not accurate and makes a ridiculous claim.

Not that PolitiFact is entirely consistent (yellow highlights added):

TRUE – The statement is accurate and there’s nothing significant missing.

MOSTLY TRUE – The statement is accurate but needs clarification or additional information.

HALF TRUE – The statement is partially accurate but leaves out important details or takes things out of context.

BARELY TRUE – The statement contains an element of truth but ignores critical facts that would give a different impression.

FALSE – The statement is not accurate.

PANTS ON FIRE – The statement is not accurate and makes a ridiculous claim.

I wonder how much longer they'll let the discrepancy ride?  But on to the point, regarding a rating very recently given to President Obama for the following statement:

"If you're a -- if you are a wealthy CEO or a … hedge fund manager in America right now, your taxes are lower than they have ever been. They're lower than they've been since the 1950s."

The conclusion from the same story:

(W)hen Obama said that "if you're a … wealthy CEO or a … hedge fund manager in America right now, your taxes are … lower than they've been since the 1950s," he's close: Their tax rates are at or near the lows for the years elapsed since then.

The top marginal income tax rates were lower between 1988 and 1992 than they are today, but otherwise, Obama is right. They were higher for the other years. Meanwhile, the rates that are used to tax carried interest for hedge-fund managers have been at historical lows since 2003. And effective tax rates for high-income earners were either at their lowest since 1960 or very close to their lowest (at least according to the most recent data available). On balance, we rate Obama’s statement Mostly True.

Think about what PolitiFact just explained.  Obama's statement is not correct in terms of the top marginal tax rate.  Marginal rates "were lower between 1988 and 1992."  And supposedly the effective tax rates on the highest earners were "either at their lowest since 1960 or very close to their lowest."   To verify Obama's claim, PolitiFact chose a study featuring an annualized spot check for "average federal tax rate" which may not even be the same thing as the effective tax rate.  But PolitiFact could have used a Tax Policy Center study of effective tax rates published a few months ago that would have contradicted Obama's claim.

Note that PolitiFact found Obama "Mostly True" on its Truth-O-Meter.

Note again the definition of "Mostly True":  "The statement is accurate but needs clarification or additional information."

So PolitiFact judges that an inaccurate statement is accurate.

Welcome to the wonderful world of PolitiFact fact checking.

PolitiFact and Bill Adair still won't admit to selection bias

Visit msnbc.com for breaking news, world news, and news about the economy

Is Adair that dishonest or does he not really understand the problem?

PolitiFact "report cards" tell you virtually nothing about the person receiving the fact check. They tell you something about PolitiFact and what statements are chosen for scrutiny.

Blumner shows how consistency isn't always a virtue

Credit this week for Robyn "Blumñata" Blumner of the St. Peterburg Times' editoral page. She demonstrated consistency in condemning President Obama on the issue of presidential power for which she has excoriated President Bush in the past.

But Blumner's column also provides evidence of the unfortunate type of consistency:

These rhetorical criticisms that the candidate leveled at Bush are sad reminders of what we expected of Obama. He was to dismantle the prior occupant's jerry-built rationales for unlimited and unanswerable executive power. But Obama's refusal to follow the strictures of the War Powers Act says that he, too, is willing to manipulate language to ignore inconvenient limits on his power. Bush had John Yoo at the Office of Legal Counsel approving the use of torture by absurdly defining it so narrowly that it no longer included waterboarding.

The unfortunate type of inconsistency occurs when one consistently condemns something like waterboarding without producing a coherent argument.

Blumner called waterboarding torture in part because of deeply flawed essay by Judge Evan Wallach.  Wallach's essay treated substantially different types of techniques under the term "waterboarding" in order to argue it constituted torture.  Wallach's argument wallowed in the fallacy of equivocation throughout.

Suppose somebody buys an equivocation-laced argument such as Wallach's.  What better way to attack the argument on the other side than by claiming that its definitions are too narrow?  That's exactly what Blumner does.

Let's hope Blumner eventually expands her use of the good kind of consistency to avoid the bad kind.

PolitiFlub: Promise kept by Obama on tapping Strategic Petroleum Reserve? (Updated)

Yesterday PolitiFact added another "Promise Kept" to President Obama's list.

Ordinarily PolitiFact's Obameter ratings for promises kept interests me little.  Presidents tend to promise way more than they can ever deliver as a prerequisite to their election.  Obama had the good fortune to take office with a unified government and a filibuster-proof Senate majority.  So the project's a bit of a yawner.

Yesterday's item caught my attention, however.  There had to be context behind a promise to tap the Strategic Oil Reserve.  But the PolitiFact accounts mostly downplayed that context.  Here's how PolitiFact presents the so-called promise:

"Will swap oil from the Strategic Petroleum Reserve to cut prices . . . a limited, responsible swap of light oil from the Strategic Petroleum Reserve (SPR) for heavy crude oil to help bring down prices at the pump."

PolitiFact makes it sound like President Obama was pledging to use a release of oil from the Reserve as a method for dealing with the high price of petrol.

But the original context (Page 20) was more specific than that (bold emphasis added):

Swap Oil from the Strategic Petroleum Reserve to Cut Prices: Barack Obama and Joe Biden believe we have an economic emergency that requires a limited, responsible swap of light oil from the Strategic Petroleum Reserve (SPR) for heavy crude oil to help bring down prices at the pump. 

Unless President Obama believes that the oil prices constitute "an economic emergency" then this promise remains unfulfilled.  And perhaps Obama believes exactly that.  But why would PolitiFact bury that aspect of the promise with an ellipsis and refrain from considering it in the story?

My take:  PolitiFact prepared a layup for the president.  Each of the last three presidents (Bush, Clinton, Bush) tapped the Strategic Petroleum Reserve.  By obscuring Obama's statement about economic emergency, the "Promise Kept" rating fails to register as a signal of failing economic policy and instead accords with the narrative of a president who addresses problems:  He's fixing the high price of oil, not admitting to the existence of an economic emergency.


Update:

The latest Michael Ramirez cartoon makes much the same point about Obama's decision.  Hat tip to Power Line.

Is Chris Mooney correct about Jon Stewart and the PolitiFact squabble?

The other day I updated my assessment of the Stewart-PolitiFact brouhaha by evaluating the three additional survey sets mentioned in a Chris Mooney post from a few weeks back.

I credited Mooney with appropriate caution in that case.  But apparently I credited him a bit too much.  Mooney has since weighed in to defend Stewart:

(I)n an environment in which conservatives are more inaccurate and more misinformed about science and basic policy facts, the “fact checkers” nevertheless feel unduly compelled to correct “liberal” errors too—which is fine, as long as they are really errors.

But sometimes they aren’t. A case in point is Politifact’s recent and deeply misguided attempt to correct Jon Stewart on the topic of…misinformation and Fox News.

This must mean that Mooney thinks he has evidence that conservatives as a group are more inaccurate and more misinformed about science and basic policy facts.  I had earlier misinterpreted Mooney by thinking he had stopped short of that dubious conclusion.

Straight on to the evidence:

My research, and my recent post, most emphatically supports this statement. Indeed, I cited five (1, 2, 3, 4, 5) separate public opinion studies in support of it—although I carefully noted that these studies do not prove causation (e.g., that watching Fox News causes one to be more misinformed). The causal arrow could very well run the other way—believing wrong things could make one more likely to watch Fox News in the first place.

Mooney apparently thinks he has a middle ground between that staked out by PolitiFact (misinformed=uninformed) and that cultivated by Jane Hamsher of FireDogLake (misinformation=>misinformed).  I've read descriptions along the lines of "misinformed is believing you know something when you don't," which is distinguished from simply not knowing.  Mooney tried to narrow the meaning of Stewart's statement to apply to "science and basic policy facts" rather than to things like which party controls Congress.  I'm not sure Mooney's case is justifiable, but I don't expect the attempted distinction to matter in the end.

Mooney:

Politifact wasn’t even aware of the studies I’ve cited. Instead, the site’s attempt to debunk Stewart largely relied on misunderstanding what he meant.

PolitiFact cited two from Mooney's initial list of five (the two PIPA surveys).  I'm not sure how Mooney missed that.  Perhaps he accidentally omitted a word or two (Mooney later notes that PolitiFact used the PIPA studies).  As I pointed out, the remaining three studies share the same problems that make the PIPA studies unsatisfactory.  Selection bias in determining what information to test infects all five.  To test the general level of political policy and/or scientific information one needs a test designed for that purpose.  Each of Mooney's citations manifests a narrow focus.  Each of the studies features ambiguous statements of fact, which will tend to skew the answers and, as a result, the conclusions.

Mooney:

What Stewart obviously meant—and what I mean—is that when it comes to politicized, contested issues where the facts have been made murky due to political biases, it is Fox viewers who are the most likely to believe incorrect things—to fall prey to misinformation. A quintessential example of such an issue is global warming, or whether Saddam Hussein’s Iraq possessed weapons of mass destruction or was collaborating with Al Qaeda. There are many, many others.

I suspect that the segue from "politicized, contested issues where the facts have been made murky" to "most likely to believe incorrect things" is not as clean as Mooney appears to suggest.  The WMD issue serves as a case in point.  It is absolutely undeniable that Saddam Hussein's Iraq possessed weapons of mass destruction.  Possessing them was one of the prerequisites for the ceasefire condition requiring their destruction.

What?  Mooney is talking about immediately prior to the invasion?  Even then, Iraq unquestionably possessed weapons of mass destruction, albeit old, small in number and of very questionable effectiveness.

It is exactly that type of ambiguity that renders the PIPA studies and their like relatively worthless as a measure of individual and group misinformation.  Notably, that type of ambiguity tends to occur precisely when the author thinks it is obvious what is meant because of their own ideological predisposition.  A liberal like Mooney will automatically assume that it refers to large stockpiles of ready-to-go WMD.  A conservative, maybe not so much.  Therefore the conservative is misinformed?

Mooney also skimps on the fact that large differences occur regarding beliefs in various supposed facts.  For example, his citation of the Kaiser Family Foundation study examining beliefs about the health care reform legislation found that Republicans were more likely than Democrats to possess an awareness of the individual mandate feature (73 percent to 63 percent).  The questions asked and the phraseology used make all the difference.

Mooney:

It is of course around contested political facts, and contested scientific facts, where we find active, politically impelled, and emotionally laden misinformation campaigns—and it is in the latter realm that Fox News viewers are clearly more misinformed. Once again, I’ve cited 5 studies to this effect—concerning the Iraq war, the 2010 election, global warming, health care reform, and the Ground Zero Mosque. By contrast, Politifact only cites two of these studies, and attempts to critique one of them (the 2010 election study)—misguidedly to my mind, but who really even cares. It is obvious where the weight of the evidence lies at this point, unless further, relevant studies are brought to bear.

With all due respect to Mooney, the "weight of the evidence" is a vapor.



Is it possible that Fox News viewers as a group are misinformed in the category Mooney names?  Sure, it's possible.  And PolitiFact won't get to the bottom of it without finding a new and groundbreaking study.  PolitiFact uses a fallacious "burden of proof" criterion and may justify rating Stewart "False" according to that standard.  But on the full set of facts as we have them, Stewart is just wrong.  There are no studies that scientifically support the conclusion he's peddling.  Mooney shares the error.


Afters:

Mooney updated his post with a reference to another study supposedly supporting Stewart:

I've run across (thanks to Steve Benen) a sixth survey that supports Stewart.

The cited NBC poll supports Stewart no better than the other five.  Note the selection bias and the ambiguity (click to enlarge):

Mooney.  Dude.  Link the primary source.

Democrats' message board vigilantes

As if PolitiFact's Matrix isn't bad enough, now we've got message-board vigilantes.

Check out this pair of posts from PolitiFact's FaceBook page:

No doubt Hussain confirmed that I am not a real person by using his tricorder.

Wallenius needed no tricorder.  Her techniques derive from the classic sleuths such as Sherlock Holmes, Lord Peter Wimsey and Columbo.  Or maybe Adam West's Batman:

Jen Emmelman
Sheila, how do you know that about Bryan? I mean I am aware that he is a troll, but how did you figure out the rest?


Sheila Fahey Wallenius
Jen: two things - recent meeting I attended exposing the "professional troll training" the tightie righties are engaging in, so i knew what to look for. One of those things to look for is see where they're posting.

If you looked at his Wall - - going back nearly a full MONTH - - the ONLY PLACE that "Bryan" posts IS on "PolitiFact" - that tells me he's been specifically assigned to PolitiFact. He'll be back I'm sure, but now I've ID'd him, reported him and blocked him, so it might slow him down a bit for awhile.

Others are "assigned" to the various Obama campaign pages; still others to the White House page, etc. etc.


Sheila Fahey Wallenius Jen, I believe there's a You Tube video somewhere taken by someone with a cell phone cam at one of the "professional troll trainings"

The tactics they use are fascinating - dishonest and despicable as hell - but fascinating.

But, y'know... when you don't have the TRUTH, FACTS & EVIDENCE on your side, that's the kind of scumbag stuff I would expect the tightie righties and teabaggers would resort to.

My troll tactic with Wallenius was to point out that PolitiFact uses Heritage Foundation material in its source lists.  Wallenius denied it:

PolitiFact CITES Heritage Foundation, ONLY when they're drawing a comparison between statements coming from a right leaning group to those made by a lef-leaning (sic) group.

It wasn't hard finding an exception to Wallenius' claim.

Apart from her unsupported claim regarding PolitiFact's use of Heritage Foundation as a source, the disturbing thing about Wallenius is her resemblance to the evil spectre she tried to paint of me.

I'm supposedly some sort of agent, trained by the GOP to troll innocent message boards and assigned to the PolitiFact message board in particular--let alone the fact I've been criticizing PolitiFact under my own name and plainly announced the purpose of my FaceBook account many months ago. Meanwhile, Wallenius actually attends some sort of seminar to help her ferret out GOP agents and neutralize them.  One of us is a party operative, anyway.

Holy GOP imposter, Sheila!



Thank you, reality-based community. Wonderful stuff.

More misinformation about the misinformed

This is an update to the recent controversy over Jon Stewart's PolitiFacted claim that polls show Fox News viewers as the most consistently misinformed group in "every poll."

In my reply to Jane Hamsher's defense of Stewart I dealt briefly with the problems inherent in the two studies produced by the Program on International Policy Attitudes.

Many on the Web have started referencing a list of five polls cited by Chris Mooney purporting to demonstrate "that those who watch Fox, or watch it frequently, are more likely to be misinformed."  Two from Mooney's list are the same two PIPA surveys cited by Hamsher.  The other three share a similar set of problems.

Please note that Mooney wrote in advance of the Stewart controversy and that I take the quotation from Mooney in the preceding paragraph somewhat out of context.  Mooney is admirably cautious about drawing solid conclusions from flimsy evidence, and the quotation tends not to convey his careful approach to the issue.

Stanford University/Jon Krosnick
Krosnick's survey dealt exclusively with issues related to global climate change, and thus serves as a poor standard for judging the overall set of misinformation possessed by a group.  This is another manifestation of the selection bias problem found in both PIPA studies.  With no control on the selection of items intended to reflected general misinformation the end result will reflect selection bias.  Additionally, the study offers no method of discerning whether Fox News actively misinformed its viewers.  As Krosnick notes in his conclusion:

It is impossible to discern from these results what causal processes produced the observed relations.

Kaiser Family Foundation
The KFF study focused on a set of beliefs regarding the health care reform legislation.  Obviously, as with global climate change, misinformation about a particular health care reform bill serves as a relatively poor measure of general misinformation, political or otherwise.  In addition, a number of the truth propositions qualify as ambiguous.  For example, "Allow a government panel to make decisions about end-of-life care for people on Medicare" might serve to describe the effects of cost saving measures occurring as the result of the Independent Payment Advisory Board created under the reform legislation.

Nisbet & Garrett at Ohio State University
The Ohio State researchers examined misinformation about the so-called "Ground Zero Mosque," certainly another example of a narrow topic that fails to obviously serve as an indicator of political misinformation in general.  The researchers focused on a set of four "rumors" regarding the proposed mosque.


What does it all mean?

If Stewart wants to say that Fox viewers are misinformed as to some specific statement then the survey data can offer him some support, depending on the specific charge.  But none of the five surveys mentioned by Mooney serves as a good support for the type of broad claim Stewart made during his appearance on Fox News.

Hey ... maybe Stewart spoke false because he was appearing on Fox News ... ?

The misperceptions misperception mushrooms (Updated x2)

PolitiFact, it seems, has mired itself in the misperceptions marsh.

The problem started when the fact checkers went to test a recent claim made by comedian/Daily Show host Jon Stewart during an interview with Fox News' Chris Wallace (yellow highlights added):

WALLACE: I don't think our viewers are the least bit disappointed with us. I think our viewers think, finally, they're getting somebody who tells the other side of the story.
STEWART: Right.
(CROSSTALK)
WALLACE: And in -- no, no, no. One more example.
STEWART: Who are the most consistently misinformed media viewers? The most consistently misinformed? Fox, Fox viewers, consistently, every poll.
WALLACE: Can we talk about your network? Can we talk about Comedy Central?

PolitiFact's story ended up rating Stewart's statement "False," noting that the bulk of the available polling showed that Fox viewers were better informed than those who follow some other news sources.

Then things got interesting.