Good on Brendan Nyhan

I've been critical enough of the research of political scientist Brendan Nyhan that I owe some recognition to a pretty good column he wrote up at Columbia Journalism Review.

The title has a good hook, and it's worth a read:

Obama ‘evolves,’ Romney ‘flip-flops’

PolitiFact pimps NPR's Brendan Nyhan puffery

The truth-hustlers at PolitiFact have invoked fellow truth-hustler Brendan Nyhan to undergird their self-serving narrative of special expertise in divining the truth.

Who doesn't remember Bill Adair defending PolitiFact's 2011 "Lie of the Year" award based on the fact that most people live in a partisan echo chamber?

National Public Radio indulged its Nyhan fetish again under the story heading "Partisan Psychology: Why Do People Choose Political Loyalties Over Facts?"

Try this representative sample:

The flipped perceptions on gas prices isn't an aberration, said Dartmouth College political scientist Brendan Nyhan. On a range of issues, partisans seem partial to their political loyalties over the facts. When those loyalties demand changing their views of the facts, he said, partisans seem willing to throw even consistency overboard.

Nyhan and Reifler (as they regularly seem to do) fail to take into account the complexity of political issues.  Many things affect gas prices, and partisans and non-partisans alike will adopt complex explanations for changes in the price of gas.  Survey data such as that used by Nyhan strips away subtleties and slots the answers into a narrow set of choices.  Most and perhaps all of the difference noted between the apparent change in party belief probably results from simply placing greater emphasis on one aspect of the issue, almost as though people have a natural ability to sell used cars.  If the car isn't better because of outstanding gas mileage then it's better because of large trunk capacity.  Likewise the presidential role in setting gas prices gets communicated according to a favorable narrative.  And it doesn't necessarily mean that the person selling the apparently inconsistent narrative doesn't understand the truth.

Don't buy a used car from Nyhan and Reifler, by the way.

NPR and truth-hustler guests Adair and Nyhan

National Public Radio brought two of my favorite truth-hustlers on its "All Things Considered" program.   I refer to Bill Adair and Brendan Nyhan.

(clipped from npr.org 4/29/12)

Bill Adair serves as founding editor for Politico PolitiFact, a fact-checking operation started by a left-leaning Florida newspaper.  PolitiFact sacrifices journalistic objectivity in part for the sake of its marketing gimmick, called the "Truth-O-Meter."

Political scientist Brendan Nyhan mangles facts from the realm of academia.  Nyhan has tried to show that partisans don't accept facts that contradict their ideology.  His research often uses facts that beg the question (more on Nyhan), suggesting that Nyhan falls victim to his own research goal.

The relevant radio program segment deals with a column from Chicago Tribune columnist Rex Huppke, who bemoaned the state of truth following Rep. Allen West's statement, using Huppke's paraphrase, that "as many as 81" members of the Democratic Party are members of the Communist Party.

Huppke gets some kind of award for irony.  Journalists took West out of context.  West was jokingly, though to make a seriously point, referring to the Congressional Progressive Caucus.  The context makes that absolutely clear.

Stories like these are to Adair and Nyhan what the Trayvon Martin case is to Jesse Jackson and Rev. Al Sharpton.  The media want the narrative that the former two spin and so seek them out for commentary when these issues make the news.  Adair and Nyhan are the truth-hustler counterparts to race-hustlers.

Adair shows up to remind us how great it is that Politico PolitiFact is there to save the day, if only we can place sufficient trust in its work.

Adair (transcript mine, from the NPR audio):

"What's funny is sometimes I'll get an email that'll say 'You guys are so biased.'  But I won't know who we're supposed to be biased in favor of, because we get criticized a lot by both sides.  And I think that's just the nature of a very rough-and-tumble political discourse."

Funny is Adair still using the "we get criticized from both sides" dodge to avoid the issue of bias.  Fortunately a journalist asked Adair exactly the right question earlier this year with equally hilarious results.

PolitiFact's system is perfect for filtering the truth according to media ideology.

Academia, unfortunately, carries an ideological slant somewhat akin to that found in the U.S. media.  Nyhan perhaps represents one of academic liberalism's top leaders in the war over political truth. 

NPR brought forth an old example of Nyhan's supposed "backfire" effect, where a correction of a falsehood leads to stronger belief in the falsehood. Though Nyhan's own research  (see descriptions of "Study 2") appears to show that the phenomenon does not occur with clear corrections, that hardly dampens mainstream media enthusiasm for the idea.  They can claim they're doing a great job but the audience is the problem.

There is something of an information crisis, but Adair and Nyhan probably do as much damage as good in addressing the problem.  We're not getting the best information from either journalists or academia.  Journalists typically do not have the expertise to sift through complex issues of truth.  Academics have shifted left ideologically and do an inadequate job of critically reviewing the journals that ought to provide our best sources of trustworthy information.

We don't have a reliable gatekeeper for our pool of information.  And it's hard to come up with good solutions to the problem.

Test everything. Hold on to the good.
--Paul the Apostle, 1 Thess. 5:21

The meta-savviness of Brendan Nyhan

Political scientist Brendan Nyhan is back--not that he ever went away--with another of his patented faux-objective assessments of the U.S. media's relationship with its audience.

Nyhan's fundamental trouble remains his difficulty in assessing the aquarium habitat while always swimming in the tank with the rest of the fish.

His pet issue in his story is the tendency of news reports to continue to use "he said/she said" accounts of political disputes:

The first obligation of journalists is to the truth. As such, it is important that reporters set the record straight when ads like these are misleading their audience. The problem, however, is that many national reporters—and the state reporters who increasingly emulate them—have been sucked in by the cult of the savvy. For these journalists, producing meta-level analysis of the effectiveness of deception as a campaign tactic is more important than correcting the factual record for readers.

I, for one, do not see why Jay Rosen's description of the journalistic "cult of savvy" would be incompatible with the solution Nyhan recommends:

A better approach would be for reporters to characterize the accuracy of ads in their own voice and to invoke non-partisan experts like PolitiFact. In some cases, it may even be possible to find credible sources on the side of the candidate airing the misleading ad who are willing to state the truth.

So journalists will let readers in on their supposedly specialized knowledge regarding the truth of political claims.  And then what?  The cult of savvy will proceed to meditate on the effects its findings should have on the reader, no doubt abundantly citing Nyhan's flawed research in the process.  Both steps leave ample room for the cult of savvy to direct the journalistic approach.  Glenn Greenwald, for example, pointed out the tendency at PolitiFact to use expert sources from both sides of an issue. The main difference with the traditional he said/she said approach is that PolitiFact not infrequently ends up making an arbitrary decision as to which expert opinion carries the day.

If the journalists taking that step are both expert and not ideologically biased, then fine.  But who buys either proposition?

Apparently the meta-savvy Brendan Nyhan buys it, as he refers to PolitiFact as "non-partisan experts."

Give me a break.

Nyhan explains Birther reversal--sort of

Awhile back I did some tweaking of political scientist Brendan Nyhan over his prediction that the release of President Obama's long form birth certificate would not significantly dent the proportion of Americans who believe Obama was not American-born.

A Washington Post poll showed a big shift in belief as well as a surprising (to me, anyway) number of self-identified Democrats who doubt Obama was born in the United States.

I figured that left Nyhan with some explaining to do.  And, to his credit, Nyhan has stepped up to the plate to do just that:

So why was this correction so effective when others tend to fail? (PDF) The answers aren't entirely clear yet, but here are some initial thoughts. First, the birth certificate's release was an unusually definitive debunking that became a major news event, so there was saturation coverage of some very strong corrective information.

Nyhan's first possible explanation is interesting given that his past "misperception" studies place little weight on the strength of the debunking material.  Indeed, even the supposed misperceptions are often controversial regarding the degree of truth they hold.

Nyhan ought to have considered the strength of the debunking in this case before putting his doubts in print.  It ended up making him look a tad foolish.  He should also have considered that Obama's failure to release the long form certificate when it was apparently in his power to do so did plenty to help fuel the "Birther" movement.  Of course cutting off a major fuel source will affect the level of belief under such conditions.

Nyhan:

Second, no prominent elites on the right contested the validity of the birth certificate, which meant that coverage of its release was almost entirely one-sided.

That fits with my counter-Nyhan argument that the source of contradicting information has much to do with whether we accept a supposedly strong debunking account.  Perhaps it is a failure on my part, but I don't recall Nyhan taking such things into account for purposes of his past research.

Nyhan:

Finally, it's possible that support for the myth was soft because poll respondents didn't really believe it but were using poll questions about Obama's religion and place of birth as a way to express disapproval (as some commentators and pollsters have argued).

Such influences on polling, if they exert an appreciable effect on the results, certainly must make it tough for people like Nyhan to do their research.  How is a researcher to know when people are telling the truth about their beliefs?

Nyhan's three would-be explanations all tend to undercut the conclusions he's drawn from his research.  Will this contradictory information correct his misperceptions?  Was the debunking unambiguous enough to shake his beliefs?

WaPo poll: Release of Obama birth certificate cuts # of birthers in half

This looks like job for ...

Brendan Nyhan!

I've been tweaking above-the-fray political scientist Nyhan over his series of "misperception" studies for some time.  This one gives him some explaining to do, particularly in light of something he wrote in anticipation of the reaction to President Obama sharing his long-form Hawaiian certificate of live birth:

Given how much evidence is already available, it's hard to see why a long-form birth certificate would suddenly change the minds of people who are predisposed to believe in the myth. The hardcore are already shifting to new rationales for questioning Obama's right to hold office and deconstructing the PDF released by the White House for supposed evidence of forgery.

Perhaps if Nyhan's conclusions had been as equivocal as his data then he'd have less to explain.

Here's one blogger enthusiastically awaiting Nyhan's explanation.  Especially for the drop in Democrat birtherism from 15 percent all the way down to 7 percent, or about the same reduction by percentage as found among  conservative Republicans.

Brendan Nyhan and the fudge factor

Is Paul Krugman the only liberal who's comfortable with death panels?

PolitiFact chose Sarah Palin's "death panel" comment on FaceBook as its "Lie of the Year" in 2009.  In 2010, political scientist (and professional handwringer) Brendan Nyhan continues to bemoan the damaging branding of Democrats' attempts to reform health care.

I've found Nyhan less than scientific in his pronouncements over time, and his 2010 paper on the supposed influence of GOP disinformation on public attitudes about health care reform provides additional grist for the mill.  In brief, Nyhan's reasoning suffers from unhealthy dollops of equivocation.


Equivocation

The equivocation occurs in the term "death panel."  Nyhan doesn't seem sure what he means by it from one moment to the next, unless he understands it as a semantic grab-bag designed to fill whatever need his reasoning requires.  On page one, in the abstract, "death panels" represent the refusal of care to the elderly because of government bureaucracy:

(M)isconceptions also clouded the recent debate about health care reform under President Obama, including the myth that the elderly would have medical care denied by so-called government “death panels.”

When Nyhan begins to trace the history of the so-called "Obama 'Death Panel' myth," however, death panels transform into end-of-life counseling that might encourage the elderly to forgo medical treatment (p. 8):

McCaughey had her greatest impact on the debate during the summer of 2009 when she invented the false claim that the health care legislation in Congress would result in seniors being directed to “end their life sooner.”

Nyhan goes on to quote McCaughey from an appearance on a radio program with Fred Thompson (R-Ten.).  The same statement is later identified as the start of the "death panel" myth on a chart:

Only after the claim was "embellished by Sarah Palin" (8/7/09 by Nyhan's chart) did it begin to resemble the version Nyhan identified in his abstract.

Question:  In what respect is the so-called myth spreading if it alters its character almost completely in a few weeks?

Sarah Palin's statements about "death panels," taken in context, cannot easily be dismissed.  Though not for lack of trying by the mainstream media and liberal elites like Nyhan:

After coming under criticism, Palin defended her statement by citing the counseling provision identified by McCaughey and academic articles previously written by Obama adviser Ezekiel Emanuel (Palin 2009a). However, independent observers condemned her claim about “death panels” as false (FactCheck.org 2009b; PolitiFact.com 2009c): there was simply no evidence that funding for voluntary end-of-life consultations would create a mechanism for ‘‘‘bureaucrats” to withdraw care from “[t]he sick, elderly, or disabled.”

Saying that Palin "defended her statement by citing the counseling provision" is misleading.  Rather, Palin's follow-up to her "death panel" post was a reply to President Obama, who had mused that Palin's "death panel" referred to the end-of-life counseling provision in the bill.

Palin:

The provision that President Obama refers to is Section 1233 of HR 3200, entitled “Advance Care Planning Consultation.” [2] With all due respect, it’s misleading for the President to describe this section as an entirely voluntary provision that simply increases the information offered to Medicare recipients. The issue is the context in which that information is provided and the coercive effect these consultations will have in that context.

Palin goes on to make a strong case for herself, not that a person getting the story from Nyhan would have any idea (speaking of spreading misperceptions).  Nyhan's next statement went on to acknowledge that health care reform may lead to more restrictive rationing but that those consequences hardly justified Palin's statement about death panels.  One wonders what aspects of science or logic permit Nyhan to reliably make that judgment.

Despite Nyhan's acknowledgment of increased rationing, he counts it as a misperception when poll respondents indicate that they believe the government will choose not to treat some people.

I compare misperceptions about the Clinton plan to those about the Obama plan using the following question from a CNN/ORC poll in September of 2009 (2009):

Based on what you have read or heard about (Barack) Obama’s health care plan, please tell me whether you think each of the following would or would not happen if that plan became law....If Obama’s plan became law, do you think senior citizens or seriously-ill patients would die because government panels would prevent them from getting the medical treatment they needed?

41% of respondents said government panels would withhold medical treatment, 57% said it would not happen, and 2% had no opinion.

Based on Nyhan's admission, doesn't it seem that the 57 percent are the ones probably carrying the misperception?

Not only does Nyhan change the definition of "death panel" midstream, he magically transforms a reasonable expectation into a "misperception."

Nyhan explicitly disavows the fallacy of post hoc ergo propter hoc but can't seem to resist invoking it in practically the next sentence:

(T)he likelihood of support for Obama’s plan decreased from 72% among Americans who did not believe the “death panel” claim in 2009 to 20% among those who did. As noted above, these differences, which hold even among non-Republicans, were not necessarily caused by misinformation, but they do suggest its potential importance for public opinion.

Implications and Conclusions

The evidence presented in this article suggests that misinformation played an important role in the two most recent debates over health care reform.

The supposed suggestion relies on the post hoc ergo propter hoc inference, though Nyhan doesn't even provide before-and-after survey data that would support the assertion that attitudes changed after the "death panel" myth took flight.  Perhaps, as with the 2009 stimulus bill, we can measure the effect of the misinformation by how much better the surveys would have looked without the existence of the death panel myth.

 I find it very difficult to be impressed by Nyhan's paper.

Researchers hoist with own petard?

In a recent post I took note of the publication of new research purported to support the notion that partisans reject information that contradicts strongly held beliefs more readily than do non-partisans.  I've identified and read the study by Brendan Nyhan and Jason Reifler.

The verdict?  Read on.

One of the instantly striking aspects of the study, titled "When Corrections Fail: The Persistence of Political Misperceptions," was the redefinition of "misperception" employed by the authors:

(W)e define misperceptions as cases in which people’s beliefs about factual matters are not supported by clear evidence and expert opinion—a definition that includes both false and unsubstantiated beliefs about the world.

That definition occurs in a section devoted to the case for redefining the term for purposes of the study (pp. 304,305).  I suggest to the researchers that a term redefined to reflect that "factual matters that are the subject of contemporary political debate are rarely as black and white as standard political knowledge questions" will result in an increased ambiguity in the finished results where the redefined term occurs.  In particular, if news reports seize on the term "misperception" without mentioning the spin placed on the term by Nyhan et al., the news report will end up producing or reinforcing an unsubstantiated belief in the reader.  Which we might call a "misperception" regardless of whether it contradicts the expert opinions of Nyhan and Reifler.  Clear evidence, after all, constitutes a requirement according to their definition.

The study provides fairly clear evidence that Nyhan and Reifler were "hoist by their own petard."

They're baaaack ... (Updated)

Boston.com (The Boston Globe) has resurrected the story about studies supposedly confirming that political leaners right and left entrench rather than rethink when confronted with facts that contradict their erroneous views.

It sounded familiar, and with good reason.  The main point of the story rested on studies led by political scientist Brendan Nyhan:

Recently, a few political scientists have begun to discover a human tendency deeply discouraging to anyone with faith in the power of information. It’s this: Facts don’t necessarily have the power to change our minds. In fact, quite the opposite. In a series of studies in 2005 and 2006, researchers at the University of Michigan found that when misinformed people, particularly political partisans, were exposed to corrected facts in news stories, they rarely changed their minds. In fact, they often became even more strongly set in their beliefs. Facts, they found, were not curing misinformation. Like an underpowered antibiotic, facts could actually make misinformation even stronger.

The story later identified Nyhan as the lead researcher.

I dissected that study as well as one led by John Bullock in a Sept. 17, 2008 post.

Nyhan's problem was his attempt to simulate the manner in which individuals encounter contradictory facts, as by reading a newspaper.  That approach is flawed because people can have good reasons for not trusting the newspaper over their own beliefs.  Beyond that, the specific cases used in Nyhan's study had internal ambiguities that very likely affected the outcomes.

The story also refers to a study from 2000 conducted by James H. Kuklinski:

Kuklinski’s welfare study suggested that people will actually update their beliefs if you hit them “between the eyes” with bluntly presented, objective facts that contradict their preconceived ideas.

The source of the contradictory information makes a difference.  What else would we expect?

Find the Kuklinski study here.

The story referred to one other study:

New research, published in the journal Political Behavior last month, suggests that once those facts — or “facts” — are internalized, they are very difficult to budge.

So far as I could determine, the story did not provide any substantial material from the study in Political Behavior, nor did it name any of the authors responsible for the study.  But I look forward to surveying it once I've located it.


Update:

The new study in Political Behavior is apparently a reworked version of the old Nyhan study and was again led by Nyhan.  I'll update in a new post after I assess whether the researchers effectively minimized the problems with the earlier study.


July 14, 2010:  Reworked for the first sentence in my fifth paragraph for clarity.

Does science suggest that fighting smears is futile?

A news story I picked up through Michelle Malkin's Hot Air has piqued my interest, given the amount of time I spend trying to sort through media disinformation for the truth.

The story, by Shankar Vedantam, concerned experiments conducted by political scientist John Bullock and other experiments by political scientists Brendan Nyhan and Jason Reifler. The experiments, according to Vedantam and the researchers, suggested that a smear successfully affected perception even after the smear was debunked. Further, the research by Nyhan and Reifler supposedly suggested that conservatives were more prone to accept a smear despite having access to information that falsified it.

But there are problems with this type of research, as I shall demonstrate.

I will deal with the experiments in the order Vedantam presents them.


Roberts and NARAL

The first concerned a smear of current Supreme Court Justice John Roberts produced by NARAL.

clipped from www.factcheck.org

NARAL Pro-Choice America TV Ad:
"Speaking Out"

Announcer: Seven years ago, a bomb destroyed a women's health clinic in Birmingham, Alabama.

(On screen: Footage of bombed clinic)

(Tex on screen: New Woman/All Women Health Clinic; January 28, 1998)

Emily Lyons: When a bomb ripped through my clinic, I almost lost my life.

Announcer: Supreme Court nominee John Roberts filed court briefs supporting violent fringe groups and a convicted clinic bomber.

(On screen: Footage of Roberts; image of April 11, 1991 brief from Bray v. Alexandria)

(Text on screen: Roberts filed court brief supporting clinic protesters)

Emily Lyons: I'm determined to stop this violence so I'm speaking out.

Announcer: Call your Senators. Tell them to oppose John Roberts. America can't afford a Justice whose ideology leads him to excuse violence against other Americans.

Picking up with Shankar's story:

Bullock then showed volunteers a refutation of the ad by abortion-rights supporters. He also told the volunteers that the advocacy group had withdrawn the ad. Although 56 percent of Democrats had originally disapproved of Roberts before hearing the misinformation, 80 percent of Democrats disapproved of the Supreme Court nominee afterward. Upon hearing the refutation, Democratic disapproval of Roberts dropped only to 72 percent.

On the face of it, this looks bad. The data appear to suggest that 16 percent of the Democrats in the study allowed a plainly refuted smear to color their opinions. But how solid is that conclusion?

I suggest that the conclusion only follows if the ad provides no damaging information that is not thoroughly and unambiguously countered by the refutation. That is a tall order, and one that seems well outside the reach of Bullock's experiment. Shankar's version of the experiment fails to do it justice:

Near the end of the experiment, treatment-group subjects received this information:

Recall the ad transcript you read about earlier in this survey. The ad was strongly criticized by many people, some of whom were prominent supporters of abortion rights.
Walter Dellinger, an ally of the group that aired the ad and an important attorney in the Clinton administration, called the ad “unfair and unwarranted.” He added that “It is unfair to suggest that John Roberts, in advancing a somewhat narrow interpretation of [the anti-discrimination law], was supporting ‘violent fringe groups and a convicted clinic bomber’—as unfair as it would be to suggest that the six Justices who were part of the majority in Bray joined a decision supporting violent fringe groups.”
Arlen Specter, a Republican senator and supporter of abortion rights, called the ad “blatantly untrue and unfair.”
Stung by these criticisms and many others, NARAL Pro-Choice America withdrew the ad from television.

(Bullock, "The Enduring Importance of False Political Beliefs," p. 39)

The supposed refutation itself contains information that might influence the test subjects to change their opinion of Roberts for the worse, given that he supported a "narrow" interpretation of an anti-discrimination law. That is aside from other obvious factors, such as a general familiarity with NARAL, which might influence test subjects to think that even if the ad was false, the organization would not have aired it if Roberts presented no worries for a pro-choice voter.

This phase of the experiment does not significantly support the notion that a refuted smear continues to affect attitudes.


Guantanamo, Koran, Toilet

The second smear involved the false reports of Guantanamo guards flushing a copy of the Koran down the toilet.

Volunteers were shown a Newsweek report that suggested a Koran had been flushed down a toilet, followed by a retraction by the magazine. Where 56 percent of Democrats had disapproved of detainee treatment before they were misinformed about the Koran incident, 78 percent disapproved afterward. Upon hearing the refutation, Democratic disapproval dropped back only to 68 percent -- showing that misinformation continued to affect the attitudes of Democrats even after they knew the information was false.

The construction of this experiment was more complicated than the other, and it seems that Shankar misinterpreted the results. No baseline attitude toward detainee treatment occurred prior to exposure to the texts used in the experiment. The 56 percent of Democrats who disapproved of detainee treatment were part of a control group who read an account of detainee mistreatment with the Koran-flushing episode excised. The 78 percent figure Shankar presents as disapproval in the (Democratic portion of the) control group appears to be a figure actually related to approval for congressional hearings on detainee treatment. I located no pre-refutation figure for the treatment group, which leads me to believe that the control group was intended to provide the baseline (contrary to the picture painted by Shankar). The refutation would have been meaningless and irrelevant to the control group, since Newsweek would be retracting a part of the story they had not read.

Apart from Shankar's difficulty in reporting the experiment, did it support Bullock's thesis? Since 68 percent of the treatment group disapproved detainee treatment even after the refutation compared to 56 percent in the control group, perhaps so. The key variable is the text of the Newsweek retraction, which was not included in Bullock's paper.

If the retraction was similar to the following, however, there is good reason to doubt:

Last Friday, a top Pentagon spokesman told us that a review of the probe cited in our story showed that it was never meant to look into charges of Qur’an desecration. The spokesman also said the Pentagon had investigated other desecration charges by detainees and found them “not credible.” Our original source later said he couldn’t be certain about reading of the alleged Qur’an incident in the report we cited, and said it might have been in other investigative documents or drafts. Top administration officials have promised to continue looking into the charges, and so will we. But we regret that we got any part of our story wrong, and extend our sympathies to victims of the violence and to the U.S. soldiers caught in its midst.
(Outside the Beltway, quoting from Mark Whitaker's "From the Editor's Desk")

A "retraction" like that above tends to feed into the level of trust one feels for the media source as opposed to the government source, which provides a natural and potentially reasonable alternate explanation for the belief perseverance effect Bullock was exploring.

Hussein and WMD

The next example in Shankar's story comes from Nyhan and Reiffler.

Political scientists Brendan Nyhan and Jason Reifler provided two groups of volunteers with the Bush administration's prewar claims that Iraq had weapons of mass destruction. One group was given a refutation -- the comprehensive 2004 Duelfer report that concluded that Iraq did not have weapons of mass destruction before the United States invaded in 2003. Thirty-four percent of conservatives told only about the Bush administration's claims thought Iraq had hidden or destroyed its weapons before the U.S. invasion, but 64 percent of conservatives who heard both claim and refutation thought that Iraq really did have the weapons. The refutation, in other words, made the misinformation worse.

I don't imagine it was an easy matter to find volunteers willing to read the Duelfer report (not the work of mere minutes).

While the future size and direction of the Iraq Survey Group are currently under review, the requirement remains to collect further information related to threats posed by residual elements ofthe former Regime’s WMD programs. There will also be new information from individuals and sources, which will come to light. Moreover, certain defined questions remain unanswered. For example, we cannot express a firm view on the possibility that WMD elements were relocated out of Iraq prior to the war. Reports of such actions exist, but we have not yet been able to investigate this possibility thoroughly.

It should be obvious merely from the above excerpt (and there are many similar caveats in the report) that the Duelfer report did not unequivocally refute the notion that Iraq possessed WMD. Indeed, though the report emphasized the lack of solid evidence supporting the notion, it also emphasized the devious intent of the Hussein regime along with some of its success in deceiving the West.

At this point it is proper to ask: Why was the Duelfer report chosen as a refutation of the notion that Iraq had WMD during the lead-up to the invasion in 2003? Were the authors of the study victims of the phenomenon under investigation?

From Shankar again:

Bullock, Nyhan and Reifler are all Democrats.

So, if the findings suggested by Bullock, Nyhan and Reifler are legitimate then we have additional reason to regard their judgment of a proper refutation with suspicion.

The apparent solution is found in the combination of Shankar's misleading report and researcher bias. Nyhan and Reifler evidently did not use the Duelfer report as the refutation but a New York Times distillation of the Duelfer report. From the appendix:

Study 1 (WMD): News text
Wilkes-Barre, PA, October 7, 2004 (AP) -- President Bush delivered a hard-hitting speech here today that made his strategy for the remainder of the campaign crystal clear: a rousing, no-retreat defense of the Iraq war.

Bush maintained Wednesday that the war in Iraq was the right thing to do and that Iraq stood out as a place where terrorists might get weapons of mass destruction.

“There was a risk, a real risk, that Saddam Hussein would pass weapons or materials or information to terrorist networks, and in the world after September the 11th, that was a risk we could not afford to take,” Bush said.

[Correction]
While Bush was making campaign stops in Pennsylvania, the Central Intelligence Agency released a report that concludes that Saddam Hussein did not possess stockpiles of illicit weapons at the time of the U.S. invasion in March 2003, nor was any program to produce them under way at the time. The report, authored by Charles Duelfer, who advises the director of central intelligence on Iraqi weapons, says Saddam made a decision sometime in the 1990s to destroy known stockpiles of chemical weapons. Duelfer also said that inspectors destroyed the nuclear program sometime after 1991.

Something in the minds of Nyhan and Reifler is able to overlook the ambiguities in the Times' attempt to quash the notion that Hussein had WMD.

The question asked by the researchers hardly even has anything to do with the refutation:

Study 1 (WMD): Dependent variable
Immediately before the U.S. invasion, Iraq had an active weapons of mass destruction program, the ability to produce these weapons, and large stockpiles of WMD, but Saddam Hussein was able to hide or destroy these weapons right before U.S. forces arrived.
-Strongly disagree [1]
-Somewhat disagree [2]
-Neither agree nor disagree [3]
-Somewhat agree [4]
-Strongly agree [5]

Duh, if Hussein hid the weapons in Syria (for example) then "Saddam Hussein did not possess stockpiles of illicit weapons at the time of the U.S. invasion in March 2003, nor was any program to produce them under way at the time." The refutation needs to match the assertion in question or else the results mean little.
Note: The study included two different versions of the question, and the second, though still problematic, is less flawed than the one discussed above. Indeed, the researchers found strikingly different results with the second version, as I discovered after making my assessment of the material in the appendix:

Model 1 indicates that the WMD correction again fails to reduce overall misperceptions. However, we again add an interaction between the correction and ideology in Model 2 and find a statistically significant result. This time, however, the interaction term is negative – the opposite of the result from Study 1.

So when conservatives read a refutation that actually addresses the issue, the response does not reflect any increase in the supposedly false belief. My take, not necessarily that of the researchers, though they do touch on the possibility.

We should note the irony in Shankar offering the Duelfer report as a refutation of what turns out in the reseach paper to be an implication that it is known that Hussein did not hide WMD (for example) in Syria.


Tax Cuts Increase Revenue

Shankar's presentation of the next case, offered again by Nyhan and Reifler, borders on the bizarre:

A similar "backfire effect" also influenced conservatives told about Bush administration assertions that tax cuts increase federal revenue. One group was offered a refutation by prominent economists that included current and former Bush administration officials. About 35 percent of conservatives told about the Bush claim believed it; 67 percent of those provided with both assertion and refutation believed that tax cuts increase revenue.

It is no easy feat reconciling the above account with the details of the experiment:

[New York Times/FoxNews.com]
August 6, 2005
President George W. Bush urged Congress to make permanent the tax cuts enacted during his first term and draft legislation to bolster the Social Security program, after the lawmakers return from their August break.

“The tax relief stimulated economic vitality and growth and it has helped increase revenues to the Treasury,” Bush said in his weekly radio address. “The increased revenues and our spending restraint have led to good progress in reducing the federal deficit.”

The expanding economy is helping reduce the amount of money the U.S. government plans to borrow from July through September, the Treasury Department said on Wednesday. The government will borrow a net $59 billion in the current quarter, $44 billion less than it originally predicted, as a surge in tax revenue cut the forecast for the federal budget deficit.

The White House’s Office of Management and Budget last month forecast a $333 billion budget gap for the fiscal year that ends Sept. 30, down from a record $412 billion last year.

[Correction]
However, even with the recent increases, revenues in 2005 will remain well below previous projections from the Congressional Budget Office. The major tax cut of 2001 and further cuts in each of the last three years were followed by an unprecedented three-year decline in nominal tax revenues, from $2 trillion in 2000 to $1.8 trillion in 2003.
Last year, revenues rebounded slightly to $1.9 trillion. But at 16.3 percent of the gross domestic product, last year’s revenue total, measured against the size of the economy, was the lowest level since 1959.

The main problem should be immediately apparent upon reading the question posed to the test subjects:

President Bush’s tax cuts have increased government revenue.
-Strongly disagree [1]
-Somewhat disagree [2]
-Neither agree nor disagree [3]
-Somewhat agree [4]
-Strongly agree [5]

See the disconnect? Bush claims that his tax cuts "helped" increase government revenue. The test subjects are apparently supposed to conclude from that statement that the tax cuts increased revenue. The question alters the cause and effect relationship suggested by the president's words. That should account for the low number who agreed without reading the "correction." As the researchers noted, the subjects were probably more likely to think deeply about the issue after the correction, in part because the correction doesn't really address what Bush said in the first place. I've dealt with this tax issue at Bad Blogs' Blood, by the way.

Once conservatives start thinking about whether or not a tax cut increases revenue, they are likely to consider that a robust economy brings in more revenue than a weaker economy, all other things being equal. A tax policy that strengthens the economy is thus obviously increasing revenue, though the issue is complicated by the fact that higher tax rates also produce more revenue with everything else (including the strength of the economy) being equal. When both the tax rate and the strength of the economy vary, the comparison proves difficult. That simply spells trouble for researchers who provide an oversimplified picture to test subjects and subsequently try to draw conclusions from their findings.


Conclusion

Ideological bias makes it difficult for political scientists to construct valuable experiments.

Shankar's concluding paragraph had me wearing a wry smile:

Reifler questioned attempts to debunk rumors and misinformation on the campaign trail, especially among conservatives: "Sarah Palin says she was against the Bridge to Nowhere," he said, referring to the pork-barrel project Palin once supported before she reversed herself. "Sending those corrections to committed Republicans is not going to be effective, and they in fact may come to believe even more strongly that she was always against the Bridge to Nowhere."

At the risk of making you believe all the more that conservatives believe that Palin "always" opposed the Bridge to Nowhere, Mr. Reifler, check this out.