Does science suggest that fighting smears is futile?

A news story I picked up through Michelle Malkin's Hot Air has piqued my interest, given the amount of time I spend trying to sort through media disinformation for the truth.

The story, by Shankar Vedantam, concerned experiments conducted by political scientist John Bullock and other experiments by political scientists Brendan Nyhan and Jason Reifler. The experiments, according to Vedantam and the researchers, suggested that a smear successfully affected perception even after the smear was debunked. Further, the research by Nyhan and Reifler supposedly suggested that conservatives were more prone to accept a smear despite having access to information that falsified it.

But there are problems with this type of research, as I shall demonstrate.

I will deal with the experiments in the order Vedantam presents them.


Roberts and NARAL

The first concerned a smear of current Supreme Court Justice John Roberts produced by NARAL.

clipped from www.factcheck.org

NARAL Pro-Choice America TV Ad:
"Speaking Out"

Announcer: Seven years ago, a bomb destroyed a women's health clinic in Birmingham, Alabama.

(On screen: Footage of bombed clinic)

(Tex on screen: New Woman/All Women Health Clinic; January 28, 1998)

Emily Lyons: When a bomb ripped through my clinic, I almost lost my life.

Announcer: Supreme Court nominee John Roberts filed court briefs supporting violent fringe groups and a convicted clinic bomber.

(On screen: Footage of Roberts; image of April 11, 1991 brief from Bray v. Alexandria)

(Text on screen: Roberts filed court brief supporting clinic protesters)

Emily Lyons: I'm determined to stop this violence so I'm speaking out.

Announcer: Call your Senators. Tell them to oppose John Roberts. America can't afford a Justice whose ideology leads him to excuse violence against other Americans.

Picking up with Shankar's story:

Bullock then showed volunteers a refutation of the ad by abortion-rights supporters. He also told the volunteers that the advocacy group had withdrawn the ad. Although 56 percent of Democrats had originally disapproved of Roberts before hearing the misinformation, 80 percent of Democrats disapproved of the Supreme Court nominee afterward. Upon hearing the refutation, Democratic disapproval of Roberts dropped only to 72 percent.

On the face of it, this looks bad. The data appear to suggest that 16 percent of the Democrats in the study allowed a plainly refuted smear to color their opinions. But how solid is that conclusion?

I suggest that the conclusion only follows if the ad provides no damaging information that is not thoroughly and unambiguously countered by the refutation. That is a tall order, and one that seems well outside the reach of Bullock's experiment. Shankar's version of the experiment fails to do it justice:

Near the end of the experiment, treatment-group subjects received this information:

Recall the ad transcript you read about earlier in this survey. The ad was strongly criticized by many people, some of whom were prominent supporters of abortion rights.
Walter Dellinger, an ally of the group that aired the ad and an important attorney in the Clinton administration, called the ad “unfair and unwarranted.” He added that “It is unfair to suggest that John Roberts, in advancing a somewhat narrow interpretation of [the anti-discrimination law], was supporting ‘violent fringe groups and a convicted clinic bomber’—as unfair as it would be to suggest that the six Justices who were part of the majority in Bray joined a decision supporting violent fringe groups.”
Arlen Specter, a Republican senator and supporter of abortion rights, called the ad “blatantly untrue and unfair.”
Stung by these criticisms and many others, NARAL Pro-Choice America withdrew the ad from television.

(Bullock, "The Enduring Importance of False Political Beliefs," p. 39)

The supposed refutation itself contains information that might influence the test subjects to change their opinion of Roberts for the worse, given that he supported a "narrow" interpretation of an anti-discrimination law. That is aside from other obvious factors, such as a general familiarity with NARAL, which might influence test subjects to think that even if the ad was false, the organization would not have aired it if Roberts presented no worries for a pro-choice voter.

This phase of the experiment does not significantly support the notion that a refuted smear continues to affect attitudes.


Guantanamo, Koran, Toilet

The second smear involved the false reports of Guantanamo guards flushing a copy of the Koran down the toilet.

Volunteers were shown a Newsweek report that suggested a Koran had been flushed down a toilet, followed by a retraction by the magazine. Where 56 percent of Democrats had disapproved of detainee treatment before they were misinformed about the Koran incident, 78 percent disapproved afterward. Upon hearing the refutation, Democratic disapproval dropped back only to 68 percent -- showing that misinformation continued to affect the attitudes of Democrats even after they knew the information was false.

The construction of this experiment was more complicated than the other, and it seems that Shankar misinterpreted the results. No baseline attitude toward detainee treatment occurred prior to exposure to the texts used in the experiment. The 56 percent of Democrats who disapproved of detainee treatment were part of a control group who read an account of detainee mistreatment with the Koran-flushing episode excised. The 78 percent figure Shankar presents as disapproval in the (Democratic portion of the) control group appears to be a figure actually related to approval for congressional hearings on detainee treatment. I located no pre-refutation figure for the treatment group, which leads me to believe that the control group was intended to provide the baseline (contrary to the picture painted by Shankar). The refutation would have been meaningless and irrelevant to the control group, since Newsweek would be retracting a part of the story they had not read.

Apart from Shankar's difficulty in reporting the experiment, did it support Bullock's thesis? Since 68 percent of the treatment group disapproved detainee treatment even after the refutation compared to 56 percent in the control group, perhaps so. The key variable is the text of the Newsweek retraction, which was not included in Bullock's paper.

If the retraction was similar to the following, however, there is good reason to doubt:

Last Friday, a top Pentagon spokesman told us that a review of the probe cited in our story showed that it was never meant to look into charges of Qur’an desecration. The spokesman also said the Pentagon had investigated other desecration charges by detainees and found them “not credible.” Our original source later said he couldn’t be certain about reading of the alleged Qur’an incident in the report we cited, and said it might have been in other investigative documents or drafts. Top administration officials have promised to continue looking into the charges, and so will we. But we regret that we got any part of our story wrong, and extend our sympathies to victims of the violence and to the U.S. soldiers caught in its midst.
(Outside the Beltway, quoting from Mark Whitaker's "From the Editor's Desk")

A "retraction" like that above tends to feed into the level of trust one feels for the media source as opposed to the government source, which provides a natural and potentially reasonable alternate explanation for the belief perseverance effect Bullock was exploring.

Hussein and WMD

The next example in Shankar's story comes from Nyhan and Reiffler.

Political scientists Brendan Nyhan and Jason Reifler provided two groups of volunteers with the Bush administration's prewar claims that Iraq had weapons of mass destruction. One group was given a refutation -- the comprehensive 2004 Duelfer report that concluded that Iraq did not have weapons of mass destruction before the United States invaded in 2003. Thirty-four percent of conservatives told only about the Bush administration's claims thought Iraq had hidden or destroyed its weapons before the U.S. invasion, but 64 percent of conservatives who heard both claim and refutation thought that Iraq really did have the weapons. The refutation, in other words, made the misinformation worse.

I don't imagine it was an easy matter to find volunteers willing to read the Duelfer report (not the work of mere minutes).

While the future size and direction of the Iraq Survey Group are currently under review, the requirement remains to collect further information related to threats posed by residual elements ofthe former Regime’s WMD programs. There will also be new information from individuals and sources, which will come to light. Moreover, certain defined questions remain unanswered. For example, we cannot express a firm view on the possibility that WMD elements were relocated out of Iraq prior to the war. Reports of such actions exist, but we have not yet been able to investigate this possibility thoroughly.

It should be obvious merely from the above excerpt (and there are many similar caveats in the report) that the Duelfer report did not unequivocally refute the notion that Iraq possessed WMD. Indeed, though the report emphasized the lack of solid evidence supporting the notion, it also emphasized the devious intent of the Hussein regime along with some of its success in deceiving the West.

At this point it is proper to ask: Why was the Duelfer report chosen as a refutation of the notion that Iraq had WMD during the lead-up to the invasion in 2003? Were the authors of the study victims of the phenomenon under investigation?

From Shankar again:

Bullock, Nyhan and Reifler are all Democrats.

So, if the findings suggested by Bullock, Nyhan and Reifler are legitimate then we have additional reason to regard their judgment of a proper refutation with suspicion.

The apparent solution is found in the combination of Shankar's misleading report and researcher bias. Nyhan and Reifler evidently did not use the Duelfer report as the refutation but a New York Times distillation of the Duelfer report. From the appendix:

Study 1 (WMD): News text
Wilkes-Barre, PA, October 7, 2004 (AP) -- President Bush delivered a hard-hitting speech here today that made his strategy for the remainder of the campaign crystal clear: a rousing, no-retreat defense of the Iraq war.

Bush maintained Wednesday that the war in Iraq was the right thing to do and that Iraq stood out as a place where terrorists might get weapons of mass destruction.

“There was a risk, a real risk, that Saddam Hussein would pass weapons or materials or information to terrorist networks, and in the world after September the 11th, that was a risk we could not afford to take,” Bush said.

[Correction]
While Bush was making campaign stops in Pennsylvania, the Central Intelligence Agency released a report that concludes that Saddam Hussein did not possess stockpiles of illicit weapons at the time of the U.S. invasion in March 2003, nor was any program to produce them under way at the time. The report, authored by Charles Duelfer, who advises the director of central intelligence on Iraqi weapons, says Saddam made a decision sometime in the 1990s to destroy known stockpiles of chemical weapons. Duelfer also said that inspectors destroyed the nuclear program sometime after 1991.

Something in the minds of Nyhan and Reifler is able to overlook the ambiguities in the Times' attempt to quash the notion that Hussein had WMD.

The question asked by the researchers hardly even has anything to do with the refutation:

Study 1 (WMD): Dependent variable
Immediately before the U.S. invasion, Iraq had an active weapons of mass destruction program, the ability to produce these weapons, and large stockpiles of WMD, but Saddam Hussein was able to hide or destroy these weapons right before U.S. forces arrived.
-Strongly disagree [1]
-Somewhat disagree [2]
-Neither agree nor disagree [3]
-Somewhat agree [4]
-Strongly agree [5]

Duh, if Hussein hid the weapons in Syria (for example) then "Saddam Hussein did not possess stockpiles of illicit weapons at the time of the U.S. invasion in March 2003, nor was any program to produce them under way at the time." The refutation needs to match the assertion in question or else the results mean little.
Note: The study included two different versions of the question, and the second, though still problematic, is less flawed than the one discussed above. Indeed, the researchers found strikingly different results with the second version, as I discovered after making my assessment of the material in the appendix:

Model 1 indicates that the WMD correction again fails to reduce overall misperceptions. However, we again add an interaction between the correction and ideology in Model 2 and find a statistically significant result. This time, however, the interaction term is negative – the opposite of the result from Study 1.

So when conservatives read a refutation that actually addresses the issue, the response does not reflect any increase in the supposedly false belief. My take, not necessarily that of the researchers, though they do touch on the possibility.

We should note the irony in Shankar offering the Duelfer report as a refutation of what turns out in the reseach paper to be an implication that it is known that Hussein did not hide WMD (for example) in Syria.


Tax Cuts Increase Revenue

Shankar's presentation of the next case, offered again by Nyhan and Reifler, borders on the bizarre:

A similar "backfire effect" also influenced conservatives told about Bush administration assertions that tax cuts increase federal revenue. One group was offered a refutation by prominent economists that included current and former Bush administration officials. About 35 percent of conservatives told about the Bush claim believed it; 67 percent of those provided with both assertion and refutation believed that tax cuts increase revenue.

It is no easy feat reconciling the above account with the details of the experiment:

[New York Times/FoxNews.com]
August 6, 2005
President George W. Bush urged Congress to make permanent the tax cuts enacted during his first term and draft legislation to bolster the Social Security program, after the lawmakers return from their August break.

“The tax relief stimulated economic vitality and growth and it has helped increase revenues to the Treasury,” Bush said in his weekly radio address. “The increased revenues and our spending restraint have led to good progress in reducing the federal deficit.”

The expanding economy is helping reduce the amount of money the U.S. government plans to borrow from July through September, the Treasury Department said on Wednesday. The government will borrow a net $59 billion in the current quarter, $44 billion less than it originally predicted, as a surge in tax revenue cut the forecast for the federal budget deficit.

The White House’s Office of Management and Budget last month forecast a $333 billion budget gap for the fiscal year that ends Sept. 30, down from a record $412 billion last year.

[Correction]
However, even with the recent increases, revenues in 2005 will remain well below previous projections from the Congressional Budget Office. The major tax cut of 2001 and further cuts in each of the last three years were followed by an unprecedented three-year decline in nominal tax revenues, from $2 trillion in 2000 to $1.8 trillion in 2003.
Last year, revenues rebounded slightly to $1.9 trillion. But at 16.3 percent of the gross domestic product, last year’s revenue total, measured against the size of the economy, was the lowest level since 1959.

The main problem should be immediately apparent upon reading the question posed to the test subjects:

President Bush’s tax cuts have increased government revenue.
-Strongly disagree [1]
-Somewhat disagree [2]
-Neither agree nor disagree [3]
-Somewhat agree [4]
-Strongly agree [5]

See the disconnect? Bush claims that his tax cuts "helped" increase government revenue. The test subjects are apparently supposed to conclude from that statement that the tax cuts increased revenue. The question alters the cause and effect relationship suggested by the president's words. That should account for the low number who agreed without reading the "correction." As the researchers noted, the subjects were probably more likely to think deeply about the issue after the correction, in part because the correction doesn't really address what Bush said in the first place. I've dealt with this tax issue at Bad Blogs' Blood, by the way.

Once conservatives start thinking about whether or not a tax cut increases revenue, they are likely to consider that a robust economy brings in more revenue than a weaker economy, all other things being equal. A tax policy that strengthens the economy is thus obviously increasing revenue, though the issue is complicated by the fact that higher tax rates also produce more revenue with everything else (including the strength of the economy) being equal. When both the tax rate and the strength of the economy vary, the comparison proves difficult. That simply spells trouble for researchers who provide an oversimplified picture to test subjects and subsequently try to draw conclusions from their findings.


Conclusion

Ideological bias makes it difficult for political scientists to construct valuable experiments.

Shankar's concluding paragraph had me wearing a wry smile:

Reifler questioned attempts to debunk rumors and misinformation on the campaign trail, especially among conservatives: "Sarah Palin says she was against the Bridge to Nowhere," he said, referring to the pork-barrel project Palin once supported before she reversed herself. "Sending those corrections to committed Republicans is not going to be effective, and they in fact may come to believe even more strongly that she was always against the Bridge to Nowhere."

At the risk of making you believe all the more that conservatives believe that Palin "always" opposed the Bridge to Nowhere, Mr. Reifler, check this out.