Tuesday, June 21, 2011

The misperceptions misperception mushrooms (Updated x2)

PolitiFact, it seems, has mired itself in the misperceptions marsh.

The problem started when the fact checkers went to test a recent claim made by comedian/Daily Show host Jon Stewart during an interview with Fox News' Chris Wallace (yellow highlights added):
WALLACE: I don't think our viewers are the least bit disappointed with us. I think our viewers think, finally, they're getting somebody who tells the other side of the story.
STEWART: Right.
(CROSSTALK)
WALLACE: And in -- no, no, no. One more example.
STEWART: Who are the most consistently misinformed media viewers? The most consistently misinformed? Fox, Fox viewers, consistently, every poll.
WALLACE: Can we talk about your network? Can we talk about Comedy Central?
PolitiFact's story ended up rating Stewart's statement "False," noting that the bulk of the available polling showed that Fox viewers were better informed than those who follow some other news sources.

Then things got interesting.
PolitiFact's audience and assorted liberals erupted in protest, resulting in one of those generally unsatisfying updates:
When we gave a False rating to a statement by Daily Show host Jon Stewart, the response from readers was swift and virtually unanimous. They said we were wrong.
Having claimed that PolitiFact will play to its primarily liberal audience--note the "virtually unanimous" response hostile to Fox News--I started following the aftermath of the story with interest.  In its initial followup story PolitiFact offered a somewhat unusual concession to its detractors:
Ultimately, we ruled that for Stewart’s comment to qualify as accurate, the data had to be "consistent" -- a term he used not once but three times. Since not "every poll" showed that result, we rated his claim False.

The striking thing about the comments we received -- almost 100 e-mails plus countless tweets and Facebook postings during the first 24 hours -- is how quickly readers zeroed in on a distinction our piece did not draw. They said we erred by failing to distinguish between a viewer being "misinformed" and a viewer beinig (sic) "uninformed" or "ill-informed."
Typical of a PolitiFact response, the update contains no real attempt to justify the rating against the criticism.  Given the dismay of PolitiFact's political base (liberals), however, I would not register much surprise if PolitiFact updated the item to upgrade the rating.

But PolitiFact probably should leave the rating as it is even though the criticism contains a shred of validity.

I should explain.

The adjective "misinformed" could apply in at least two distinct ways.  First, it could mean that Fox Viewers hold particular incorrect views because they were given false information (apparently by Fox News).  Alternatively, the term can serve as a synonym for "ignorant," as in simply believing untrue things.

Stewart's would-be defenders, in an exercise of futility, try to parse the term in his favor.  PolitiFact's secondary story provided an example from Jane Hamsher (FireDogLake):
The three Pew polls measure how informed viewers are.  They don’t even belong in the discussion, because they don’t go to Stewart’s point.   (...)

On the other hand, the two PIPA studies measure how misinformed viewers are. That’s a very different yardstick.  Listening to Dana Bash may freeze a few neurons in the “off” position, and I may not get the news value that I should out of the segment, but unless she says something that is manifestly untrue I can’t claim to have been misinformed.
Hamsher somewhat botched the parsing job by confusing the adjective form of "misinformed" with the verb form (a person can, after all, qualify as ignorant even if never misinformed about anything), but we can at least see that she favors taking Stewart to mean that Fox News imparted to its viewers false information, a possible interpretation I just covered.  But her interpretation can't bail out Stewart.  Neither meaning can rehabilitate Stewart's statement.

If we take Stewart to mean that polls consistently show Fox News viewers the most ignorant (misinformed) then we're stuck with PolitiFact's analysis using the PEW polling data.  That's exactly what the complaining liberals try to avoid with the alternate interpretation.

The alternate interpretation fails because it leaves Stewart without any supporting data.

As Hamsher notes, Stewart ends up relying on two studies by the Program on International Policy Attitudes.  Both PIPA's 2003 study and its 2010 study suffer from massive problems, such as a failure to control for researcher bias in choosing which points of knowledge to count as misperceptions.  But Hamsher's attempt to help out Stewart fails because neither study makes a scientific attempt to discover the source of the supposed misperceptions.  They simply asked a set of questions, some dealing with the level of belief in a series of statements and others asking respondents about their sources for news (2003, 2010).

The design of the PIPA studies precludes them from scientifically supporting the claim that Fox News viewers received their mistaken beliefs from Fox News reporting.

That leaves Stewart with no defense.  But I'll be interested to see if the liberal outcry convinces PolitiFact otherwise.


Update:

Near the time I first published this post, Jon Stewart's response to PolitiFact hit the Web.



Comedian, sore loser or both?

Sure, Jon Stewart's primarily a comedian.  But his act tonight appears indistinguishable from sore loser behavior as he tries to lessen the severity of his error by presenting even more inaccuracies.

Stewart presents a set of supposed PolitiFact fact checks of Fox News that turned out false or thereabouts.  But he simply takes claims made on the Fox News channel (or presumably so) and attributes them to Fox News.  Take, for example, PolitiFact's "Lie of the Year" for 2009--Sarah Palin's "death panel" statement.  The statement was made on Palin's FaceBook page.  And PolitiFact's "Lie of the Year" story attributed the statement solely to Palin.

Overlooking the fact that PolitiFact blew the "Lie of the Year" analysis (as well as for its 2010 pick), Stewart is covering for himself with lies.

Classy.


Update 2

Quite a few on the Web have tried to go beyond Jane Hamsher's attempt to find substantiation for Stewart's claim, with Chris Mooney's list of studies often presented in support.  I've posted a rebuttal to that use of Mooney's work here.

25 comments:

  1. You are right; The "death panels" comment was made by Sarah Palin on Facebook, and Politifact.org's original and Lie of the Year ruling focuses on it solely in that context. However, the assertion was repeated on Fox News, at the very least, once:
    http://video.foxnews.com/v/4475784/return-of-death-panels
    Based on this one video I found on FoxNews.com, when someone was filling in on the O'Reilly Factor (I don't recognize him and couldn't catch his name), it sounded like "death panels" were discussed ad nauseam prior to this segment being recorded. And to be clear, the subject was not being reported on as something other people were discussing, the fill-in host was making the same phony claim that Palin made.

    ReplyDelete
  2. (second try, with a bit more attention to grammar & stuff)

    Let's say that Fox repeated the claim 7,000 times. Does that justify Stewart proclaiming Fox News as the New England Patriots of lying because the channel won "Lie of the Year" two consecutive times?

    I stick with my take. His reaction was childish and vindictive.

    ReplyDelete
  3. of course it was "childish and vindictive" hes a comedian. one that bases his humor on satire and cynicism. thats why its entertainment... surely you know you cannot watch COMEDY CENTRAL and expect a 100% unbiased media outlet... that is just absurd. so what if he compares fox news to the patriots? it gets him a laugh and has merit in that the pats won the super bowl several times recently, and fox news won the lie of the year awards twice in a row. thats what an analogy is...

    new england patriots are to fox news claims
    as is
    super bowl to lie of the year

    you see, they correlate in that the pats won the super bowl, and fox news won lie of the year. that is all hes saying...

    take an english class, and when this question appears youll know then how incorrect your assumption is.
    jon stewart is to comedian
    as
    satire is to vindictive
    (-1) correct answer: harmless

    ReplyDelete
  4. swaff wrote:

    "you see, they correlate in that the pats won the super bowl, and fox news won lie of the year. that is all hes saying..."

    You appear to miss what I am saying. It isn't true that Fox News won PolitiFact's "Lie of the Year." Not twice. Not once.

    Call it comedy if you wish. Just don't call it true.

    ReplyDelete
  5. It just seems to me like you're trying way to hard to expose a bias that just isn't there.

    I agree with the community on this one, Politifact was quick to call Stewart wrong, Stewart definitely said misinformed, not uninformed. The entire debate wasn't stewart attempting to call fox viewers stupid, he was trying to say that fox news intentionally distorts information with a conservative bias.

    In context, it's pretty obvious that he means that viewers have been misinformed, rather than just ignorant.

    This is straight from politifact, " Pew asked respondents 23 questions, such as... who the president of Russia is"

    I doubt fox news has extensively covered topics like who the president of Russia is, and if they have i retract this statement, but these seem FAR more like general knowledge questions than something you would be learning from fox news.

    This is where politifact is wrong.


    But as for you, i'll tell you that as a reader i am completely unconvinced that the PIPA polls are as bad as you say. Why? because you've given me absolutely nothing but your word on it, maybe you could provide a few questions you don't like next time?

    Not to mention, in your attempt to debunk Stewart's response, you only gave one example of something that was wrong, and it wasn't even really wrong, as fox news repeated what sarah palin said (as colonel pointed out) So even though the politifact debunking was technically about sarah palin, fox news said it too.

    ReplyDelete
  6. Twurzz wrote:

    But as for you, i'll tell you that as a reader i am completely unconvinced that the PIPA polls are as bad as you say. Why? because you've given me absolutely nothing but your word on it, maybe you could provide a few questions you don't like next time?

    I hotlinked both PIPA surveys (the questionnaires, not the summaries) so that anyone with your type of curiosity could easily see for themselves. The material is not dauntingly long. Examples:

    Q3: Is it your impression that Iraq did or did not use chemical or biological weapons in
    the war that just ended?

    Q4: Please select what you think is the best description of the relationship between the
    Iraqi government under Saddam Hussein and the terrorist group al-Qaeda.

    Q5: Since the war with Iraq ended, is it your impression that the US has or has not found
    Iraqi weapons of mass destruction?

    Q6: Is it your impression that the US has or has not found clear evidence in Iraq that
    Saddam Hussein was working closely with the Al Qaeda terrorist organization?

    Q7: Thinking about how all the people in the world feel about the US having gone to war
    with Iraq, do you think:

    Q8: Thinking about how people in Europe feel about the US having gone to war with
    Iraq, do you think:


    Heh. I forgot about Q15. PIPA had its own misperception:

    Q15. As you may know, the Bush administration has said that Iraq played an important role in the September 11th attacks. Would you say that you:

    The Bush administration did not at all sell the idea that Iraq was linked to the 9-11 attacks. Rather, it sold the idea (which was substantiated by intelligence) that Iraq had contacts with the organization responsible for those attacks.

    Horse. Water. Drink.

    ReplyDelete
  7. Twurzz went on:

    in your attempt to debunk Stewart's response, you only gave one example of something that was wrong, and it wasn't even really wrong, as fox news repeated what sarah palin said (as colonel pointed out)

    It's pretty awesome that Fox News gets to claim Sarah Palin's award simply for repeating what she said. Maybe I'll repeat a few lies, put a few facsimiles up on my mantle and claim that I'm the New England Patriots of lying. Take that, Jon Stewart.

    ReplyDelete
  8. Bottom line, if I do a thorough debunking of the PIPA studies the blog post is so long that nobody's going to read it. Instead I describe the problem and provide the path so the reader can confirm what I'm saying. If your mind's already made up then you won't bother checking it out. If your mind isn't made up then you ought to question what you heard from PolitiFact. If you're really curious then you'll confirm what I wrote and then see that I'm correct when you look at the linked material.

    ReplyDelete
  9. Bryan, what is wrong with those pipa questions? To me at least it seems like information that people would be getting from the news...

    And with the sarah palin/fox news thing, If something is untrue when person 1 says it, isn't it equally untrue if person 2 says it?

    ReplyDelete
  10. The entire point of jon going on fox news was to say that fox is misinforming people.

    If Palin says something wrong, and fox news repeats it to their viewers, then they're misinforming people...

    ReplyDelete
  11. Twurzz asked:
    Bryan, what is wrong with those pipa questions?

    1) They do not serve as a representative sample of political knowledge (what Stewart needs to support his claim).
    2) In fact, the PIPA questions pretty clearly represent things liberals would think conservatives are misinformed about. The questions might as well have been designed to reach the desired conclusion.

    Twurzz also wrote:
    The entire point of jon going on fox news was to say that fox is misinforming people.

    None of the studies gets to the source of the supposed misinformation, so Stewart receives no support from any of them as to that point. So Stewart's still wrong. It may be that Fox News viewers are misinformed more than any other news audience but no study we have serves to reasonably demonstrate that proposition.

    ReplyDelete
  12. What questions would you like to have included that you think would've made the poll more representative?

    I can't think of much else that people would be misinformed about, conservative or liberal, the studies point to the conclusion that fox news viewers are misinformed, you just choose to call the studies biased (with no evidence to support YOUR claim other than your personal opinion and assumptions you have made)


    1) They do not serve as a representative sample of political knowledge (what Stewart needs to support his claim). [b]- Opinion with nothing to back it up[/b]
    2) In fact, the PIPA questions pretty clearly represent things liberals would think conservatives are misinformed about. The questions might as well have been designed to reach the desired conclusion. [b]- assumption[/b]

    ReplyDelete
  13. 1) It is obvious that questions concentrated on the Iraq War (and only three central aspects of the Iraq War) are not representative political knowledge questions. Think about it.

    2) It is not an assumption. It is the result of public debate experience with liberals during the time leading to the war where they continually complained that there was no proof of WMD, the world doesn't want war, and Hussein had nothing to do with the 9-11 attacks.

    Be patient. I'm working on my own critique of the 2003 PIPA study, and if I don't find a strong enough one already finished for the 2010 one I'll critique that one also.

    Don't assume that the statements of others are assumptions, by the way.

    ReplyDelete
  14. Q?: Thinking about how Saddam Hussein treats the various factions in Iraq, do you think:
    1) Hussein treats the Shiites and Kurds well.
    2) Hussein treats the Shiites well and oppresses the Kurds
    3) Hussein oppresses the Kurds and the Shiites.

    ReplyDelete
  15. "1) It is obvious that questions concentrated on the Iraq War (and only three central aspects of the Iraq War) are not representative political knowledge questions. Think about it."

    You realize there were two polls you linked, right?

    "2) It is not an assumption. It is the result of public debate experience with liberals during the time leading to the war where they continually complained that there was no proof of WMD, the world doesn't want war, and Hussein had nothing to do with the 9-11 attacks." - Then its a generalization?

    ReplyDelete
  16. By the way, i would say that the 2010 poll is probably a lot more relevant, considering how much more recent it is, that the questions are more general, and that Jon wasn't attacking fox nearly as much in 2003 as he is now.

    ReplyDelete
  17. Trwurzz wrote:
    You realize there were two polls you linked, right?

    Yeah, and you realize that my criticism, if acknowledged, puts you down to one, and that being the one that PolitiFact admitted had received a substantial degree of methodological criticism.

    By the way, i would say that the 2010 poll is probably a lot more relevant ...

    That's unfortunate for you, then, because that poll has received a good amount of devastating criticism already.

    ReplyDelete
  18. Twurzz asked:

    Then its a generalization?

    Keep guessing. I'll let you know if you're warm.

    ReplyDelete
  19. Jon said that the polls show fox news viewers are misinformed, The polls looking for misinformed viewers support Jon Stewart's claim.

    This doesn't change just because you don't like the polls.

    You're turning this into a red herring argument.

    ReplyDelete
  20. Stewart said that Fox News viewers were misinformed about what in particular?

    Pick one and then stick with it. I won't be allowing you to equivocate.

    ReplyDelete
  21. He said misinformed... through intense critical thinking, i assume that they're misinformed about topics that someone would learn from the news?

    Like the healthcare bill, economy, wars in the middleast, etc.

    ReplyDelete
  22. If you choose that referrant for "misinformed" then on what basis do you dismiss the polls PolitiFact cited other than the PIPA set?

    Haven't you scuttled your argument at the outset?

    ReplyDelete
  23. from politifact, about on of the polls:

    ( questions ) "such as who the vice president is, who the president of Russia is, whether the Chief Justice is conservative, which party controls the U.S. House of Representatives and whether the U.S. has a trade deficit."

    These aren't really things that you would be informed about from watching the news, or they wouldn't be covered on the news often. - hence being uninformed instead of misinformed, these aren't questions that people would really give the wrong answer to, they would just give no answer at all.

    The exception of course being who is the vice president, which is obviously pretty much common knowledge, most people that have never turned on a tv will know who the vice president is.

    ReplyDelete
  24. I covered that in my first post...

    ReplyDelete
  25. Twurzz (sorry for being slow to approve your comment before this last one--somehow it escaped my notice),

    People do give wrong answers based on what you call being "uninformed." Misinformed and uninformed are essentially synonymous, though "misinformed" can also mean (via its association with the verb form) given incorrect information. The problem with taking that route is that *none* of the cited studies show the origin of the misinformation. So they don't show Fox misinforming anybody about anything. So Stewart's still wrong by that measure.

    I trust you'll play deal-a-definition and try a different way of parsing the meaning?

    ReplyDelete

Please remain on topic and keep coarse language to an absolute minimum. Comments in a language other than English will be assumed off topic.