Grading PolitiFact: Sen. DeMint and unanimous consent (Updated)

This is bad.  Just a warning.

Update:  PolitiFact assigned a new reporter to the story and produced a different version.  See my separate review of the second attempt here.

The issue(s):

The image immediately below occurs on a page listing the most recent of PolitiFact's national stories.  The caption below the flaming "Truth-O-Meter" graphic will turn out ironically prophetic.

The second image (below) repeats the main fact-check claim and comes from the page that includes the fact check story.  The deck below the headline material features a somewhat different claim, that 94 percent of bills are passed unanimously.  That is, a particular procedure called "unanimous consent" is used to pass bills in the Senate at the 94 percent rate.

The second image features the flaming "Truth-O-Meter" image nearby, so the reader may infer that both claims mentioned in the image are ridiculous claims according to PolitiFact standards.

The fact checkers:

Lukas Pleva:  writer, researcher
Bill Adair:  editor

This is my first review of Lukas Pleva's work.  The PolitiFact site identifies him as an intern from the University of Chicago.


Analysis:

Pleva most likely pleased the rest of the PolitiFact staff no end with the cheesy opening paragraph.  Such openings occur often in PolitiFact stories and are meant to convey the bold attitude of their fact checks.  That type of communication has no place in objective journalism, yet PolitiFact continues to present its stories, internal content aside, as objective journalism.  I won't count that against Pleva.  Most likely they made him do it.

On to the content:

On May 26, 2010, Sen. Jim DeMint, a Republican from South Carolina, posted a video of one of his speeches on YouTube. Speaking about legislative openness, DeMint said that Congress is plagued with secrecy and lack of debate on important issues: "At this point in the Senate, 94 percent of all the bills are passed unanimous consent --  94 percent. So this is hardly a lack of productivity. And what this means is 94 percent of the bills that pass the Senate have no debate, no vote, no amendments, no reading of the bill, no online disclosure, and very often no score from the Congressional Budget office."

DeMint's number seemed suspiciously high, so we decided to investigate.

Ignoring the first paragraph, the fact check is great through this point.  It communicates Sen. DeMint's message with reasonable accuracy and offers a full quotation of DeMint with no ellipses.  The fourth graph sustains that standard by recounting how PolitiFact contacted DeMint for the source of his claim.

That's where the trouble starts:

We found (DeMint) ignored key details in the report.

To begin, of the 855 measures, only 327 were legally binding bills and joint resolutions, according to the CRS report. The rest were simple and concurrent resolutions which do not have the force of a law and thus do not require the President's signature. An example is a resolution "congratulating Charles County, Maryland, on the occasion of its 350th anniversary." We wouldn't expect any debate on that one.

As we wonder why the above constitutes a key detail, Pleva goes off the rails:

That leaves 327 binding measures. Of those, however, the CRS report says 64 were actually passed with debate. We're down to 263. Of those, nine had similar versions in the House that were debated. That leaves 254. Finally, CRS notes that 42 were passed unanimously the same day that they were received, which means that some discussion may have happened, but it would have been very limited. But we'll be generous and let DeMint count the 42 anyway. Still, only 254 bills were passed by the Senate with no debate and no amendments. That's 27.9 percent -- significantly less than the purported 94.
(colorful highlights added)

Pink:  We are left with 327 binding measures.  On what basis did we just discard the bulk of the measures?  Because they are not "bills"?  Because the president doesn't need to sign them?  What?  The best way I can take Pleva's approach is to assume that he feels that if certain bills were not very important then it is irrelevant whether they lacked debate, a vote, reading, online disclosure or ("often") no CRS scoring.  They just don't count.

In short, it looks like Pleva invented a criterion for not considering the bulk of the measures as meeting the standard set by DeMint.

Less charitably, I can assume that Pleva decided that simple or concurrent resolutions do not count as bills despite the fact that each is assigned a bill number.

Orange:  The CRS reports says that 64 passed with debate?  Oh, really?  What kind of debate?

From the report:

If any Senator made a statement at the time the measure was approved, even if it was an extremely brief explanation of the legislation, then the measure was listed in Attachment 5. Inclusion in this list is not necessarily an indication of lengthy debate, and in fact it is likely that some of these measures were cleared through the telephone hotline. Attachment 5 lists 77 measures as having passed by unanimous consent with at least some discussion.

In Pleva's account, measures from Attachment 5 were measures "passed with debate."  PolitiFact readers apparently do not deserve the rest of the story.  That supposedly debated measures may not have received any significant debate does not count as a significant detail in the PolitiFact version of the story.

Green:  Pleva claims we're down to 263 bills that might meet Sen. DeMint's 94 percent criteria, despite the fact that some of them may not have received any significant debate.  This miserly approach ends up with a counterbalance when Pleva grants DeMint the benefit of the doubt on unanimous consent bills passed the same day they were introduced.  Is this a precise version of fact checking?  No, of course not.  Pleva ends up helping himself to the type of imprecise communication that PolitiFact often criticizes when it comes from others.

Yellow:  This is the whopper of whoppers.  The claim that only 254 bills passed the Senate via unanimous consent forces us to discard the more charitable of the two interpretations I suggested for Pleva's claim.  If Pleva had proposed that only 254 significant bills had passed the Senate via unanimous consent then the statement would have been consistent with the notion that he was only counting significant bills, even if the standard was apparently arbitrary.  Rather, Pleva does appear to be making the ridiculous claim that most numbered bills listed in the report do not count as bills but are instead merely "measures" and thus cannot count toward DeMint's 94 percent figure.

Blue:  OK, we have a bit of a contest for whopper of whoppers.  Pleva takes the 254 bill figure that he reached chiefly by ruling out mere "measures" and uses that figure to derive a percentage of 27.9 percent (254 divided by 911).  But if measures do not count as bills, then Pleva has no justification for using 911 as the denominator in his equation.  The percentage of bills that received no debate--otherwise sticking with Pleva's method--should be calculated as 254 divided by 376 (911-535), or 67.6 percent.

A pair of commenters at PolitiFact's FaceBook page caught the mistake before I looked into this item:

Not that I agree with his assesent at all but in the interest of fairness which this site helps assure (and maybe I'm not following the logic), if we are looking at the percentage of binding bills passed with no debate, wouldn't that be 254 bills out of 327, which would be 71%. Still makes the statement wrong but not AS pants on fire as suggested. PLEASE, correct me if I'm not using the right numbers.
Thursday at 6:33pm
-- Ruth Robertson-Gouge

@Ruth, you are close to be(ing) correct in your statement. There were another 66 bills (911-855) that were presumably debated as well - i.e., the 6% that DeMint claimed. If these were also binding legislation (almost certainly), then you'd have a relevant population of 327+66= 393. So, 254 / 393 = 64.6%. Thus, while DeMint is still "Liar, Pants on Fire!" it is not by nearly as much as the Truth-o-Meter incorrectly calculated.

Thursday at 6:54pm

--Scott Tippetts

Ed. note: Tippets' percentage differs from mine because he used a 66 bill figure where he should have used 49 (911-855-# of non-binding measures where unanimous consent was not used for passage)

Pleva flubbed the math and his editor, incredibly, failed to catch it.

Pause:  Note that we have no reasonable justification for changing the meaning of the term "bill" from the way DeMint used it.  As a result, DeMint's claim from the PolitiFact deck that 94 percent of bills are passed unanimously (via unanimous consent) is unambiguously true, at least for the year cited, regardless of the animated "Pants on Fire" graphic nearby.  That leaves us dealing with DeMint's various claims concerning the unanimous consent process.

• "no debate"
• "no vote"
• "no amendments"
• "no reading of the bill"
• "no online disclosure"
• "very often no score from the Congressional Budget Office"
• "They don't have any regulation"

Oops, sorry about that last one.  That was from Chris Matthews, talking about the lack of regulation of oil companies.  PolitiFact did not check that statement by Matthews.  I included it to illustrate real-life communication.  Most statements in normal human communication are not intended as absolute statements.  When Matthews claimed that oil companies don't have any regulation, he almost certainly did not mean it literally.  So what about Sen. DeMint?  Were his statements intended as absolutes?

The best method to use in deciding that question involves the context of any given statement:



View the Congressional Record version of DeMint's comments here.

DeMint's primary audience that day consisted of his colleagues in the Senate.  Senators will very likely understand the operation of unanimous consent.  For that reason, DeMint should receive the benefit of the doubt in assessing whether he intended his remarks as absolute.

Back to Pleva:

What is even more misleading, however, is Sen. DeMint's implicit suggestion that passing legislation unanimously is some form of a new and deceptive tactic in the Democratic-run Congress.

I failed to find any implicit suggestion in DeMint's remarks to the effect that Democrats developed the technique as a new and deceptive tactic.  I took DeMint to refer to the technique as a problem with the legislative process overall, and one that ought to receive closer attention than the so-called "secret holds" that have recently come under more intense rhetorical fire.

Pleva doesn't bother presenting any evidence of the supposed implicit argument from DeMint.  Expert comments to the effect that unanimous consent is not fully secretive and that the process is not new are irrelevant in the absence of a reasonable case for the existence of that supposed implicit argument.

One quotation from an expert may be relevant:

Don Ritchie, a historian for the Senate, agrees. "It’s not a secret process: all committee hearings and markups are conducted in public and almost everything is published, as are the full texts of the bills," said Richie.

One question for Ritchie:  When do each of those things happen in the process?  Plainly it matters in the context of DeMint's comments, though not to PolitiFact.

Pleva concludes:

To recap. Jim DeMint claims that "94 percent of the bills that pass the Senate have no debate, no vote, no amendments, no reading of the bill, no online disclosure." But if we're speaking strictly about legally-binding "bills," that percentage is actually 27.9. Plus, the experts that we spoke with told us that passing non-controversial bills unanimously is a common practice, is often necessary to get things done, and, most importantly, there is nothing secretive or deceptive about the process.

DeMint's not just wrong, he is playing politics with statistics and has ignored details from the very report that his staff cited. Pants on Fire!

1)  Proper calculations leave DeMint far closer to the mark than Pleva and PolitiFact, and even that assumes the legitimacy of the dubious method of excluding one class of bills from consideration.

2)  Expert sources remarking about the lack of secrecy involved in the unanimous consent process do not appear to be replying in kind with the tenor or DeMint's remarks.  Secrecy in DeMint's speech appears to refer to the lack of transparency during the process.  Transparency after a bill has passed does not make the passage of a bill transparent as it takes place.  I doubt that the reporter's questions to his expert sources adequately probed the issue DeMint attempted to address with his comments.  For example, if a senator votes for a bill he has not read, the full contents of that bill are arguably a secret with respect to that senator.  That would represent at least one thing secretive about the process of unanimous consent.

3)  Pleva and PolitiFact made the poorest of cases that DeMint was wrong.  DeMint was correct about the percentage of unanimous consent passage for "all bills" for the year surveyed by the CRS.  DeMint was essentially correct that certain aspects of the unanimous consent process takes place out of the public eye and lack features we tend to think of as critical to the appropriate operation of the national legislature.  Pleva argued incoherently with respect to percentages and equivocally with respect to the secrecy issue.

The grades:

Lukas Pleva:  F
Bill Adair:  F

Afters:

As the time stamps from the FaceBook comments indicate, PolitiFact should be aware of problems with this report as of Thursday, June 17.

Unlike most online newspapers and the reputable blogs with which I am familiar, PolitiFact does not time stamp when publishing (Correction:  PolitiFact does include the publishing date in the sidebar) or updating its entries.  As of this posting there is one update to the PolitiFact story:

UPDATE: We updated the percentage of bills passed with no debate and no amendments to make the number more precise. We originally had said 27 percent, but it is actually 27.9 percent. The change does not affect our ruling.

Many more changes are needed, but at least the correction brings PolitiFact nearly a full percentage point closer to accuracy.  Even if the figure remains off by almost 40 percentage points.


June 20, 2010:  In my item #2 during analysis of the PolitiFact conclusion, replaced "universal" with "unanimous" in keeping with my original intent.  Also changed "percent" to "percentage points" in the final paragraph to enhance precision. 
Sept. 2, 2010:  Added correction regarding time stamp issue in the third-to-last paragraph.