PolitiFlub: Fact checkers use grading criteria selectively on Gerry Connolly rating

Let's see ... how long ago was it that Louis Jacobson proclaimed in a PolitiFact National story that PolitiFact "consistently ruled in the past that the economy is too complex to assign full blame (or credit) for job gains or losses to a president or a governor"?

Not long ago, as it turns out.

Fortunately the ups and downs of the Dow Jones industrial average pose no such problems of complexity.  Consideration of cause and effect was apparently pitched out the window when PolitiFact Virginia rated a claim by Rep. Gerry Connolly (D-Va.):

"We’ve doubled the stock market from where we started when he was sworn into office -- Bush had gotten it down to 6,500. It’s double that today," Connolly said.

Connolly's wording certainly seems to push the cause-and-effect angle, but the fact check contains no hint at all that PolitiFact Virginia docked Connolly for taking credit on behalf of the Democratic Party and the Obama administration.  At least Connolly was generous enough to share with President Bush the credit for dropping the Dow to 6,500 while modestly withholding credit to a Congress controlled by the Democratic Party during the precipitous decline.

In the end PolitiFact rated Connolly "Mostly False" based on cherry picking a time frame inconsistent with the one he specified.  One sentence in PolitiFact's summary takes note of the claimed cause and effect relationship, but without the assumption that Connolly's "exaggeration" refers to the credit offered to Democrats rather than his fudging of the numbers it looks like Connolly isn't docked for offering dubious credit:

The Dow rose 55.5 percent from Inauguration Day to the day of Connolly’s statement. Two broad indexes tell a similar story: the S&P 500 went up 59.1 percent and the Wilshire 5000 increased by 66.1 percent. It’s debatable how much of the improvement can be pinpointed to the president’s policies.

So Connolly’s claim has a foundation that is cracked by the weight of exaggeration and cherry-picked data. We rate it Mostly False.

With Connolly's cherry-picking justification set aside, his claim was off by almost 45 percent--typically the stuff of a "False" rating using PolitiMath.

If one claims to consistently rule with cause-and-effect relationships taken into account then one ought to do so consistently.  It seems likely that PolitiFact Virginia did not penalize Connolly for his claim of causation; if Connolly was penalized then the story failed to clearly communicate it.

PolitiFact Virginia: The Good Umpire speaks

PolitiFact Virginia announces its one-year birthday along with a break from the tradition of not publishing a summary of its ratings by party.  With a word about the role of a good umpire (bold emphasis added):

A frequent complaint -- from Democrats and Republicans alike -- is that we’re biased toward the other side. Sometimes, readers ask for a breakdown of our ratings by political party and are surprised to learn that we did not keep count. That’s because we’re focusing on the action in front of us, trying to make our most objective call. A good umpire shouldn’t keep track of the number of close plays at home plate he or she has called in favor of the home team or visitors.

I'll suggest there's a problem here.

What if the record shows the umpire consistently favoring the Yankees?  Does the umpire wish to remain ignorant of what the record shows about his judgments?

The comparison shows PolitiFact Virginia rating more Republican statements than Democrat statements, 79 to 58.  PolitiFact Virginia says that's fair considering the Republicans control the governor's office and the legislature, along with a spirited Republican primary for a Senate seat running against the Democratic former governor of Virginia, Tim Kaine.

There's a ready-made study topic.

Democrats scored slightly higher on a numerical basis (2.78 to 2.66), but the striking thing is PolitiFact's continued failure to note that such numbers mean very little without some control for selection bias.  Trusting editorial selection to pass as a random determinant simply doesn't cut it.

The numbers do contain some potential evidence showing a degree of even handedness at PolitiFact Virginia:  The percentage of "Pants on Fire" ratings is nearly the same for both parties, with the Republicans actually scoring slightly lower by that measure.  I suggest the number of "Pants on Fire" ratings is significant because PolitiFact offers no objective measure for using "Pants on Fire" as opposed to simply "False."

That's not to say that the, uh, Subjectivometer cannot be fooled, of course.  It always matters what particular rulings the editors choose.  In primaries, for example, one can easily rule favorably on a claim by a Republican that makes it look bad for other Republicans.  So conservatives suffer the harm while looking pretty on the "Truth-O-Meter" scale.

Bottom line, the numbers PolitiFact Virginia offers us don't mean much.  We're left to wonder whether the birthday story appears this way if Republicans score a perfect 5.0 on the Truth-O-Meter while Democrats score below a 2, or any result that seems questionable in terms of fairness.  It's hard to see why PolitiFact Virginia publishes the story unless it is designed to make the operation look fair.  The numbers don't say that without a control for selection bias.

Grading PolitiFact (Virginia): Bob McDonnell and job creation in Texas

Words matter -- We pay close attention to the specific wording of a claim. Is it a precise statement? Does it contain mitigating words or phrases?
--Principles of PolitiFact and the Truth-O-Meter

The issue:

(clipped from PolitiFact.com)

The fact checkers:

Jacob Geiger:  writer, researcher
Warren Fiske:  editor


Analysis:

Gov. Bob McDonnell (R) of Virginia wouldn't mind if his state could emulate the jobs record of Texas.  McDonnell, in an op-ed for the Wall Street Journal, made a recent statement praising Texas on that point.

"Under Gov. Rick Perry’s leadership [Texas] has created more jobs over the last decade than the rest of the states combined," McDonnell wrote.

And PolitiFact was there:

We wondered if Perry’s record on creating jobs matched McDonnell’s claim.

With the fact check identified, it is research time. PolitiFact contacted McDonnell's office for backing information before launching its own survey:

Our research took us down a road traveled by PolitiFact Texas in examining various claims from Perry that most of the jobs created in the U.S. are in the Lone Star State.

PolitiFact Texas has debunked several of these eye-popping claims, noting that they are based on statistics from a minority of states that have had overall job gains in recent years.

Though PolitiFact Virginia mentions "several" claims debunked by PolitiFact Texas, we have but one represented in the list of references.  PolitiFact Texas debunked Governor Perry to the tune of "Half True" in that case, and that story contained a link to another story featuring a claim similar to the current subject where the claim received a "False" rating.

PolitiFact, you see, pays special attention to the words politicians use.  If they say "we created x jobs" it does not necessarily mean net jobs.  And that's misleading.

Unless you're Rick Perry:

"Gov. Rick Perry says Texas has created more than 850,000 jobs, more than the other states combined" ("True")

Or Joe Biden:

"Vice President Joe Biden says more jobs were created in the last year than in all eight years under George Bush" ("MostlyTrue")

Or Mitt Romney:

"Mitt Romney tells CPAC that Canada created more jobs in January than the U.S. did" ("True")

Or Debbie Wasserman Schultz:

"Debbie Wasserman Schultz says job creation in 2010 on pace to eclipse growth during George W. Bush's entire administration" ("Half True)

You get the idea, after noting that Wasserman Schultz's "Half True" had nothing at all to do with gross job creation as opposed to net job creation.  The fact is that "job creation" is a statistic that we normally take to mean net job creation.  Yet PolitiFact applies an uncharitable interpretation to job creation statements from governors McDonnell and Perry with no apparent justification from the context.

PolitiFact Texas:

Veronica Sanchez Downey, who analyzes data for the commission, said she'd feel comfortable summarizing the commission's conclusion about Texas job gains this way: "Of all the states that gained jobs in the U.S. including the District of Columbia, Texas accounted for" 67.2 percent of the added jobs from November 2007 to November 2008. The 67.2-percent figure reflects adjustments of the data since the initial research.

Perry, of course, didn't say it that way.

Right.  Perry did not mention the term "net" while talking about job gains.  He just talked about job gains, which people tend to understand as net job gains.  That interpretation is so ubiquitous that PolitiFact typically favors it without discussion, as shown by the preceding examples.  So Perry gets "debunked" by making an exception to the normal conventions of English interpretation.

You might get more consistent results using the "Magic 8 Ball" instead of PolitiFact.

Let's get this over with:

Let’s review. 

McDonnell wrote that under Gov. Perry, Texas "has created more jobs over the last decade than the rest of the states combined."

But the Virginia governor used net job numbers. Those are different from job creation figures, which are not compiled by the BLS or other national authority. So using net job numbers and calling them job creation figures is a case of comparing apples to oranges.

We rate McDonnell’s claim Barely True.

It's just too bad for McDonnell that PolitiFact Virginia failed to take note of the several PolitiFact items that treated job creation claims in terms of net jobs, with ratings ranging from as low as "Half True" to as high as "True."

Reply hazy.  Try again.

The grades:

Jacob Geiger:  F
Warren Fiske:  F

Fiske and Geiger had the opportunity to notice PolitiFact's disgraceful inconsistency in its treatment of job creation claims.  The team somehow failed to notice PolitiFact's history of (appropriately) treating job creation claims as dealing with net job creation where the context warrants taking them that way.  The PolitiFact Virginia team instead zeroed in on a handful of bizarre cases where the obvious intent was ignored.

Afters:

What if we were to look at cases where PolitiFact has ruled on jobs claims and where considering the jobs claims in terms of gross jobs rather than net jobs might have affected the ruling?

Grading PolitiFact (Virginia): Morgan Griffith milking it?

The issue:

The fact checkers:

Jacob Geigner:  writer, researcher
Warren Fiske:  editor

Analysis:

Skipping past the cutesy introductory comments PolitiFact often offers to the meat of the story:

In a February newsletter to constituents, Griffith claimed that new EPA rules treat milk spills the same way they treat oil spills. He titled the newsletter "Crying over spilt milk." 

"What do spilt milk and oil have in common?" he wrote. "Quite a bit, according to the EPA. In fact, a new ruling by the EPA would force dairy farmers to comply with the Spill Prevention, Control and Countermeasure Program when dealing with spilt milk -- the same regulations oil and natural gas producers must follow. The EPA’s reasoning is that milk contains ‘a percentage of animal fats, which is a non-petroleum oil.’ It appears spilt milk is just as threatening as an oil spill."

The quotation from Griffith comes with sufficient context so that it accurately represents the content of the newsletter.  Good so far.

PolitiFact:

Beth Breeding, Griffith’s press secretary, said her boss’s information came from the EPA’s website. So we went there.

Right away, we found problems with Griffith’s claim. The website says milk has been regulated under the Spill Prevention Control and Countermeasure program since 1973, when the Clean Water Act took effect. The law was passed by Congress the preceding year over the veto of Republican President Richard Nixon. So this is hardly a "new ruling," as Griffith says. It has been in effect for 38 years.

Skepticism is warranted.  Bear that in mind as we allow Geigner to proceed with his case:

The EPA site says "since the SPCC rule became law in 1973, all kinds of oils including petroleum and edible oils (such as animal fats and vegetable oils) have been considered oils. This is because the SPCC rule gets its definition of ‘oil’ from the Clean Water Act, which was authored by Congress."

Who's buying it?

Here's how the Clean Water Act reads:

OIL AND HAZARDOUS SUBSTANCE LIABILITY 

SEC. 311. (a) For the purpose of this section, the term—(1) ‘‘oil’’ means oil of any kind or in any form, including, but not limited to, petroleum, fuel oil, sludge, oil refuse, and oil mixed with wastes other than dredged spoil;

The EPA apparently wants us to believe that if we have a tanker of water with some peppermint oil added to it then the tanker of water is subject to regulation under the hazardous waste procedures mandated by the Clean Water Act.  Who's buying it?  And who's buying the notion that the EPA has been duly regulating milk under the Clean Water Act since roughly 1973?  Anybody?

Oh.  PolitiFact buys it.

PolitiFact:

So what’s new?  According to the EPA, the only thing that comes close is a rule change it announced on Jan. 15, 2009. It goes in effect at the end of this month. 

The simple purpose of the change is to exclude milk and dairy farms from the spill rules governing oil products. That’s the exact opposite of what Griffith claims. Here’s what the regulation says:

"EPA proposes to exempt milk containers and associated piping and appurtenances from the SPCC requirements provided they are constructed according to the current applicable 3-A Sanitary Standards, and are subject to the current applicable Grade "A" Pasteurized Milk Ordinance," or similar state laws.

Translated into plain English, the rule means milk storage will no longer have to meet the EPA’s oil spill rules, provided storage tanks meet pasteurization laws. In Virginia it is illegal to sell "raw," or unpasteurized, milk, so the state’s dairy farmers should already be in compliance with the new standards. 

To parrot PolitiFact, there are immediate problems with the EPA's explanation and PolitiFact's acceptance of the same.

If it is true that there is nothing new in the regulations other than an exemption for milk storage, then why has the milk production industry clamored for the exemption as though it provides relief from new requirements?  On its face, it makes no sense.

A journalist ought to wonder why the dairy industry thinks it is encountering a new burden if not for the exemption.

The New York Farm Bureau has an explanation:

The SPCC rules were revised in 2002 but left considerable ambiguity about the regulations’ impact on milk, which can be considered a non petroleum-oil because of its butterfat content. In 2005, an EPA presentation made clear that the current SPCC policies could be applied to milk. Since then, milk producers have been working to ensure that EPA regulations don’t apply to milk producers. The Bush administration proposed a rule to officially exempt milk producers from any SPCC regulations, but the rule was held up as part of the Obama administration’s complete review of all EPA rules proposed under the previous administration. A rule exempting milk has never been finalized.

Now doesn't that sound quite a bit more plausible than "milk has been regulated under the Spill Prevention Control and Countermeasure program since 1973, when the Clean Water Act took effect"?  It is exactly the sort of information that helps fill the gaping hole in Geigner's story.

The EPA ring through Geigner's nose leads to the EPA-friendly conclusion:

Let’s review our findings.

Griffith claimed a "new ruling by the EPA would force dairy farmers to comply" with strict regulations for spills and leaks. He said the rules were the same as those enforced on oil and natural gas companies.

In fact, these regulations have been in place for 38 years and are not new at all. The "new ruling" from the EPA, announced in 2009 and taking effect in a few weeks, actually excludes milk from the spill standards, giving dairy farmers fewer regulations to meet. That’s the exact opposite of what Griffith claims.

Sure, Griffith got some of his information from an inaccurate editorial in The Wall Street Journal. But a  congressman who is railing against a federal agency has the means to get his facts right.

Griffith is dishing udder cow chips. We rate his statement False.

Way to go, EPA.  You made Geigner miss the story.  And somehow you did it so well that he ended up inserting some haughty 'tude in the text.

Even if we were somehow able to ignore the hole in Geigner's story, he couldn't justify the "False" rating for Griffith.  Geigner implicitly admits that raw milk producers will not obtain the proposed exemption.  So Griffith is "Barely True" at worst since the lack of an exemption makes his statement at least somewhat true.  Under current rules and lacking any exemption or delay in enforcement, raw milk spills would be treated like petroleum spills by the EPA.  And that would apparently remain the case under the permanent exemption promised in the spring.

This piece warrants the special tag "journalists reporting badly."

The grades:

Jacob Geigner:  F
Warren Fiske:  F


March 14, 2011:  I repented about the words I used to describe the hole in the story.  I chose them poorly and they did not accurately communicate the truth of the matter.  I've preserved the old wording for posterity  below: 


"why has the milk production industry clamored for time to meet the exemption standards?"
"A journalist ought to wonder why the dairy industry needs time to comply with an exemption standard."

Apologies to any who were confused or misled.

Grading PolitiFact (Virginia): Tim Kaine on jobs & pop

To assess the truth for a numbers claim, the biggest factor is the underlying message. 
--Bill Adair

The issue:

The fact checkers:

Sean Gorman:  writer, researcher
Warren Fiske:  editor

Analysis:

Kaine's claim obviously represents a numbers claim, so we'll be on the lookout for the underlying point.  That's the most important thing when fact checking a numbers claim.  Or so it is said.

Take it away, PolitiFact:

Kaine said in a speech that during the Bush administration "while the population in that period grew by 10 percent, the number of jobs in the nation grew by 1 percent."

We wondered if that was really the case. Alec Gerlach, a DNC spokesman, couldn’t identify the source on which the former Virginia governor based his claim. So we tried to find it on our own, comparing census records from January 2001 - the month Bush was sworn in - to January 2009, when Bush left office.

It makes sense to investigate the raw numbers prior to assessing the underlying point.

PolitiFact found that the population increased by about 7.8 percent according to Census Bureau estimates.  Kaine's number was inflated by about 22 percent in order to reach the minimum reasonable interpretation of "10 percent" (9.5 percent).  Insisting on the full 10 percent--and I don't--would represent an inflation of about 36 percent.

Additionally, PolitiFact found that non-farm employment increased only about .83 percent from the start of Bush's two terms to the end.  Kaine could be viewed as doing Bush a favor by rounding that figure up to 1 percent.

While Kaine’s numbers are a little off on employment and census, he’s right in suggesting that that U.S. population grew 10 times faster that (sic) jobs during George W. Bush’s presidency.

Hmmm.  Could that be the underlying point?  Or was it hidden in the three paragraphs' worth of caveats?

Even so, several economists told us comparing raw population growth and job numbers is not a preferred way to examine job creation versus demand. That’s because not all of the new people -- such as babies born during the Bush administration -- need jobs.

We were told a more meaningful measure is labor force participation: the percentage of the population between 16 and 64 that is either employed or looking for work. According to BLS, participation was at 67.2 percent when Bush took office and 65.7 percent when he left.

Here’s another thing to keep in mind: Economists have told us time and again -- on this claim and others -- that the policies of presidents and governors have only minor impact on economic cycles and job creation. "They get too much credit and too much blame," said Sylvia A. Allegretto, economist at the Institute for Research on Labor & Employment at the University of California, Berkeley.

... and hinted at in the introductory paragraph(?):

Democratic National Committee Chairman Tim Kaine may not have given the crowd at the Feb. 19 Jefferson-Jackson Dinner in Richmond the scoop on whether he’s running for the U.S. Senate, but he made no secret of his disdain for former President George W. Bush’s record on job creation.

No, apparently the 10:1 thing was the underlying point, if the conclusion is any indication:

Kaine’s numbers are a little off his and spokesman doesn’t know where they came from. Even so, his suggestion that population growth outpaced job growth by a ratio of 10 to 1 is pretty much on the mark, so we rate his claim Mostly True.

The grades:

Sean Gorman:  F
Warren Fiske:  F

The grades assume that Bill Adair knows what he's talking about when he says the underlying argument is the most important aspect of a numbers claim.  Making the supposed 10:1 ratio the underlying point is little better than making the raw numbers the underlying point (and thus the same as the superficial point).


Afters:

The presentation by Gorman and Fiske left it unclear just how far off the mark Kaine ended up with respect to the supposed 10:1 ratio.  Divide 7.8 by .83 and the resulting 9.4 ends up reflecting a ratio closer to 9:1  than 10:1.