Flawed Statistical Thinking, the IRS, and Loooooooong Odds

Let’s say there’s a casino where you can’t choose to play. Everyone is forced to participate in the gamble and instead of winning money, the “winners” get punched in the face. A man plays at the dice table 30 hours in a row and never gets punched in the face once. Lucky jerk. Then he says that the casino should be under different management. Within 15 minutes of this statement, he rolls snake eyes (the lucky face-punching number) three times.

Should we consider the possibility that his punches were perhaps the result of his statement about management? They could just be random. Randomness does work that way sometimes.

What if we then discovered that another table in that casino admitted to using loaded dice. Fans of the casino might argue that just because the dice were loaded at one table doesn’t mean they’re loaded at our poor sap’s table. But I think that most of us would take the triple face-punching and the loaded dice story, put them together, and conclude that management conspired to have their critic punched.

I float this analogy in service of recent events.

A few days ago, Nate Silver published a post noting the flawed statistical thinking from Peggy Noonan regarding IRS audits of Romney supporters. Go read it for yourself, but the long story short is that Silver is (rightfully) irritated by people who hold up a few anecdotal examples as proof of a conspiracy. However I suspect in his haste to make a larger valid point, Silver got the odds wrong on of one of the cases Noonan holds up.

In summary, Frank VanderSloot donated a substantial amount of money to groups in support of Mitt Romney. He was mentioned by name by President Obama’s campaign website in an attack that must have seemed normal to lots of people but seems mildly creepy to me. During the next four months, he had to deal with 3 audits: one individual audit, one audit by the Department of Labor and one audit of his business.

In a move that is very much unlike him and that I can only attribute to a careless reading of the events or “blogging-while-buzzed” (full disclosure: that is me right now) Silver has drastically miscalculated this case. Silver looks at the odds of a single audit and maintains (as any reasonable statistician would) that the odds of a single audit of a single individual do not rise to the level of conspiracy.

One of the things I love about randomness is how it builds. Flip a coin and call the result. Get it right and no one thinks you’re anything special. Same thing if you call the second flip correctly. By the third flip, people are getting irritated, waiting for you to be wrong. By the 10th flip, they want to change the coin because they suspect it’s rigged. Every subsequent flip drives the odds higher in a way that becomes almost impossible to comprehend. By the 34th flip, the odds of calling them all right far outweigh the population of the planet.

What we’re talking about here isn’t a single audit (which is what Silver based his math on). We’re looking at three different audits within four months of being mentioned by the President’s team as a “very bad man”.

What are the odds of that?

Fortunately for us, those odds are very easy to count. The odds of someone making over $1 million per year being audited is 12%. Let’s assume it’s similar for medium sized business. Then the odds of being audited AND your business being audited in the same year drops down to 1.4%. That’s certainly lower, but not so low that we should be overly suspicious. Hey… this kind of thing just happens.

But then add in the Department of Labor audit. The Department of Labor  “conducts more than 3,000 audits each year“. For the sake of statistical generosity, let’s call that 4,000. There are 30 million businesses in the United States, so the chance of an individual business being targeted randomly is pretty low, under 1%.

Now let’s add all those odd together. The chance of having all three of these audits in one year is 1 in 520,833 or generously rounded up to 0.002%. This is the kind of thing we look at askance and say “Huh. That’s really kind of weird.”

This would look weird even if the IRS hadn’t admitted to rigging the game (albeit in a different context). Given the admissions they’ve made so far, I think any serious mathematically minded individual should look at the odds and say “That’s odd.”

I feel that, given the circumstances, the burden of proof should lay at the feet of the auditors. I want them to prove this wasn’t politically motivated because, given that they’ve admitted to so far, this particular case is extremely weird.