Wednesday, October 22, 2008

Poll discovers Journalists think that Math is hard

OK, I just made that up, but it's pretty clear that Journalists simply don't understand statistics. Well, Stats is hard (I thought it was harder than Calculus, but that's me). However, it's not asking too much for them to have someone who does know stats to check things out for them, is it?

Because otherwise, they won't realize that the current polls are mathematically impossible:
The variance for current polls listed at Real Clear Politics for this morning ranges from the Pew poll which advertises a 14-point lead for Obama, to the Battleground poll which says the lead is only 2 points. The variance is too great (and there are polls relatively close to both ends, demonstrating proof of statistical invalidity for the published confidence level) for even the casual observer to accept as a reasonable. There are four polls which show a 10 point lead or greater for Obama, and another five which show a 6 point lead or less. It is mathematically impossible for so many polls to be valid, yet disagree to such a degree with valid methodology. [emphasis in the original]
Lots of discussion of statistical confidence intervals in the post, but it's mostly painless. The key parts, though, are about how the pollsters (Gallup, in this case) are just making up the methodology:
Gallup has admitted that this model has no precedent, and uses over-samples of urban and youth voters, in the presumption that they will sharply increase participation this year.
Other than the "this model has no precedent" bit, it's rock-solid.

Other pollsters seem to be similarly "rock-solid":
For example, in the Oct 20 poll Pew undersamples seniors and oversamples the 50-64 age group, oversamples high school only education by a large amount, and fails to note regional breakdowns or the urban/suburban/rural split. These are critical points which Pew fails to address, and which hshould [sic] make the reader wary.
JD Drummond has a must read about lousy polling methodology, and concludes that while the press is happy to be fooled, the candidates aren't acting like they are:
So, could I be wrong? I have to be honest and admit that I could. But in that case, we'd have to ask why the polls do not generally agree with each other, why Gallup is trying to spin three different models at the same time to get a grasp of the picture, why McCain and Obama are both so interested in Pennsylvania, yet neither is working very hard in Ohio right now. We'd have to explain why McCain-Palin rallies are now attracting thousands more people than Obama-Biden rallies, why Letterman suddenly found it cool to have McCain on his show and SNL decided they wanted Palin on theirs. We'd have to explain why there are not a lot of Obama signs visible, but we hear about his army of lawyers getting ready. We'd have to explain why McCain and Palin appear to be so relaxed while Obama and Biden look like they're worried.
So what does this all mean?
  1. I stand by my earlier post. I don't believe that the polls are valid. I'll even add to that, saying that I don't believe that the actual election results this year will be within the margin of error (IOW, they're all a crock).
  2. The pollsters are confused. Gallup, for example, uses three different and mutually exclusive weightings, none of which are historically valid, nd none of which agree with each other. See the Stolen Thunder link for details, but this reinforces point #1.
  3. The press doesn't have a clue that the polls are worthless. Whether this is due to mathematics ignorance or willful ignorance is left to the reader's imagination.
  4. The press may be trying to walk back from the ledge. Whether they can resist the temptation to double down on their hopey-changey candidate remains to be seen.
  5. The candidates know better. You want to know what's really happening? Watch them. No way is Obama up by 14 in Pennsylvania - why would he campaign there if he were?
It seems that a lot of this campaign has been about "fool the rubes." The polls, and the press' flogging of the polls fits this pattern. Ignorance or willful ignorance?

UPDATE 22 October 2008 19:27: OK, I'm slow on my blog reading today, because Kim du Toit sums the situation up with the Quote of the Day:
The Cliff Notes explanation is that if I’d ever designed a commercial qualitative research study with so obvious a set of prejudices and misconceptions, I’d have been fired.
UPDATE 22 October 2008 21:15: It just keeps on coming. Zombietime has a post about the "Clever Hans Effect" as a source of polling bias.
This phenomenon is now called "The Clever Hans Effect": the attitude of questioners can affect respondents' answers even if the questioner is trying to remain neutral. If a horse can notice subtle verbal or visual cues, then a person -- who is much more attuned to human culture and emotions -- can probably notice them even more so (possibly also without even realizing it).
I'd really like to see an analysis on what this does to the confidence interval and margin of error. Interestingly, the CE does not relate to accuracy, but rather to repeatability. In other words, you can have a perfectly repeatable poll stuffed full of, well, what comes out of Clever Hans.

Polls. You keep using that word. I do not think it means what you think it means.

3 comments:

Anonymous said...

Hopeful ignorance.

There is a lot going around on this very subject right now. Hell, the AP poll just came out and said they were tied, so who knows. I think Obama is ahead by a point or two.

Honestly, I am getting tired of the whole thing. Let's find out our fate, and move on.

NotClauswitz said...

The Press has been the narrative and meme driver for way too long, to the extent that they have even crafted a narrative for election theft "as American as apple pie" and equivocate the founder's experience ("George Washington won his seat in the Virginia House of Burgesses in 1758 by spending 40 pounds on booze for his neighbors."), and here now (as usual) the effort is to make the public so weary that they will succumb once again to the histrionics and haranguing, giving-in to the lies and deceit out of sheer fatigue.
It's time for the Press to go.

JD said...

I'll go with the polls are rigged at this point to suppress the McCain vote by convincing them they can't win so they don't show up to vote. . . .