Will you ever trust exit polls again?

[King]

Nope. "Kerry landslide"

Read the 3 pm exit polls, if it's clear, then go to the bar...

If not....

Read the 6 pm exit polls, if it's clear, then go to the bar...

If not....

AP online vote tallies till it's over

[Alcon]

I never all that much trusted them in the first place.

[M]

No. But they do reveal interesting demographic and issue trends.

[Cashcow]

Didn't they oversample women or something?

There is no reason to ever trust polls unless at least 5 of them show a clear (7+) lead for one candidate. That's how I sort it out.

[MODU]

I never all that much trusted them in the first place.

I second that.

[The Vorlon]

I have a very high level of faith in exit polls - 2 months after the election, after they have been post stratified and balanced, they tend to be very accurate.

[elcorazon]

As much as I enjoy getting the numbers early, in my head I'm fully aware (as this election showed) that the exit polls are merely a tool to analyze results.

Correct me if I'm wrong vorlon, but I think that they are not really scientific samples of voters (which is near impossible) but merely relatively random samples from various polling places, meaning that they can't really be analyzed until one knows how many votes there were in each polling place.

This is just a guess, but I'll bet the reason the polls were "wrong" has more to do with where most of the numbers came from, as opposed to the time of day people voted.  I'm betting that the exit polls were oversampling urban areas (either for logistical reasons, or for reasons of failing to anticipate the depth of turnout in other areas). 

The point of the exit polls is for future research primarily, and also to help with projections AFTER one knows more about the actual voting, not to just make a blanket prediction (in OHio Kerry's up by 3%- there's no way to project that from an exit poll).  Not to mention, that no matter how it's used it's still a poll, which is by definition fallible within and beyond the margin of error.

[The Vorlon]

As much as I enjoy getting the numbers early, in my head I'm fully aware (as this election showed) that the exit polls are merely a tool to analyze results.

Correct me if I'm wrong vorlon, but I think that they are not really scientific samples of voters (which is near impossible) but merely relatively random samples from various polling places, meaning that they can't really be analyzed until one knows how many votes there were in each polling place.

This is just a guess, but I'll bet the reason the polls were "wrong" has more to do with where most of the numbers came from, as opposed to the time of day people voted.  I'm betting that the exit polls were oversampling urban areas (either for logistical reasons, or for reasons of failing to anticipate the depth of turnout in other areas). 

The point of the exit polls is for future research primarily, and also to help with projections AFTER one knows more about the actual voting, not to just make a blanket prediction (in OHio Kerry's up by 3%- there's no way to project that from an exit poll).  Not to mention, that no matter how it's used it's still a poll, which is by definition fallible within and beyond the margin of error.

In any polling, be it pre-election, exit, post election, what ever, the basic theory, indeed the absolutely unbreakable foundation, is that your sample should look as much like the totat universwe you are trying to measure as possible.

In THEORY, you can get a fairly decent wave of exit polls by selecting maybe 20 or 30 precincts that taken in totality, are very similar to the entire state.

For example, if the population of the entire state is 15% African American, the total population of your 20 precincts should be also 15% African American..  You "balance" or "pre-stratify" your sample in the same way for men, women, rich, poor, young, old, etc, etc...

Needless to say, a "minor" detail like if people have jobs and work during the day should also be factored into this equation...

The first wave of exit polls we got this year were all taken before 2 pm or so local time...

So other than having way too many women, eldelrly, students, unemployed people, too few men, too few people who worked during the day and having vastly too few folks in the 35-65 age bracket, the same looked exactly like the state they were trying to measure...

From a polling perspective, this presents a bit of a problem...

AFTER the polls close, and you capture a snapshot of EVERYBODY who has voted, the exit polls are +/- the usual MOE pretty close...

If you did a poll of Florida and only made calls in Broward County.. how good would your poll be...? - A 6 year old could figure out it would not be too good...

Same basic argument.. The folks who voter in the morning are just different demographically than the folks who vote at night...

It is no more complicated than that

You can TRY to balance your sample and weight the heck out of it to compensate for a sample that looks very little like the entire population, and you might be within maybe 5% or so on the first crack... or not...

Exit polls...

2000 => Gore by 10 in Florida
2002 => "Senator" Bowles for North Carolina, Texas Senate too close to call...
2004 => Colorado, South Carolina too close to call, Kerry by 16 in Pennsylvania.....

[muon2]

Samples and patterns work in unusual ways. While trying to get the right mix of possibilities, the obvious can be missed. Here's an anecdote I heard around 1990.

A military project was working on pattern recognition software to identify enemy tanks in digital images. The program was designed to "learn" by being exposed to a wide array of images, some that would be enemy tanks, and others that would be non-targets. The programmers went through the "training" phase, carefully alternating a wide variety of traget and non-target situation. They included differences for viewing angle, time of day, and background.

When they tested the "trained" program in field, it failed miserably. It called every other image a target regardless of content - even if it was blank! Then again, the program had indeed identified the simplest pattern. The input sample alternated between targets and non-targets, so it behaved exactly the same way. It "learned" to alternate between true and false.

[elcorazon]

but vorlon, isn't it true that the exit polls continued to have results that were quite different from the actual results, even after they were ALL tabulated when the polls were closed.  Isn't that why many Bush states that weren't too close couldn't be called until long after the polls closed?  And why states that were fairly close, but Kerry won were called a bit sooner (the exit polls had Kerry winning more handily than the real numbers showed)? 

If your explanation, vorlon, is all that was at play here, then, the exit polls in fact weren't even "wrong"; the people were just looking at the wrong data.

Or did the networks misuse the data AGAIN by assuming that early exit poll numbers were real numbers.

One more point:  Throughout my working years, I've always voted first thing in the morning.  That way, whenever I leave work (even if it's AFTER polls close - which in some states is 6:00 p.m.), I can head right home.  I realize that many  have rigid morning regimens they don't want to interrupt, but I wonder if your anecdotal description of who voted in the morning is fair.  I suspect many voters were working people in the morning (I voted before 8:30 in the morning on my way to work).  Those with nothing to do all day are as likely to vote in the afternoon as the morning I'd guess.  I would think there would be a disproportionate number of elderly voting in the morning.

[J. J.]

Turnout in my voting district, heavily Black and heavily Democratic, was exceptionally large when the polls opened; there was very little late day voting.  No likes at the polling stations I drove past at 5:00-7:00 PM.

[Davo]

You must be logged in to read this quote.

Exactly.  I don't see why so many people are trusting the exit poll internals when the horse race numbers had Kerry winning 350 electoral votes. 

[Alcon]

I notice that exit polls always overestimate the Democrat Presidential candidate in Pennsylvania - at least in 2000 and 2004 by similar margins. This caused the premature network calls this year for the state.

[elcorazon]

here's an interesting article about the issue of exit polls and this election:

http://truthout.org/unexplainedexitpoll.pdf

[J. J.]

I would treat an exit poll like all other polls.  Don't believe one, but believe several.

[muon2]

here's an interesting article about the issue of exit polls and this election:

http://truthout.org/unexplainedexitpoll.pdf

I found the article interesting, yet is has a significant flaw IMO. The author carefully walks through the statistical nature of the polling data, and draws correctly the conclusion that the discrepancy is not due to random fluctuations. However, the author seeems to be overly dismissive of systematic errors that could arise.

The probability of systematic errors in the discrepancy between exit polls and vote tallies is clear. In 10 out of 11 battleground states there is an overestimate of Kerry votes in the exit polls. The mean overestimate in the 11 states is 4.4%, and the standard deviation of the overestimate mean is 0.8%. This is a so-called 5.5 sigma effect, and would be significant in most any scientific study.

The author tries to dismiss systematic errors of technique in the polls, but largely relies on hearsay, and admits that there is as yet no access to raw data on which to draw more solid conclusions. By dismissing these errors to easily, he still allows a substantial mismeasurement of the count to remain as a possibility in his conclusion.

However, the author overlooks the fact the 11 different states all see similar discepancies. The importance of this is that the 11 different states reflect many different voting technologies and counting procedures. To have them all provide a similar systematic shift is hard to fathom. Yet all 11 states used the same exit polling methodology, so a systematic shift could be accounted for by some aspect of the exit polls alone.

Occam's razor suggests that it is a problem with the exit polls. To dangle vote fraud conducted on a national scale as an alternative has no basis in the statistical data presented in this paper.

[elcorazon]

here's an interesting article about the issue of exit polls and this election:

http://truthout.org/unexplainedexitpoll.pdf

I found the article interesting, yet is has a significant flaw IMO. The author carefully walks through the statistical nature of the polling data, and draws correctly the conclusion that the discrepancy is not due to random fluctuations. However, the author seeems to be overly dismissive of systematic errors that could arise.

The probability of systematic errors in the discrepancy between exit polls and vote tallies is clear. In 10 out of 11 battleground states there is an overestimate of Kerry votes in the exit polls. The mean overestimate in the 11 states is 4.4%, and the standard deviation of the overestimate mean is 0.8%. This is a so-called 5.5 sigma effect, and would be significant in most any scientific study.

The author tries to dismiss systematic errors of technique in the polls, but largely relies on hearsay, and admits that there is as yet no access to raw data on which to draw more solid conclusions. By dismissing these errors to easily, he still allows a substantial mismeasurement of the count to remain as a possibility in his conclusion.

However, the author overlooks the fact the 11 different states all see similar discepancies. The importance of this is that the 11 different states reflect many different voting technologies and counting procedures. To have them all provide a similar systematic shift is hard to fathom. Yet all 11 states used the same exit polling methodology, so a systematic shift could be accounted for by some aspect of the exit polls alone.

Occam's razor suggests that it is a problem with the exit polls. To dangle vote fraud conducted on a national scale as an alternative has no basis in the statistical data presented in this paper.

nice analysis muon.

[Democratic Hawk]

The CNN national exit poll wasn't too wide off the mark. I've been number crunching on party ID/ideology and found:

Party ID:

Bush - Democrat 4.07 + Republican 34.41 + Independent 12.48 = 50.96%

Kerry - Democrat 32.93% + Republican 2.22 + Independent 12.48 = 47.89%

Ideology:

Bush - Liberal 2.73 + Moderate 20.25 + Conservative 28.86 = 51.54%

Kerry - Liberal 17.85 + Moderate 24.30 + Conservative 5.10 = 47.25%

Dave

[J. J.]

There was something I noticed on the ground on election day, very high early turnout.  It looked at about 2:00 PM that there was an exceptionally high African American turnout, going by my voting district.  Yes, but it was early voting and it was going on across the state.  You had someone from PA's "T" reporting a 55% turnout at noon.

I suspect that a lot of the early voting were Democrats, voting early.  The early polls might have correctly gotten them and the GOP voted later.  There was almost no late voting in my area.