Part of my undergrad thesis was actual about this exact phenomenon in congressional elections!
I did a study comparing polls in the last month or so of the election with the actual election results...using about 150 polls from the 1998 and 2000 election cycles.
What I found was that challengers outperformed the polls by an average of 3%, even when holding a number of other factors constant (including party, year, source of poll, etc). The gains was statistically significant and normally distributed (I don't remember the SD off the top of my head...probably around 2%).
HOWEVER, there were a set of race in which the opposite was true. These were races in which the challenger had what I call a "valance" advantage over the incumbent. That is, where the challenger had better personal qualities, regardless of ideology. Generally, these were race where:
- The challenger was a former representive from that district or former statewide nominee
- The challenger was a local celebrity
- The incumbent was involved in a significant personal scandal
In these case, the incumbent actually outperformed his polls. The incumbent's overperformance was similarly normally distributed around 3%.
I think you might have seen this exact phenomenon at work in yesterday's Virginia primary involving Jim Moran.
Anyway, the rest of the thesis was about creating a rational-choice spatial model of voter preferences using these results, with the valence/ideology distinction at its core.
I don't know if this has any relevance to presidential elections, where both candidates are better known.
You state one of the biggest issues with polls: they don't always follow a pattern. Sounds like it was a good project though. Would love to read some of it if you have it electronically. As for Moran, he was going to win the primary anyway. The contender stated Monday, when trying to identify himself to the voters, that he was a "Teddy Kennedy" Democrat. No matter how bad Moran is, I'd vote for him over a "Teddy" any day.
