Talk Elections

I'm a bit afraid ARG wasn't so bad after all, at least in its last polls in big swing states:
VA, OH, NC, MO, IN, FL, CO.

Either its absolute numbers were good or the gap between Obama and McCain was right....

CNN Opinion Research had the best (OH, FL, CO, NV) and some bads (VA, NC, IN).

Rasmussen hasn't been very good this time. Mason-Dixon and Quinnipiac neither.
Survey USA not very convincing.

Isn't this the BIG SURPRISE of this election: ARG rather accurate in the end !!!
Wake me up, I must be in the middle of a weird nightmare...

538.com will probably give us an analysis on this, but maybe some home experts here have more precise facts than this vague topic ?

I'm working on this right now. I'm only including polls taken during the last week. A bit arbitrary but I have to make the cut-off somewhere.

Anyway, so far:

Zogby: Standard error: 2.66%, McCain bias of 0.73%

ARG: Standard error: 4.17%, Obama bias of 1.78%

SUSA: Standard error: 3.73%, McCain bias of 0.95%

Strategic Vision: Standard error: 1.82%, McCain bias of 2.48%

I note that Strategic Vision does have a bit of a GOP bias but was still pretty accurate. Not based on a lot of polls though. To be continued...

Are those results normalized at all by which states the polls were done in?

For example, I would suspect a pollster which happened to skip over Nevada, Iowa, and Alaska to have a much better average than one which polled all three. 

No, because there isn't really any objective way to normalize in this way. The assumption of what states were hard to poll are based on the polls themselves. I see your point but it's hard to account for in an effective way.

Overall, the polling was pretty good. They did however blew up in the states you mention and also in Arkansas. The main battlegrounds were polled good however.

Anyway, here is my preliminary list, from best to worse:

Strategic Vision
Quinnipac
Zogby
CNN/Time
PPP
SUSA
ARG
R2000
Rasmussen
Mason-Dixon

The most surprising result is obviously Ras and MD in last.

In terms of bias:

Republican bias:

SV 2.48%
MD 1.71%
SUSA 0.95%
Zogby 0.73%
Ras 0.68%

Democratic bias:

ARG 1.78%
R2000 1.52%
CNN/Time 0.97%
Quinnipac 0.81%
PPP 0.53%

There's an objective way to normalize for that, I'd think.  Average error relative to average state error.  Creates a good deal of noise, though, and rewards screw-ups in small states.

MD blew KY (underpolled McCain) and NV (overpolled McCain) really bad. They also did pretty bad in OH, PA and AZ (pro-Obama in AZ, pro-McCain in the other 2).

Rasmussen is really pulled down by blowing Nevada, Arkansas and Alaska. Other than that they were more or less allright in most places.

Yeah.  Looking at it in a qualitative way, I think Rasmussen was the big winner.  SV continues to be a very strong pollster with a fairly regular GOP bias.  SurveyUSA is OK but too volatile.  ARG sucks even when they're stealing other people's results.

Same old story as '04, it seems, except Mason-Dixon sucks now.  :(

That would work if every pollsted polled the same state or maybe if the overall number was big enough. But when I say some pollster are bad that is based on those pollsters doing badly in some states. I can of course say that some states are hard to poll because some pollsters did badly in those states. But it is a little bit too circular for my liking. And I think the regression anaysis that could be made would have too little data. Anyway, I feel my work with this stuff is done. If you want to do this analysis though, be my guest. :)

I think you're right on every count and I'm way too lazy to try other methods.  :P  No complaints from here.

I think Zogby is a pretty big story though. He blew Indiana but otherwise did a pretty good job.

I've been looking at the averages, and in swing state polls taken within the last 2 weeks (haven't done non-swing states yet, but with the large margins there, I think there's probably greater room for error), ARG, Zogby and CNN/Time were actually some of the best. Rasmussen and Mason-Dixon had on average the highest amount of error, and they both also had the largest error in McCain's favor. Meanwhile, PPP, which was supposed to be a Democratic hack firm, didn't seem to err in one direction or another.

Final Polls in CO, NV, NM, MT, MN, WI, MI, IA, MO, IN, OH, PA, VA, NC, GA, FL, NH, AZ, CA, NY, NJ.

Average Absolute Error
Rassmussen: 3.42      
Survey USA: 2.60      
PPP: 2.54      
Quinnipiac: 1.08      
Research2000: 3.03   
Mason-Dixon: 3.14      
ARG: 2.04      
CNN/Time: 2.18      
Zogby: 1.775

Average Non-Absolute Error (McCain +, Obama -)
Rasmussen: 2.56      
Survey USA: 0.9      
PPP: -0.16      
Quinnipiac: -0.43      
Research2000: -0.69      
Mason-Dixon: 2.68      
ARG: -0.44      
CNN/Time: -0.74      
Zogby: 1.35

We can do a head to head comparison between any pair of pollsters by focusing just on the states where both competed.  For example, Pollster 1 would beat Pollster 2 by 0.35 if he averaged 0.35% closer to the actual margin in states where both 1 and 2 released a poll in the last week.  This somewhat mitigates the effect of different states being harder to poll. 

What would the results table look like for such a league? 

Why don't you tell me? ;)

Ok, I may do a few of those, even though it's a lot more work than my original approach.

Rasmussen v SUSA
2.44%            2.29%

I'm in the process of doing one right now, though possibly based off of a different data set (I included the last poll from every firm which had one with a median date after 10/27).  I'll edit this post once the results are finished. 

______________________________________

hmm...the problem I'm running into here is that some pairs only have one or two states in common, leading to very noisy results (quite a few cycles where A beats B beats C beats A).  The straight averages are below (Positive means the firm in the row is on average that much closer to the mean, negative means the firm in the column)


A couple notes:
-As mentioned above, I took all polls with median day after 10/27 (including a couple 10/27-10/28 polls).  If more than one poll came from the same firm, I took the last.   This does leave out the 10/23-10/28 CNN/Time polls. 

-The entries marked with a * in the table above were based on less than 5 states and likely to be rather noisy.  R2000 and Quinnipac never polled the same state. 

-Joe Average is the poll gotten by averaging all of the included polls.  Since its effectively a poll with a larger sampler size, it will normally be better performing than an average individual component. 

Quinnipac only polled two states, but they were within half a point in Florida and Pennsylvania.  They were one of only two polls to on average perform better than Joe Average, the other being...Zogby.

One more set of numbers:

For each pollster I took the average of

[(Average Absolute Pollster Error in state)-(Given Pollster's Absolute Error in state)] over all states in which the pollster polled.  Again positive numbers are good.  From top to bottom.

Quinnipac (1.49, 3 states)
Zogby (0.66, 7 states)
PPP (0.49, 15 states)
Research 2000 (0.43, 11 states)
Strategic Vision (0.06, 7 states)
Survey USA (0.05, 14 states)
Rasmussen (-0.29, 19 states)
ARG (-0.43, 14 states)
Mason Dixon (-0.97, 11 states)

This still isn't perfect, and it gives states like Alaska where the pollsters vary widely a disproportionate impact over states where the pollsters are close to each other. 

(EDIT: This is particularly critical in the case of Zogby, who gets a huge bump just from getting Nevada about 5 points closer than anyone else did...)

ARG was really really really terrible before Super Tuesday.  But then after Super Tuesday, they either magically became decent pollsters, or they just started copying other pollsters' numbers.  Also, ARG has always produced much crazier numbers in polls taken several weeks/months before the election, rather than just before the election, and all of these pollster ratings only assess the final polls from each pollster, rather than the earlier ones.

This is a problem that was pointed out on pollster.com recently.  We would ideally like to be able to accurately track the election, not just produce a good poll the day or two before.  Yet we have no way of testing the accuracy of polls taken weeks in advance of the election, because there's no objective number to compare them against.  A lot of the polls show very divergent results early on, but then converge towards the final numbers on election day.

RCP was the real winner in my book, or at least the theory of averaging.  IN was about the only state that was a bit off.  There is no magical single pollster, but when you average them out, you get a pretty good picture of whats going on.

538 did sort of the same thing, but I don't like Silver's arbitrary, Candidate A has a % chance stuff.

I agree fully.

Agree with you that both painted a pretty good picture of the state of the race, but I didn't like RCP's arbitrarily removing certain pollsters and stuff.

Silver's % to Win wasn't arbitrary.  It was based on a statistical operation.  It probably wasn't reflective of the actual chance to win, but that wasn't really the intention.

And, yeah, RCP's punting of R2K was lame.

Well, it was arbitrary in the sense that it weighed different factors according to his, somewhat arbitrary, judgement calls. If I'm thinking of the right site, which model I read about.

Can you give me an example?

He definitely did a few weights I disagree with (his undecided break-down and pollster quality weights were based on the primary -- yuck), but I still think his methodology is better than averaging the last three polls.

And R2K's arbitrary exclusion of Research2000 polls, no matter how many complaints they got, was not just disagreeable, but indefensible.  (not the biggest deal in the world, but still, wtf?)

Pollster > RCP.

Oh, I'm not saying the Atlas model is the best either. Far from it. But my basic issue with this stuff is that we want to know how people will vote. Polls ask EXACTLY that. Mixing polls with other stuff supposedly capturing voter intention is suspect in my opinion. Now, I haven't lived long, though I believe it's a little bit longer than you. ;) But in that life-time my definite experience is that JJ's election rules are actually pretty good: factors not captured by polls seldom exist. I can sympathize with Silver's intentions but polls should be enough in theory and in practice it seems to me that they usually are. So I prefer a somewhat objective weighting together of polls. Of course, bias tends to enter into that as well.

Bottom line is that I like to use a rigid model which has not been subjected to any sort of bias or valuing from the constructor of a model (such as "pollster A is 1.345 times better than pollster B").

I don't have specific examples to offer you. But any kind of weighting would be somewhat arbitrary. If you take Factors A, B, C and say "A is 50%, B is 35% and C is 15%" that is a bit arbitrary. Reading through how he constructed his model and weighted my mental note was "interesting, but seems a bit arbitrary". This was like 6 months ago or something so I don't recall exactly what I was thinking about. I could look it up, but maybe you get my point anyway? Please note that I'm not specifically disagreeing with something, saying "A should be 42.5% instead" merely that I have no idea what weight A should have and I'm suspicious of assigning a weight to it.

I'm not sure whether this entire rant makes sense or anything...if it doesn't I will have to read the damn model again. :(

I think you're misunderstanding what the regression model was for.  The regression model ended up worth less than one crappy poll in most states.  It wasn't for picking up trends the polls didn't pick up. It was for picking up the slack when a state wasn't being polled.  For instance, the assumption was the Dakotas -- being highly similar -- would move together.  So if SD didn't have a poll for a month, but ND had a few, SD would be adjusted to move along with ND.  In some cases, like Indiana, Silver's model was skeptical because the demographics suggested that it shouldn't be close.  But eventually the IN regression came into step, and even when it was out of step, it was the difference of a point and a half at most.  Why?  Because Indiana did have polls, and the model ceded to them.

Silver also provided a raw, trend-adjusted polling model.  That included only subsequent national poll movement (downweighted), pollster quality (see previous caveat), sample size.  True, how much sample size is weighed is arbitrary.  But we do that kind of things in our mind, and it always ends up more arbitrary that way.

The RCP average is really the same thing as the Atlas, a "dumb" average.

Ah, it's all coming back to me now. Point accepted on non-poll factors. But I think was a bit sceptical of the weighting of the polls. I guess it was a general impression of a lot of weights and stuff that didn't seem solid enough. I like to keep as close to raw data as possible.

I half-agree with you.  I could have lived without the pollster quality weights.  But I like the sample size weights, and the time weights.  Those are things that are hard to mentally quantify on-the-fly.  On those alone, I think it was superior to the RCP average.  Purity is not necessarily optimal.  Besides, both present the pure underlying data.  They just calculate it in different ways ("dumb" average vs. maybe-overcomplicated quantified analysis.)

It's a trade-off between reliability and relevance (yeah, I'm taking an accounting course, right now...bleh). Weighting should make it more relevant/accurate but it does force you to make various judgement calls on issues and thus help introduce bias. I guess the bottom line is, to what extent did 538 over-perform RCP?