Fivethirtyeight.com's performances

[Antonio the Sixth]

During the presidential campaign, I consulted this site frequently to follow the trend of the election and I think it's one of the most reliable websites.
But, over that, I think it's more interesting to see fivethirtyeight's margin of error by state ( those where it had a democratic leaning, those where it had a republican one ), and try to explain why.
So, here is my map of the 2008 presidential election results compared with the fivethirtyeight's latest prediction ( November 4, afternoon ). Red states are those where Obama got a better margin of victory than expected, Blue are those where he got a worse one. The legend is as following :
- Grey : within 1%
- Between 1% and 2,5%
- Between 2,5% and 5%
- Between 5% and 7,5%
- Between 7,5% and 10%
- More than 10%

So what does this map show us ? First, FTE's prediction were quite correct, because for only 12 states he was wrong by more than 5 points. Now, I've the impression that FTE underestimated Obama in the states where he won, and overestimated him in those where he lost ( that's rigth for every state excepted Iowa ), maybe due to an excessive ponderation of the polls. Moreover, if you except this fact, I think that FTE's margin of error corresponds to 2004-2008 trends : we can see that in states like Arkansas, Louisiana or Vermont. So, maybe FTE gives too much importance to past elections for its projections.

What do you think about it ?

[Antonio the Sixth]

The 2008 election was never a stable situation. Those who followed the polling trends saw  states like Michigan and North Carolina make wild swings during the summer. By September (immediately after the Republican National Convention) the Presidential nominees were about even in popular votes even though Obama had a huge structural advantage -- a bunch of states, nearly enough to win with a bunch of states in easy reach for Obama.

Poll results were unstable because situations were changing. Places that seemed to get close got the attention of McCain (not too bad) and Palin (dreadful). States that the GOP thought vulnerable (like Michigan, Wisconsin, New Hampshire, and Iowa) suddenly went out of reach. So McCain/Palin went after the states that were close but had to be won -- like Colorado, Virginia, Ohio, and Florida. Palin got overexposed in Colorado and Virginia, and that likely ensured an Obama victory.   

In the end, one thing that was certain but whose timing wasn't -- the economic meltdown -- created an electoral disaster. Most people thought New Mexico and Nevada would be close, and they weren't.

You're right but, as I already said, I used FTE's latest projection, who was done november 4 afternoon, so it should have considered facts that you mentionned. Actually, he did it quite good, but that's anyways interesting to see where it under/overestimated Obama.

[Antonio the Sixth]

Bump. Expect the 2010 Senate edit of 538.com's performances as soon as we have certified results everywhere.

[Antonio the Sixth]

Senate performance was pretty good, I think. House was dreadful until quite late - that model needs some work.

Yeah, indeed. He gave me a fake hope that democrats could retain the House or lose it quite honorably...

The gubernatorial performance was somewhat erratic too (look at South Carolina, Illinois, Minnesota, Maine, Connecticut or RI), but this is mostly due to pollster's errors.

[Antonio the Sixth]

Also, funny to look at its old projections to see how the races evolved.

For example, his december 2009 old model, in which he basically predicts that GOP would take DE, CT and NV and that dems would take MO : he proved to be 100% wrong ! Of course, the 3 rep non-gains were all due to a "candidate effect" : Dodd was expected to run (and lose), and nobody knew who were Sharron Angle and Christine O'Donnel. Also, the general pic moving toward republicans easily explains Missouri.

Now let's look at his first elaborate model on Jan 22. It had democrats losing 7 seats : ND (99% odds), AR (73%), NV (73%), PA (72%), CO (70%), DE (66%) and IL (51%). However, it expected Giannoulias to win in Illinois. It also gave Bayh 63% chances to hold IN and Feingold 84% to hold WI (). On average, it gave dems 54.7 seats, but that was only because of the high uncertainty.

Mar 10 Democrats lose ND (100%), DE (93%), AR (87%), IN (77%), NV (69%), CO (55%) and PA (51%). IL had 67% chances to go dem and WI 78%. 54.9 dem seats in average.

July 18 still had dems losing 7 seats : ND (100%), AR (100%), IN (95%), CO (73%), DE (64%), PA (62%) and NV (58%). Dem odds were 51% in IL and 74% in WI. Dem seats were expected to be 53.4.

All "old" Senate forecasts can be found here.


Also, here is something that makes more sense. It's the retrospective projection made by 538.com, NY Times version. They re-calculated what would have been their projections using their ultimate model since February 2010. Here are their month-by month projections, starting with Feb 1st and ending at Oct 7. The most important thing is that those projections are based on actual candidates : that means you can see how evolved the Delaware race between Coons and O'Donnel since February. The shades go from tossup (lightest) to safe (darkest). Grey States are those for which predictions were not yet available at the time.





And here is the final projection, still available on the website :


As you can see, Nevada and PA were clearly overconfident, but you can blame Mr Rasmussen for that.

[Antonio the Sixth]

Governor
(Emmer, MN)   11.5%
Quinn, IL   18.3%
(Dubie, VT)   20.2%

Emmer and Dubie both trail in the final count. While Emmer has some (very small) chances if he asks for a recount, Shumlin beat Dubie by 1.5 points and will be chosen by the legislature.

You must be logged in to read this quote.

No, you can't.

37% of the database was made of Rasmussen polls. Without them, the races would have appeared far closer.

[Antonio the Sixth]

You know what ? You're right. I'm doing that immediately.

Let's see for Senate :

States with >90% chances to go dem : 9/9 = 100%
States within 75% and 90% chances to go dem :  2/2 = 100%
States within 60% and 75% chances to go dem : none
States within 50% and 60% chances to go dem : none
States within 50% and 60% chances to go rep : none
States within 60% and 75% chances to go rep : 1/3 = 33%
States within 75% and 90% chances to go rep : 0/1 = 0%
States with >90% chances to go rep : 22/22 = 100%

Then for governors :

States with >90% chances to go dem : 6/6 = 100%
States within 75% and 90% chances to go dem : 5/5 = 100%
States within 60% and 75% chances to go dem : 1/1 = 100%
States within 50% and 60% chances to go dem : none
States within 50% and 60% chances to go rep : 1/1 = 100%
States within 60% and 75% chances to go rep : 1/1 = 100%
States within 75% and 90% chances to go rep : 0/1 = 0%
States with >90% chances to go rep : 21/21 = 100%

If you add them up (to have a bigger sample) :

States with >90% chances to go dem : 15/15 = 100%
States within 75% and 90% chances to go dem : 7/7 = 100%
States within 60% and 75% chances to go dem : 1/1 = 100%
States within 50% and 60% chances to go dem : none
States within 50% and 60% chances to go rep : 1/1 = 100%
States within 60% and 75% chances to go rep : 2/4 = 50%
States within 75% and 90% chances to go rep : 0/2 = 0%
States with >90% chances to go rep : 43/43 = 100%

Funny to see how republicans won in only one third of races where they were favored with 60% to 90% odds.

[Antonio the Sixth]

Favorites who were given a 50-75% chance of winning won: 4 out of 6 times, or 67% of the time
Favorites who were given a 75-90% chance of winning won: 7 out of 9 times, or 78% of the time
Favorites who were given a 90-100% chance of winning won: 58 out of 58 times, or 100% of the time

You're right, those numbers look far more consistent. As for previous models, it's a pretty harder work to do, even though something can be done with the Senate maps I've made.

[Antonio the Sixth]

Since the official rsults have now been published everywhere except in Alaska, here is the map showing 538.com's performance for Senatorial races.

The map indicates which party overperformed and by how many points. Grey means the difference between the forecast and the actual results is inferior to 1%. The 30% shade means between 1 and 2.5%, then between 2.5-5, 5-7.5, 7.5-10, and finally over 10%.


A little comparison with 2008 is interesting :

A first conclusion is that 538 performed far worse this time. In 2008, more than half of the States (27/51) were guessed with an error of less than 2.5%, and only a quarter (12/51) were off by more than 5 points. For 2010 Senate elections, less than three fifths (14/36) were in the first case, and over one third (10/36) in the second (not counting Alaska). Are Senate elecitons inherently harder to polls, or were polls particularly poor in this cycle ?

Another interesting thing is that the pattern looks a lot more random this time. Of course, democrats seem to have overperformed more often in democratic States and republicans in republican States, but there are huge exceptions like Alabama or New Hampshire. Last but not the least, there is an important similarity between the two maps : all seem to bie slightly biased against democrats (ie dems did better than expected). It's certainly something to note, though two electoral cycles are not sufficient to draw a conclusion.


PS : For the analysis of gubernatorial races, I'm waiting to get the official result from Maine, Tennessee, Texas and Iowa, which still lack them for some reason.

[Antonio the Sixth]

Ok, now let's finish with the governor map (the key is the same) :

This time the bias seems to lean more toward democrats, and the forecast seems to be even more inaccurate than in the Senate. Wyoming, Nebraska and Tennessee are particularly far off.

[Antonio the Sixth]

I think evaluation of prognosticators needs to take more account of early predictions (as I post about on 538 here). If you only focus on your final forecast, what stops from you posting garbage until shortly before election day?

But races can change dramatically until the election day. So if for example you see a big difference between an early forecast and the final results it doesn't necessarily mean 538 was wrong, just that things have changed.

You must be logged in to read this quote.

Vote fraud.

Go away please.

[Antonio the Sixth]

Since the official rsults have now been published everywhere except in Alaska, here is the map showing 538.com's performance for Senatorial races.

The map indicates which party overperformed and by how many points. Grey means the difference between the forecast and the actual results is inferior to 1%. The 30% shade means between 1 and 2.5%, then between 2.5-5, 5-7.5, 7.5-10, and finally over 10%.

Interesting. It appears that Nate projected most races as closer than they actually were.

Yeah. Actually, it was already the case (even more so) in the 2008 presidentials. Every State where dems were underestimated voted for Obama, and every State where democrats were overestimated voted McCain except Iowa.

[Antonio the Sixth]

I think evaluation of prognosticators needs to take more account of early predictions (as I post about on 538 here). If you only focus on your final forecast, what stops from you posting garbage until shortly before election day?

But races can change dramatically until the election day. So if for example you see a big difference between an early forecast and the final results it doesn't necessarily mean 538 was wrong, just that things have changed.

Sure, you have to take that into account. But it's really just another form of uncertainty that exists in other ways right until Election Day (for instance polling errors). Your 20% predictions two years out or two days out should still be right 20% of the time. The big problem for evaluating predictions, early or late, is the small sample size of elections 538 has covered thus far. 

Indeed, that's the big problem. I think we will be able to give some meaningful evaluation only after 10 election cycles, ie 20 years.

[Antonio the Sixth]

I waited a bit to make sure the results I used were the final ones, but here is finally the Presidential map of over/underperformance compared to 538.com forecasts. Here is it!

I have tweaked the color coding a bit so as to have more shades. The thresholds are 10 (only WV was off by more than 10 points - sigh...), 7.5, 5, and then it's increments of 1 point (in grey States, Nate was off by less than 1 point).

A quite different map from 2008, anyhow. The pattern seems messier, and with a few areas that clearly stand out (appalachia, the inland northwest, CA, AK, the coastal northeast and the Deep South).


Also, here is another interesting map. It says which component of Nate's forecast (red is the polling average, blue is the "fundamentals" model, and green is the final forecast) was the most accurate in predicting the result. It must be noted that the final forecast is an average of the previous two, which means that it can only be more accurate indicator if the two others skew in different directions. That doesn't mean averaging the two factor isn't the best system.

While in many States the fundamentals were a much better predictor of the result in most States, the polling average tends to be more predictive in the swing States - the States where more polls are conducted. Thus, despite appearances, this actually confirms the validity of Nate's system (which gives a greater weigh to the polling average based on the number of polls).


The same maps will be up for the Senate races soon.

[Antonio the Sixth]

If you are disappointed by the polls' performance for the Presidential race, wait until you see the Senate map.

Also here, there is a clear global tendency. It's impressive to see how democrats almost everywhere did better than expected. Wow.


As for the components.

Here polling averages actually did a bit better, though you really find situations of all kind

[Antonio the Sixth]

Also, it's interesting how little correlation there is between the errors between Presidential and Senate races. The two seem almost unconnected.