FiveThirtyEight Update: GOP still slight favorite to win Senate

[Mr. Morden]

Here's my question: Have Senate projections ever overestimated DEMOCRATS? Because I vaguely remember the Republicans being slight favorites to hold the Senate in 2006 (they lost it), Republicans were supposed to hold MN in 2008 (lost), win NV/CO in 2010 (lost), and win ND/MT in 2012 (lost). I can't think of an instance when it was widely projected/opined by the pundits that a Democrat would win, then they didn't.

A majority of pundits had the Democrats barely holding on to the Senate in 2002, but the GOP beat expectations and took over.  That was long enough ago that I don't remember the specific races that people missed.  I think Georgia and Minnesota, with Minnesota obviously being due to a fluke set of circumstances.

[GaussLaw]

Some of you guys are the definition of hacks sometimes, as good as your analyses can be.

http://www.realclearpolitics.com/epolls/2012/senate/nd/north_dakota_senate_berg_vs_heitkamp-3212.html
http://www.realclearpolitics.com/epolls/2012/senate/mt/montana_senate_rehberg_vs_tester-1826.html
Berg was up 5.7 points in the final RCP average.  He was consistently leading in the final string of polls.  ND was a Heitkamp miracle and it would've been impossible to call her the favorite heading up to the election.  

In Montana, Rehberg was slightly leading in the final RCP average.  That, combined with the R nature of Montana, would've made Rehberg slightly favored, as Silver projected.

Surprises happen in elections.  It is true that in 2010 and 2012 the Republicans under-performed on election day in Senate races.  This is no guarantee that such will happen in 2014.  

I definitely concur with Del Tachi's post also.

[IceSpear]

You guys are the definition of hacks sometimes.

http://www.realclearpolitics.com/epolls/2012/senate/nd/north_dakota_senate_berg_vs_heitkamp-3212.html
http://www.realclearpolitics.com/epolls/2012/senate/mt/montana_senate_rehberg_vs_tester-1826.html
Berg was up 5.7 points in the final RCP average.  He was consistently leading in the final string of polls.  ND was a Heitkamp miracle and it would've been impossible to call her the favorite heading up to the election. 

In Montana, Rehberg was slightly leading in the final RCP average.  That, combined with the R nature of Montana, would've made Rehberg slightly favored, as Silver projected.

Surprises happen in elections.  It is true that in 2010 and 2012 the Republicans under-performed on election day in Senate races.  This is no guarantee that such will happen in 2014. 

I definitely concur with Del Tachi's post also.

In that case, wouldn't the problem be the polls consistently underestimating the Democrats as opposed to pundits and prognosticators then? The problem is that these surprises always tend to be concentrated in the same direction (except, apparently, since 2002, quite a while ago). Since it does seem hackish at first glance, I'll do an effortpost to prove my point. Let's consider the last 3 elections.

2012 President, Popular Vote: RCP has Obama +0.7. Final margin: Obama +3.9. A pretty substantial 3.2 Republican bias among the pollsters.

2012 President, Electoral College: RCP has Romney carrying Florida, when Obama carried it. A Republican bias of 1 state.

2012 Senate: RCP has Republicans carrying MT/ND. A Republican bias of 2 seats.

2012 Generic Congressional Ballot: RCP has Republicans +0.2. Final margin: Democrats +1.2. A 1.4 point Republican bias.

2012 Gubernatorial: RCP has Republicans carrying MT. A Republican bias of 1 seat.

2010 Senate: RCP has Republicans winning CO/NV. A Republican bias of 2 seats.

2010 Generic Congressional Ballot: RCP has Republicans + 9.4. Final margin: Republicans +6.8. A 2.6 point Republican bias

2010 Gubernatorial: RCP had Republicans winning CT/IL. A Republican bias of 2 seats.

2008 President, Popular Vote: RCP has Obama +7.6. Final margin: Obama +7.3. An extremely small Democratic bias of 0.3 points.

2008 President, Electoral College: RCP has McCain carrying Indiana and North Carolina when Obama carried them. A Republican bias of 2 states.

2008 Senate: RCP has Republicans carrying MN. A Republican bias of 1 seat.

2008 Generic Congressional Ballot: RCP has Democrats +9.0. Final margin: Democrats +10.7. A Republican bias of 1.7 points.

2008 Gubernatorial: No bias

Assuming random chance, these surprises and misses should be even or very close to even between the parties. In only one of these instances was there a Democratic bias, and it ended up being the smallest of them all. If you don't see a trend here, you're either blind or a hack. Again, I'm not saying this is definitive proof of anything, but it's certainly worth considering, especially when a miss in only one or two seats could determine who ends up controlling the Senate.

We'll see soon enough whether this is just a coincidence or a trend. If this trend does continue to hold in 2014, I would hope people begin to take notice and adjust to it in their models.

[GaussLaw]

You guys are the definition of hacks sometimes.

http://www.realclearpolitics.com/epolls/2012/senate/nd/north_dakota_senate_berg_vs_heitkamp-3212.html
http://www.realclearpolitics.com/epolls/2012/senate/mt/montana_senate_rehberg_vs_tester-1826.html
Berg was up 5.7 points in the final RCP average.  He was consistently leading in the final string of polls.  ND was a Heitkamp miracle and it would've been impossible to call her the favorite heading up to the election. 

In Montana, Rehberg was slightly leading in the final RCP average.  That, combined with the R nature of Montana, would've made Rehberg slightly favored, as Silver projected.

Surprises happen in elections.  It is true that in 2010 and 2012 the Republicans under-performed on election day in Senate races.  This is no guarantee that such will happen in 2014. 

I definitely concur with Del Tachi's post also.

In that case, wouldn't the problem be the polls consistently underestimating the Democrats as opposed to pundits and prognosticators then? The problem is that these surprises always tend to be concentrated in the same direction (except, apparently, since 2002, quite a while ago). Since it does seem hackish at first glance, I'll do an effortpost to prove my point. Let's consider the last 3 elections.

2012 President, Popular Vote: RCP has Obama +0.7. Final margin: Obama +3.9. A pretty substantial 3.2 Republican bias among the pollsters.

2012 President, Electoral College: RCP has Romney carrying Florida, when Obama carried it. A Republican bias of 1 state.

2012 Senate: RCP has Republicans carrying MT/ND. A Republican bias of 2 seats.

2012 Generic Congressional Ballot: RCP has Republicans +0.2. Final margin: Democrats +1.2. A 1.4 point Republican bias.

2012 Gubernatorial: RCP has Republicans carrying MT. A Republican bias of 1 seat.

2010 Senate: RCP has Republicans winning CO/NV. A Republican bias of 2 seats.

2010 Generic Congressional Ballot: RCP has Republicans + 9.4. Final margin: Republicans +6.8. A 2.6 point Republican bias

2010 Gubernatorial: RCP had Republicans winning CT/IL. A Republican bias of 2 seats.

2008 President, Popular Vote: RCP has Obama +7.6. Final margin: Obama +7.3. An extremely small Democratic bias of 0.3 points.

2008 President, Electoral College: RCP has McCain carrying Indiana and North Carolina when Obama carried them. A Republican bias of 2 states.

2008 Senate: RCP has Republicans carrying MN. A Republican bias of 1 seat.

2008 Generic Congressional Ballot: RCP has Democrats +9.0. Final margin: Democrats +10.7. A Republican bias of 1.7 points.

2008 Gubernatorial: No bias

Assuming random chance, these surprises and misses should be even or very close to even between the parties. In only one of these instances was there a Democratic bias, and it ended up being the smallest of them all. If you don't see a trend here, you're either blind or a hack. Again, I'm not saying this is definitive proof of anything, but it's certainly worth considering, especially when a miss in only one or two seats could determine who ends up controlling the Senate.

We'll see soon enough whether this is just a coincidence or a trend. If this trend does continue to hold in 2014, I would hope people begin to take notice and adjust to it in their models.

Fair points for sure.  I would point out that even though there was a GOP bias on the 2010 generic ballot, Silver actually underestimated the number of seats the GOP would pick up. 

Still, the case is by far the strongest in 2012 and 2010, and much, much weaker in 2008.

I would argue that Democrats simply over-performed in 2012 and wave years are tough to measure either way. 

You have some good points, but a few cycles are not sufficient to prove a point like this, especially given the idiosyncratic nature of specific Senate races.

[7,052,770]

Sure, sure. Just like Brazil were favorites to win the World Cup, Nate.

I thought we had all agreed to toss 538 into the dustbin of history after their World Cup debacle.

I know you're just being cute, but Nate Silver's odds for the World Cup turned out to be fantastic, just like his odds for the past few Senate elections.

If he gives something an 8% chance of happening, it better happen around 8% of the time or else his odds suck. The fact that said things do happen around 8% of the time means his odds are good, not bad.


ETA: I did this during the knock-out rounds and didn't go back and finish it, but it wouldn't really change the final result. In fact, his probabilities would look a little bit "better" after he was "wrong" (remember that he is never "right" or "wrong") about the Brazil-Germany game. This is an illustration of how amazingly spot-on his World Cup probabilities were.

[7,052,770]

I love how when Nate Silver predicts every state right in the 2012 election and forecasts an almost inevitable Democratic majority for decades to come he's an Atlas folk hero, but now that he's using the same methodologies to arrive at the conclusion of a GOP Senate come 2015 he's now unreliable or along the lines of any other Sabato or Rothenburg type.

Yeah, it's pretty shameful, but most people have a weird, irrational fear of numbers and little understanding of what probability means. Silver has always been outstanding at both political and sports odds, and I see no reason to think he won't continue to be.

[Keystone Phil]

I love how when Nate Silver predicts every state right in the 2012 election and forecasts an almost inevitable Democratic majority for decades to come he's an Atlas folk hero, but now that he's using the same methodologies to arrive at the conclusion of a GOP Senate come 2015 he's now unreliable or along the lines of any other Sabato or Rothenburg type.

Nice move, Atlas.  Nice move.

And I can't wait to see the waterworks here when the GOP gets to 51 on election night.

This place is flooded with Dem hacks? NO!

[IceSpear]

I love how when Nate Silver predicts every state right in the 2012 election and forecasts an almost inevitable Democratic majority for decades to come he's an Atlas folk hero, but now that he's using the same methodologies to arrive at the conclusion of a GOP Senate come 2015 he's now unreliable or along the lines of any other Sabato or Rothenburg type.

Nice move, Atlas.  Nice move.

And I can't wait to see the waterworks here when the GOP gets to 51 on election night.

This place is flooded with Dem hacks? NO!

Literally nothing could be more hackish than predicting Rick Santorum would win in 2006, except possibly if there were Democrats predicting Blanche Lincoln would win in 2010. Are you really in a position to throw stones here?

Anyway, I was never a Nate Silver cultist. He's good at what he does and I respect his work, but he's not infallible.

[7,052,770]

Anyway, I was never a Nate Silver cultist. He's good at what he does and I respect his work, but he's not infallible.

Who's claiming that he is?

All the Silver fanboys ever claim is that when he says something has an X% chance, it usually happens around X% of the time. All this talk about being "right" and "wrong" comes from people who don't have a good understanding of probabilities.

[IceSpear]

Anyway, I was never a Nate Silver cultist. He's good at what he does and I respect his work, but he's not infallible.

Who's claiming that he is?

All the Silver fanboys ever claim is that when he says something has an X% chance, it usually happens around X% of the time. All this talk about being "right" and "wrong" comes from people who don't have a good understanding of probabilities.

The problem with that logic is it removes all accountability from his results. Say whichever some dude in Oklahoma manages to defeat Inhofe, it would make Silver (and everyone else, obviously) look ridiculous. But then you could simply counter with: "But Silver said there was a 1% chance that would happen, and you'd expect things with a 1% probability to occur sometimes, so Silver was still right!"

There's no way to verify his probabilities since we can't run the same elections over and over in simulations. That's why people only care about his calls, not his probabilities. And he seemed to have no problem accepting the "called all 50 states!" accolades which gave him prominence even though he was "just giving odds".

[7,052,770]

But if 1% probability events don't happen around 1% of the time, then Silver's odds aren't good. Such upsets need to happen 1% of the time. Of course it may be decades before he has enough 99%-1% calls in order to analyze how good his model is at that point (and I'm sure he'll be tweaking the model in the meantime), but so far his Senate probabilities have aligned pretty closely to real life.

He gave Heitkamp an 8% chance of victory, but that's the only winner with an ~8% chance to win, so that all looks about right. Reid and Tester were underdogs in his model too, and there have probably actually been too few upsets by his odds.

As the years go on and we get a larger sample to analyze, it may turn out that his probabilities are actually great, horrible, just OK, or anywhere in between. But right now, all we can say is that his Senate probabilities look at least decent at this point.

And he seemed to have no problem accepting the "called all 50 states!" accolades which gave him prominence even though he was "just giving odds".

Yeah, I was annoyed he didn't correct Jon Stewart when he called him "the guy who correctly called all 50 states" in the Daily Show interview, but it's not like Silver promotes himself like that. I guess Silver just isn't the statistical purist that I am since the misperceived "clean sweep" probably made him hundreds of thousands of dollars.

[Mr. Morden]

Is there a good breakdown somewhere for Silver's election forecasts of what % of the time a 20% event or a 30% or a 40% event ends up happening?  Even with limited elections in which he's made such predictions, there should be enough predictions by now to generate something like that.

And it should be done for various stages in advance of the election.  A model that's good one week before the election isn't necessarily good six months before the election.

[Mr. Morden]

I do recall in years past that there were several Senate races that Silver had as being extremely unlikely to flip (like ~1%) which later became somewhat competitive.  That does seem a bit suspicious, as that's the sort of thing that you would expect to be extremely rare.  Ideally, if you're looking at very early predictions, there should be a lot of uncertainty, and so you wouldn't have many cases where the probabilities are 99%-1%, but those would then become more common as election day approaches, and time runs out for big surprises to happen.

[IceSpear]

But if 1% probability events don't happen around 1% of the time, then Silver's odds aren't good. Such upsets need to happen 1% of the time. Of course it may be decades before he has enough 99%-1% calls in order to analyze how good his model is at that point (and I'm sure he'll be tweaking the model in the meantime), but so far his Senate probabilities have aligned pretty closely to real life.

He gave Heitkamp an 8% chance of victory, but that's the only winner with an ~8% chance to win, so that all looks about right. Reid and Tester were underdogs in his model too, and there have probably actually been too few upsets by his odds.

As the years go on and we get a larger sample to analyze, it may turn out that his probabilities are actually great, horrible, just OK, or anywhere in between. But right now, all we can say is that his Senate probabilities look at least decent at this point.

And he seemed to have no problem accepting the "called all 50 states!" accolades which gave him prominence even though he was "just giving odds".

Yeah, I was annoyed he didn't correct Jon Stewart when he called him "the guy who correctly called all 50 states" in the Daily Show interview, but it's not like Silver promotes himself like that. I guess Silver just isn't the statistical purist that I am since the misperceived "clean sweep" probably made him hundreds of thousands of dollars.

Yeah, there's nothing overtly staggering about his percentages (although he seems to give out 1% too liberally, for example D in OK/R in RI is definitely closer to 0% than 1%, and as you say upsets seem more uncommon than they should be). The troubling thing is that every miss he had was always being too bullish for the Republicans, same as RCP which I documented in my earlier post. This is probably more of a function of the polls themselves though, with turd pollsters like Gravis flooding the zone and cranking out pro-GOP outliers. The aggregators and forecasters are merely victims of these firms, which should be ignored.

[Lief 🗽]

Is there a good breakdown somewhere for Silver's election forecasts of what % of the time a 20% event or a 30% or a 40% event ends up happening?  Even with limited elections in which he's made such predictions, there should be enough predictions by now to generate something like that.

And it should be done for various stages in advance of the election.  A model that's good one week before the election isn't necessarily good six months before the election.

I really wish someone would do that. I think it would be fascinating, especially if his percentages are 'accurate' long before the actual election, because that would be a novel and really useful addition to election prognosticating. It's strange (and IMO makes his entire endeavor suspect) that he hasn't done something like this.

This is probably more of a function of the polls themselves though, with turd pollsters like Gravis flooding the zone and cranking out pro-GOP outliers. The aggregators and forecasters are merely victims of these firms, which should be ignored.

But the entire point of his complex, proprietary model is that it's supposed to counteract that. Otherwise he's no better than a random dude simply averaging all the polls the day before the election.

[7,052,770]

No need for any weird conspiracy theories. All of his prior probs are on the website if you search for them.

I went ahead and compiled 2008. I'll let other people take care of 2010 and 2012:

[Attorney General, LGC Speaker, and Former PPT Dwarven Dragon]

No need for any weird conspiracy theories. All of his prior probs are on the website if you search for them.

I went ahead and compiled 2008. I'll let other people take care of 2010 and 2012:

Merkley had a better chance than Hagan in '08? Begich and Landrieu had no chance at ALL of losing, but Johanns of all people had a minimal chance? Chambliss at almost 90%, and McConnell at over 90%? He did great on choosing the favorite in each race, but the chances of each outcome..not so much.

[○∙◄☻¥tπ[╪AV┼cVê└]

No need for any weird conspiracy theories. All of his prior probs are on the website if you search for them.

I went ahead and compiled 2008. I'll let other people take care of 2010 and 2012:

Merkley had a better chance than Hagan in '08? Begich and Landrieu had no chance at ALL of losing, but Johanns of all people had a minimal chance? Chambliss at almost 90%, and McConnell at over 90%? He did great on choosing the favorite in each race, but the chances of each outcome..not so much.

Begich didn't win by much given that he had a 100% chance, LOL.

[7,052,770]

Merkley had a better chance than Hagan in '08?

Yes, Merkley had a slightly higher lead in the polls in the later stages of 2008, and Oregon is a more Democratic-friendly state than North Carolina.

Begich and Landrieu had no chance at ALL of losing, but Johanns of all people had a minimal chance?

The numbers are rounded to the nearest whole number. "100%" does not necessarily mean exactly 100%, just greater than 99.5%. Functionally, there's very little difference between "100%" and "98%". Even if Silver were exactly correct in his probabilities, it would be decades before we could really see the difference in a 100% race and a 98% race.

The fact that Johanns had a slightly higher chance of getting upset (he did have a strong challenger that year and a slightly smaller lead in the polls than most of the 100%ers) than Landrieu is not really relevant.

Chambliss at almost 90%, and McConnell at over 90%?

Chambliss and McConnell had been leading the polls consistently for months. The leads were small enough that it was conceivable that the Democrat could pull off an upset, but not very likely (around 10%) each.

He did great on choosing the favorite in each race, but the chances of each outcome..not so much.

If you're going to make an assertion like that, you should back it up. What do you think probabilities should have been in each race, and based on what?

Here's a link to 538's 2008 articles: http://fivethirtyeight.com/tag/senate-polls/

Begich didn't win by much given that he had a 100% chance, LOL.

Yeah, you did cherry-pick the one number on the whole board that looks suspect, but you're right. Either this was a super-rare 99th percentile event from Stevens (would should happen, but rarely), or Silver's model blew that one race. Races like this should happen, but if they aren't super-rare, it will call Silver's numbers into question.

To be fair, remember that Stevens was convicted just a couple days before the election and Begich took a 20-point lead in the polls. Apparently it was enough for Silver's model to give Begich more than a 99.5% chance. This is why I'd love to see the model applied into the past to see if he can "predict" those races too.

To his credit, I remember Silver saying shortly after the election that he'll modify the model in the future to be even less trusting of Alaska polling.

[Attorney General, LGC Speaker, and Former PPT Dwarven Dragon]

McConnell and Chambliss were ahead in the polls and won their races, but the close (non-runoff) margin in each suggests that both probabilities should have been slightly lower to be "correct"

Merkley won by much less than Hagan did in '08, so in order for the probability to be "correct", Merkley would have to be below Hagan.

Landrieu's margin was rather small (52.1 to 45.7), while Johanns's margin was significantly larger (57.5 to 40.1). If Silver's probabilties were "correct" they wouldn't have given Johanns a (slightly) lower probability than Landrieu, and Landrieu shouldn't have been at essentially 100%.

Nate Silver's model is professed to be something that can go beyond bad polling and with the use of other factors, create an authentic model with probabilities that look valid after election day. In 2008, this wasn't exactly the case.

[Vosem]

Guys, the percentages predict chances of winning and not margin, so a race with more data will have a higher percentage. Say there's two elections: one in which 100 polls have been conducted that all show the Toast Party ahead by 2%, and one in which 1 poll has been conducted that shows the Toast Party ahead by 20%. The model could well be more certain about victory in the first race, not because it thinks that victory will be more decisive, but because there's more data pointing to that victory.

[7,052,770]

I asked the 538 guys to look at this over Twitter and actually got a response from Harry Enten, who writes political articles for them, so maybe they'll post an analysis of past 538 probs soon.

[7,052,770]

McConnell and Chambliss were ahead in the polls and won their races, but the close (non-runoff) margin in each suggests that both probabilities should have been slightly lower to be "correct"

Merkley won by much less than Hagan did in '08, so in order for the probability to be "correct", Merkley would have to be below Hagan.

Landrieu's margin was rather small (52.1 to 45.7), while Johanns's margin was significantly larger (57.5 to 40.1). If Silver's probabilties were "correct" they wouldn't have given Johanns a (slightly) higher probability than Landrieu, and Landrieu shouldn't have been at essentially 100%.

Nate Silver's model is professed to be something that can go beyond bad polling and with the use of other factors, create an authentic model with probabilities that look valid after election day. In 2008, this wasn't exactly the case.

That's not really how it works though. Just because you won by a larger margin doesn't mean you were more likely to win the day before. Elasticity is an issue. For example, Mississippi was Romney's 17th best state in 2008 by percentage of the vote, but that doesn't mean that if Romney had only won 16 states, he would've lost Mississippi, because Mississippi is a very inelastic state -- most of its citizens are committed to a party regardless of national trends.

Also, you shouldn't assume that the "actual" result is the 50th percentile result. Perhaps one candidate had a great ground game and could have easily gotten 5,000 fewer votes but really couldn't have done much more to get 5,000 more votes.

[Never]

I asked the 538 guys to look at this over Twitter and actually got a response from Harry Enten, who writes political articles for them, so maybe they'll post an analysis of past 538 probs soon.

That's neat! Hopefully this will clear some things up.

[Antonio the Sixth]

I asked the 538 guys to look at this over Twitter and actually got a response from Harry Enten, who writes political articles for them, so maybe they'll post an analysis of past 538 probs soon.

Did he say when they plan to release their real model? It would be nice if they did before November...