Nate Silver explains FiveThirtyEight's Senate forecasting model

[Never]

Article

Nate Silver found it necessary to explain the intricacies of the FiveThirtyEight Senate forecasting model to give readers a better understanding of how it works. Interestingly, Silver also saw merit for criticizing Sam Wang's Senate modeling in the process:

You must be logged in to read this quote.

[Slander and/or Libel]

Silver's getting awfully, awfully defensive.

[Bacon King]

So he's calling out a guy for making his error bars too narrow??

[Never]

So he's calling out a guy for making his error bars too narrow??

Silver seems to have a fair point. At least in 2010, Wang apparently structured his model in a way that almost completely ruled out results that actually happened, and it wasn't even a partisan bias on Wang's part from what I can tell, since he overestimated Angle, a Republican, while underestimating the GOP's chances in the House in the same year.

[Senator Cris]

Silver is right.

[Chancellor Tanterterg]

Silver is right.

[°Leprechaun]

Nate Silver pointed out a bias in Rasmussen polls:

http://www.electoral-vote.com/evp2014/Info/rasmussen.html

It is interesting, now, that he gives the Republicans a 53% chance of winning the Senate, which is close enough to 50% to call control of the Senate a tossup. It will be interesting to see how close this election is in key swing states like Iowa and North Carolina, which have been close in Presidential elections as well.

[IceSpear]

Yeah, this criticism sounds pretty accurate to me.

[The Ex-Factor]

So he's calling out a guy for making his error bars too narrow??

Silver seems to have a fair point. At least in 2010, Wang apparently structured his model in a way that almost completely ruled out results that actually happened, and it wasn't even a partisan bias on Wang's part from what I can tell, since he overestimated Angle, a Republican, while underestimating the GOP's chances in the House in the same year.

For what it's worth, Wang's rebuttal is:

You must be logged in to read this quote.

And his riposte is that he predicted all the 2012 Senate races accurately, including North Dakota.:
http://election.princeton.edu/2012/11/06/senate-prediction-final-election-eve/

Personally I'm sympathetic to the argument we can improve on models of merely polling averages but Silver's state fundamentals model strikes me as absurdly complex (10 parameters?!) and very susceptible to overfitting. This comment on his latest blog entry explains it very well:

You must be logged in to read this quote.

[Bacon King]

So he's calling out a guy for making his error bars too narrow??

Silver seems to have a fair point. At least in 2010, Wang apparently structured his model in a way that almost completely ruled out results that actually happened, and it wasn't even a partisan bias on Wang's part from what I can tell, since he overestimated Angle, a Republican, while underestimating the GOP's chances in the House in the same year.

Silver calls out Wang for predicting a two point Angle victory when Silver predicted a three point Angle victory. He calls out Wang's predicted 51 seat GOP House gain when Silver predicted 53. The only difference was that Wang's model only had a standard error of 0.5 points for the Senate race and two seats in the House, while Silver's prediction came with a plus-or-minus thirty seats in the House and a similarly broad range for the Senate election.

Silver's complaint is accurate but he's making a mountain out of a molehill here. Sure, in past elections Wang's model significantly underestimated uncertainty but that's irrelevant because he corrected it after 2012. Nate Silver is literally calling out another election predictor as unreliable by hyping up the fact that the guy used to have a single methodological error that has long since been corrected.

[Antonio the Sixth]

Silver is right.

[Lief 🗽]

Silver is fast becoming a joke.

[The Other Castro]

Silver is fast becoming a joke.

For releasing a full description on how his model works?

[Lief 🗽]

Silver is fast becoming a joke.

For releasing a full description on how his model works?

For his attack on Wang and his general attitude.

[Landslide Lyndon]

Silver is fast becoming a joke.

For releasing a full description on how his model works?

For his attack on Wang and his general attitude.

That's what happens when a nerd is thrust in the spotlight.

[Chancellor Tanterterg]

I like how Nate Silver is suddenly persona non grata around these parts now that he has Republicans retaking the Senate.  Good 'ole Atlas objectivity right there.

[KCDem]

Hopefully Sam Wang shuts down this clown.

[○∙◄☻¥tπ[╪AV┼cVê└]

Hopefully Sam Wang shuts down this clown.

[badgate]

"Nate Silver. Nate Silver Nate Silver. Nate

.
..
..
...
donate now

Silver" - actual DCCC email

[Nichlemn]

Silver is right.

I love how people make a huge deal out of Nate Silver getting a couple of Senate races wrong, but make little of Wang getting a prediction wrong with an implied probability of trillions-to-one against.

It's not like Wang has changed a whole lot, either. His predictions are still overconfident and put too much stock into sparse polling (for instance, predicting Orman as 80% to win in Kansas on the basis of (then) a single poll).

As for fundamentals: they make sense to incorporate when polling is scarce and have a proven track record, as this  The Upshot post shows. And just thinking in an intuitive or Bayesian sense: suppose there was a previously unpolled House race in a 90% Obama district with a relatively scandal-free Democratic incumbent and a minor Republican. Then a poll by a reasonably respectable firm shows the race tied at 40% each for the Democrat and the Republican. How do you rate the Republican's chances? 50%? If you go by a purely polling based model, yes you do. But this is obviously ridiculous. Prior to the poll being released, you would have given the Republican a what, one in a million chance of winning? And this poll alone is going to increase your subjective probability by 500,000x? Of course not. It's very likely that some combination of this poll being an outlier, somehow rigged and/or has undecideds that are likely to heavily break for the Democrat. It's the fundamentals of this district that make us believe this. If instead this has a been an open seat in an evenly partisan district in a neutral year, calling it 50/50 would be much more reasonable.

[KCDem]

Silver is right.

I love how people make a huge deal out of Nate Silver getting a couple of Senate races wrong, but make little of Wang getting a prediction wrong with an implied probability of trillions-to-one against.

It's not like Wang has changed a whole lot, either. His predictions are still overconfident and put too much stock into sparse polling (for instance, predicting Orman as 80% to win in Kansas on the basis of (then) a single poll).

As for fundamentals: they make sense to incorporate when polling is scarce and have a proven track record, as this  The Upshot post shows. And just thinking in an intuitive or Bayesian sense: suppose there was a previously unpolled House race in a 90% Obama district with a relatively scandal-free Democratic incumbent and a minor Republican. Then a poll by a reasonably respectable firm shows the race tied at 40% each for the Democrat and the Republican. How do you rate the Republican's chances? 50%? If you go by a purely polling based model, yes you do. But this is obviously ridiculous. Prior to the poll being released, you would have given the Republican a what, one in a million chance of winning? And this poll alone is going to increase your subjective probability by 500,000x? Of course not. It's very likely that some combination of this poll being an outlier, somehow rigged and/or has undecideds that are likely to heavily break for the Democrat. It's the fundamentals of this district that make us believe this. If instead this has a been an open seat in an evenly partisan district in a neutral year, calling it 50/50 would be much more reasonable.

Just because you can't handle the truth doesn't mean it's not true.

[Lief 🗽]

Let's be honest about what this all really about. The new 538 website has been a colossal failure. It's been widely panned for having superficial and obvious political analysis, relying heavily on cultural fluff pieces that no one wanted or cares about (best burrito in America??), and embarrassingly poor predictions (7-1). Since July, its traffic has been slowly but steadily falling. The site's forecasting model is no longer new or fresh; seemingly every big news site has one now, and they were all released before 538's, spoiling whatever big bump 538 hoped to gain from releasing their model. So Nate Silver is drumming up some controversy in a transparent and desperate attempt to get some publicity and page views.  

[Antonio the Sixth]

Let's be honest about what this all really about. The new 538 website has been a colossal failure. It's been widely panned for having superficial and obvious political analysis, relying heavily on cultural fluff pieces that no one wanted or cares about (best burrito in America??), and embarrassingly poor predictions (7-1). Since July, its traffic has been slowly but steadily falling. The site's forecasting model is no longer new or fresh; seemingly every big news site has one now, and they were all released before 538's, spoiling whatever big bump 538 hoped to gain from releasing their model. So Nate Silver is drumming up some controversy in a transparent and desperate attempt to get some publicity and page views.  

I totally agree that the new 538 is pretty disappointing, but this doesn't change the fact that Silver is right on the statistical issues.

[Chancellor Tanterterg]

Let's be honest about what this all really about. The new 538 website has been a colossal failure. It's been widely panned for having superficial and obvious political analysis, relying heavily on cultural fluff pieces that no one wanted or cares about (best burrito in America??), and embarrassingly poor predictions (7-1). Since July, its traffic has been slowly but steadily falling. The site's forecasting model is no longer new or fresh; seemingly every big news site has one now, and they were all released before 538's, spoiling whatever big bump 538 hoped to gain from releasing their model. So Nate Silver is drumming up some controversy in a transparent and desperate attempt to get some publicity and page views.  

I totally agree that the new 538 is pretty disappointing, but this doesn't change the fact that Silver is right on the statistical issues.

What Antonio said, sadly the new site is junk.

[7,052,770]

538's site is awesome with the sports and other non-political matters.

However it's certainly true that it's probability-based forecast model is no longer fresh or unique, and there's really no way to determine if Silver's model is more accurate than anyone else's -- Silver could be the "winner" this fall without having the best model thanks to luck and coincidence, or he could actually have the best model mathematically but appear to have less success than a less good model -- it would take decades to get a large enough sample size to really know. That's why I really prefer 538's other articles.

Also, even though I'm a Silver fan, I find it really weird how he spent all day picking a fight with that Wang guy.