Nate Silver: Dear Media, Stop Freaking Out About Donald Trump’s Polls

[emailking]

I'm not at all saying it's a dodge. I understand the math, and I understand how probability works. I'm just wondering whether there's value in it to justify the reputation he got as some sort of wunderkind.

Maybe not. His columns are still a good read though, and his book was very enjoyable. He's admitted himself that his models are only a little bit more sophisticated than poll averages. And I think he said there's only 4 or 5 states that are hard to predict (in a general).

For me the value is in his numbers based analysis vs. just analysis, which is a dime a dozen.

[mencken]

Silver's appeal has rested on his basing things on numbers. He's supposed to have done something that allows him to supersede punditry. I very much agree that there is a lot of misunderstanding of statistics out there, and that doesn't help. But if all he's saying is, "I can never be wrong because I never put 100% or 0% odds on anything," then there's not really much value to his brand of analysis. It really comes down to poll averaging, plus a little bit of incorporation of fundamentals.

You can check how good someone is at assigning probabilities by looking at their overall track record.  Do events that they predict to be 80% likely happen 80% of the time?  Do the 90% events happen 90% of the time?, etc.  But you can't pick out a single one of those events and "prove" that the probability assigned was wrong after the fact, because there's no objective check for such a thing.  That doesn't mean that you can't laugh at the assigned probability for being too optimistic or pessimistic, based on the information at hand.  But those assessments are subjective.  That's not a dodge by Silver.  That's just a matter of definition.

I don't have a problem with people laughing at Silver's assessment of the race.  I'm just confused about people making things up about what Silver is predicting (suggesting that "this guy isn't going to win the nomination in the end" means "this guy's support is going to evaporate tomorrow"), and then saying that he's already been "proved wrong" on this because of polls with Trump at 30%.  And again, this isn't just about Silver, but all of the "party decides" people.  They are largely saying that Trump isn't going to win.  Not that he's not going to get a lot of votes.  Let's wait until the votes are counted before declaring them as having been "proved wrong".

Fair enough. But, at least on the issue of Bush and Walker, it seems like a convenient cop-out regarding their respective flame-outs to say "I only gave them 1 in 4 odds each."

[emailking]

Fair enough. But, at least on the issue of Bush and Walker, it seems like a convenient cop-out regarding their respective flame-outs to say "I only gave them 1 in 4 odds each."

And what would you like him to say? Yes I totally blew it on Walker?

Suppose Walker won and someone calls him out on it, since he predicted there was a 75% chance that wouldn't happen. What should he say then?

If he predicts Walker has a 25% chance of winning, that means he thinks Walker won't win. So why should we rag on him when that comes to pass?

[Slander and/or Libel]

Fair enough. But, at least on the issue of Bush and Walker, it seems like a convenient cop-out regarding their respective flame-outs to say "I only gave them 1 in 4 odds each."

And what would you like him to say? Yes I totally blew it on Walker?

Suppose Walker won and someone calls him out on it, since he predicted there was a 75% chance that wouldn't happen. What should he say then?

If he predicts Walker has a 25% chance of winning, that means he thinks Walker won't win. So why should we rag on him when that comes to pass?

But by that logic you could say he predicted that someone other than everyone would win, right? He thought it was more likely that anybody else would win than that any individual candidate would win.

I understand the math, and I understand what he's doing. I just think that argument is a little disingenuous, is all.

[mencken]

Fair enough. But, at least on the issue of Bush and Walker, it seems like a convenient cop-out regarding their respective flame-outs to say "I only gave them 1 in 4 odds each."

And what would you like him to say? Yes I totally blew it on Walker?

Suppose Walker won and someone calls him out on it, since he predicted there was a 75% chance that wouldn't happen. What should he say then?

If he predicts Walker has a 25% chance of winning, that means he thinks Walker won't win. So why should we rag on him when that comes to pass?

He should acknowledge where his probabilities were closer than those you could obtain from a prediction market. He was more prescient regarding Bush and Rubio's chances, and less so on most of the other candidates.

[Lurker]

Silver's appeal has rested on his basing things on numbers. He's supposed to have done something that allows him to supersede punditry. I very much agree that there is a lot of misunderstanding of statistics out there, and that doesn't help. But if all he's saying is, "I can never be wrong because I never put 100% or 0% odds on anything," then there's not really much value to his brand of analysis. It really comes down to poll averaging, plus a little bit of incorporation of fundamentals.

You can check how good someone is at assigning probabilities by looking at their overall track record.  Do events that they predict to be 80% likely happen 80% of the time?  Do the 90% events happen 90% of the time?, etc.  But you can't pick out a single one of those events and "prove" that the probability assigned was wrong after the fact, because there's no objective check for such a thing.  That doesn't mean that you can't laugh at the assigned probability for being too optimistic or pessimistic, based on the information at hand.  But those assessments are subjective.  That's not a dodge by Silver.  That's just a matter of definition.

I don't have a problem with people laughing at Silver's assessment of the race.  I'm just confused about people making things up about what Silver is predicting (suggesting that "this guy isn't going to win the nomination in the end" means "this guy's support is going to evaporate tomorrow"), and then saying that he's already been "proved wrong" on this because of polls with Trump at 30%.  And again, this isn't just about Silver, but all of the "party decides" people.  They are largely saying that Trump isn't going to win.  Not that he's not going to get a lot of votes.  Let's wait until the votes are counted before declaring them as having been "proved wrong".

Has anyone ever done so in Silver's case? (I don't know whether his number of predictions are enough to make this statistically significant, but if so, it would be interesting to see). And it would be particularly interesting to see how accurate they are, say, six months before an election and not just the day before.

And though Silver gets plenty of unfair criticism (and I agree with your points here), that is more than outweighed by the large amount of undeserved praise IMO. After all, it is not that much of an exaggeration to say that he was labeled a genius for basically looking at a polling average.


(Of course, Silver's status was somewhat helped by the fact that the vast majority of GOP pundits and "experts" acted like insane morons regarding the 2012 polls. Anyone with a basic understanding of numbers - who was not a deluded hack - would have looked brilliant in comparison to them.)

[emailking]

He should acknowledge where his probabilities were closer than those you could obtain from a prediction market.

Closer to what?

But by that logic you could say he predicted that someone other than everyone would win, right? He thought it was more likely that anybody else would win than that any individual candidate would win.

I understand the math, and I understand what he's doing. I just think that argument is a little disingenuous, is all.

The answer to your questions is yes. Any of them is unlikely to win it (potentially), even the most likely candidate. I understand the idea of it may not be palatable. You just have to take it for what it is I guess.

Has anyone ever done so in Silver's case? (I don't know whether his number of predictions are enough to make this statistically significant, but if so, it would be interesting to see). And it would be particularly interesting to see how accurate they are, say, six months before an election and not just the day before.

I haven't seen anything rigorous. I've seen random internet commenters say they did some calculations along those lines and found things didn't pan out well. They could have been cherry-picking of course.

It would be interesting if someone were to do it though, because it's really the only way to assess how good his models are or aren't.

[Slander and/or Libel]

Can that even be done in a meaningful manner, though? That is, is there really a big enough sample size? He puts "if the election were held today" percentages on races for months in advance of elections, but those percentages wouldn't be meaningful in this kind of analysis, because the election was not held on that day. The only percentages that'd be meaningful would be the ones right before the election, which are by that point bare poll averages.

Unless I'm missing something, of course.

[Lurker]

Can that even be done in a meaningful manner, though? That is, is there really a big enough sample size? He puts "if the election were held today" percentages on races for months in advance of elections, but those percentages wouldn't be meaningful in this kind of analysis, because the election was not held on that day. The only percentages that'd be meaningful would be the ones right before the election, which are by that point bare poll averages.

Unless I'm missing something, of course.

That is not correct, as far as I can tell. To quote Silver himself, regarding his predictive model for the 2014 Senate elections: "The FiveThirtyEight model is explicitly meant to be a forecast of how the election will turn out on Nov. 4, 2014 — rather than an estimate of what would happen in an election held today."


http://fivethirtyeight.com/features/how-the-fivethirtyeight-senate-forecast-model-works/

[Slander and/or Libel]

Can that even be done in a meaningful manner, though? That is, is there really a big enough sample size? He puts "if the election were held today" percentages on races for months in advance of elections, but those percentages wouldn't be meaningful in this kind of analysis, because the election was not held on that day. The only percentages that'd be meaningful would be the ones right before the election, which are by that point bare poll averages.

Unless I'm missing something, of course.

That is not correct, as far as I can tell. To quote Silver himself, regarding his predictive model for the 2014 Senate elections: "The FiveThirtyEight model is explicitly meant to be a forecast of how the election will turn out on Nov. 4, 2014 — rather than an estimate of what would happen in an election held today."


http://fivethirtyeight.com/features/how-the-fivethirtyeight-senate-forecast-model-works/

Ah, thank you for that clarification. I'd been misremembering.

So theoretically, then, Silver has provided us with REAMS of data to confirm his models, right? He updated them once a day in the runup to the elections, right?

[emailking]

Yeah. The data is there if someone wants to do it. I'm not sure if the senate models were updated daily, but I think it was weekly or maybe a few times a week. Anyway, with several elections now and all those Senate races, there's enough to do it in a statistically significant manner. It would be a lot of work though.

[Mr. Morden]

Yeah. The data is there if someone wants to do it. I'm not sure if the senate models were updated daily, but I think it was weekly or maybe a few times a week. Anyway, with several elections now and all those Senate races, there's enough to do it in a statistically significant manner. It would be a lot of work though.

Someone here did it for just one snapshot of time for one year's Senate races, and found 538's probabilities to be OK.  But it was a very small sample size taken from just one election year.  There's also the fact that the probabilities might be excellent one month before election day but terrible six months before election day, or vice versa.  The accuracy could go up and down depending on how close to an election you are.

One problem I have with the probabilities they have in some past cycles is that there are numerous cases where Silver's model gives a heavy favorite something like a 99% chance of winning early on, but then the race tightens later, and those odds drop substantially.  If you're seeing 1% events turn into 30% events with relative frequency, that would tell me that maybe you're being too quick to hand out 99/1 from the beginning.  That shouldn't happen that often.  If the early polls are only so predictive, then you really shouldn't be seeing too many races rated 99% that early.

[Sprouts Farmers Market ✘]

In memory of Torie, rest his soul, I'd like to bump my favorite relevant thread he created that was not about Hillary Clinton coughing up blood and dying tomorrow. Discuss the article of 2015.

[I spent the winter writing songs about getting better]

Will Nate Silver ever be taken seriously again?

[Gustaf]

I still think the Silver hate on here is a bit overblown. For example, saying you shouldn't freak out over early primary polls is a correct observation.

Silver's thing was always to look at hard data points and their historical predictive power. He's not a political analyst. And all the things he knew pointed towards Trump not winning. He was wrong and I believe he publicly acknowledged this.

[I spent the winter writing songs about getting better]

The reason everyone dumps on him is he's supposed to only analyze the numbers and not provide mainstream pundit style #analysis. But he did the latter this year.

If he had stuck to the same number crunching as in the past and actually pointed out why Trump was not comparable to Michele Bachmann or Herman Cain (which was pretty obvious even before 2015 ended) and just did what he always did and crunched the numbers he'd be looking pretty good right now.

[Gustaf]

I kind of disagree. In primaries there is less and less reliable polling so that approach is a lot less valid. What he did was to look at historical numbers with predictive value - like endorsement points and such. And based on that he was willing to distrust the polls.

Obviously, this was wrong, in hindsight. But I don't think it was as dumb or ludicrous as people make out now.

[Edu]

Hating on Silver is great and all, but it's also interesting how smug and utterly wrong Vosem was about the whole thing. Almost on Dick Morris level.

[Slander and/or Libel]

I kind of disagree. In primaries there is less and less reliable polling so that approach is a lot less valid. What he did was to look at historical numbers with predictive value - like endorsement points and such. And based on that he was willing to distrust the polls.

Obviously, this was wrong, in hindsight. But I don't think it was as dumb or ludicrous as people make out now.

Not being sure means putting big error bars on predictions, not claiming that one outcome is super likely.

[Lief 🗽]

Nate Silver needs to stick to averaging polls: it's all he's good at! Every time he tries any more in depth analysis, he totally embarrasses himself. I don't know what that low energy beta's obsession with Trump is... jealousy? Some sort of sick repressed lust? Either way, I think after this election he's done. ESPN is run by smart guys; they're not going to keep funding his failure of a vanity project.

Wow, BEAUTIFUL prediction! Thank you LIEF!

[Gustaf]

I kind of disagree. In primaries there is less and less reliable polling so that approach is a lot less valid. What he did was to look at historical numbers with predictive value - like endorsement points and such. And based on that he was willing to distrust the polls.

Obviously, this was wrong, in hindsight. But I don't think it was as dumb or ludicrous as people make out now.

Not being sure means putting big error bars on predictions, not claiming that one outcome is super likely.

Did you read the article? It cites a fair bit of data in arguing that polls at that stage has weak predictive power. This is undeniably true. It then says:

"So, could Trump win? We confront two stubborn facts: first, that nobody remotely like Trump has won a major-party nomination in the modern era.4 And second, as is always a problem in analysis of presidential campaigns, we don’t have all that many data points, so unprecedented events can occur with some regularity. For my money, that adds up to Trump’s chances being higher than 0 but (considerably) less than 20 percent. Your mileage may vary. But you probably shouldn’t rely solely on the polls to make your case; it’s still too soon for that."

I don't see that as being insane, dumb, ridiculous or even particularly wrong. People keep saying that Silver shouldn't have tried to be a pundit, but it seems to me that predicting Trump was a lot about punditry. Primaries don't have a lot of reliable data and what we know of historical data all pointed away from Trump. Making a wrong prediction doesn't make someone an idiot, it's the quality of the analysis leading to the prediction that matters.

[HagridOfTheDeep]

I mean, I often wonder if Trump was the 0.01% chance that happened.

[LLR]

I mean, I often wonder if Trump was the 0.01% chance that happened.

I mean, Nate himself basically called Trump the 5% chance the happened, but I mean, yeah...