PPP vs. Nate Silver and potentially cooked polls

[Queen Mum Inks.LWC]

You must be logged in to read this quote.

http://thefederalist.com/2013/09/18/did-bogus-methodology-enable-nate-silvers-perfect-2012-prediction/

I've always been a fan of PPP and didn't buy into the "They're a hack firm that just wants to help Democrats" crap that a lot of far right Republicans spewed.  But reading this was pretty interesting.  So does anybody know how much of the article is true and how much of it is not (or at least embellished)?

[Beet]

Nate Silver wasnt the most accurate aggregator in 2012; Josh Putnam, Hugh Jackman, and Drew Linzer were all more accurate according to an analysis of Brier scores. He just got the most attention because his NY Times soapbox got him a lot of sh**t from conservatives. And it's not that he had some secret sauce that allowed him to be right on all 50 states; it's that he applied rigorous probability theory to generate precise forecasts. Him getting all 50 states right was the cherry on top. But it's true that if you believe what he was saying the whole campaign, even if he judged the probability right, he still could have ended up "predicting" wrong. A coin that has a 5 percent chance of landing heads will still land heads one in 20 times; it doesn't become impossible. Besides, even if the Florida poll was cooked, what does Nate Silver have to do with it? Garbage in, garbage out. His weighting of the various pollsters seemed to based on objective performance.

To me, the meat of this article is the criticism/accusation thrown at PPP is that they don't scientifically determine the white share of the vote. But the main evidence presented is that the white vote share decreased from 69 to 65 percent. To me it certainly seems unlikely that such a thing could have happened IRL; especially as you would have expected Romney's supporters, predominantly white, to have become more energized/likely to vote after the first debate. But pollsters arent supposed to make any assumptions so... this is circumstantial evidence at best. Is there a rule that says the demographic weightings can't change by more than x percent from poll to poll?

[Lurker]

Nate Silver getting 50/50 states right isn't that impressive at all. You would have gotten exactly the same result - with Florida as the only possible exception - by doing a simple average of all poll, something not exactly requiring great mathematical skill.

For all the complicated prediction models of political scientists, there is no way to get a more accurate result than a simple aggregate of the polls.

A main reason Nate Silver's predictions appear so impressive are all the Republican pundits basically living in fantasy land. Rather than trusting the polls in November, they went with their "gut feeling" with the predictably laughable results (the "respected" Republican analysts of course being the worst of all).

What would be interesting to study is how many of Silver's long-term predictions prove true - does candidates who are given an 80% chance in the year before an election win 80% of the time?

[Oldiesfreak1854]

Nate Silver getting 50/50 states right isn't that impressive at all. You would have gotten exactly the same result - with Florida as the only possible exception - by doing a simple average of all poll, something not exactly requiring great mathematical skill.

For all the complicated prediction models of political scientists, there is no way to get a more accurate result than a simple aggregate of the polls.

Exactly.  Nate Silver is overrated.  The notion that President Obama ever had over an 80% chance of winning is ridiculous.  Granted, he was reelected, but his chances were never that high.

[Lurker]

Nate Silver getting 50/50 states right isn't that impressive at all. You would have gotten exactly the same result - with Florida as the only possible exception - by doing a simple average of all poll, something not exactly requiring great mathematical skill.

For all the complicated prediction models of political scientists, there is no way to get a more accurate result than a simple aggregate of the polls.

Exactly.  Nate Silver is overrated.  The notion that President Obama ever had over an 80% chance of winning is ridiculous.  Granted, he was reelected, but his chances were never that high.

Obama chances were that high on election day - in fact much higher if you'd based your prediction on the state polls. This is were Nate Silver is very useful: by looking at earlier elections/polls he can tell that a candidate with a 2% lead on election day tends to win 85% of the time, for instance.

(That number is just a guess on my part, but hopefully you see the point.)

[Mr. Morden]

Here's the original Nate Cohn takedown of PPP's methodology, btw:

http://www.newrepublic.com/article/114682/ppp-polling-methodology-opaque-flawed

[they don't love you like i love you]

Nate Silver getting 50/50 states right isn't that impressive at all. You would have gotten exactly the same result - with Florida as the only possible exception - by doing a simple average of all poll, something not exactly requiring great mathematical skill.

For all the complicated prediction models of political scientists, there is no way to get a more accurate result than a simple aggregate of the polls.

Exactly.  Nate Silver is overrated.  The notion that President Obama ever had over an 80% chance of winning is ridiculous.  Granted, he was reelected, but his chances were never that high.

I should point out that Nate Silver's rise to fame was based on a statistical model he did during the 2008 primaries that completely ignored polling and was based only on demographics...and was actually more accurate than ANY polling company.

[Lurker]

Nate Silver getting 50/50 states right isn't that impressive at all. You would have gotten exactly the same result - with Florida as the only possible exception - by doing a simple average of all poll, something not exactly requiring great mathematical skill.

For all the complicated prediction models of political scientists, there is no way to get a more accurate result than a simple aggregate of the polls.

Exactly.  Nate Silver is overrated.  The notion that President Obama ever had over an 80% chance of winning is ridiculous.  Granted, he was reelected, but his chances were never that high.

I should point out that Nate Silver's rise to fame was based on a statistical model he did during the 2008 primaries that completely ignored polling and was based only on demographics...and was actually more accurate than ANY polling company.

This quote makes it appear that I'm agreeing with OldiesFreak, which is obviously not the case (other than on the fact that Silver's 50/50, so hailed by everyone, was not that impressive).

As to your assertion that Silver built a statistical model "that completely ignored polling and was based only on demographics" - please back this up with a source. I took a quick look at his 2008 Super Tuesday predictions, and that claim does not seem accurate.
http://www.dailykos.com/story/2008/02/05/450099/-Final-Super-Tuesday-Projection-2-5-08

[Mr. Morden]

Nate Cohn offers more criticism of PPP here:

http://www.newrepublic.com/article/114769/ppp-methodology-results-arent-defense

essentially arguing that it looks like PPP uses ad hoc weighting to get their results closer to the polling average, in cases where other pollsters have already polled the race.

[Gustaf]

Nate Cohn offers more criticism of PPP here:

http://www.newrepublic.com/article/114769/ppp-methodology-results-arent-defense

essentially arguing that it looks like PPP uses ad hoc weighting to get their results closer to the polling average, in cases where other pollsters have already polled the race.

I've had this thought before.

1. Start a fake polling company.
2. Release bogus polls that simply report results close to averages of previously published polls.
3. Be right most of the time (and when you're wrong, everyone else is too).
4. Establish a great reputation as the most accurate pollster.

(Leap of faith)

5. Make money.

[Filuwaúrdjan]

I've had that thought as well. Do you want to make this a joint venture?

[Beet]

Nate Cohn offers more criticism of PPP here:

http://www.newrepublic.com/article/114769/ppp-methodology-results-arent-defense

essentially arguing that it looks like PPP uses ad hoc weighting to get their results closer to the polling average, in cases where other pollsters have already polled the race.

I've had this thought before.

1. Start a fake polling company.
2. Release bogus polls that simply report results close to averages of previously published polls.
3. Be right most of the time (and when you're wrong, everyone else is too).
4. Establish a great reputation as the most accurate pollster.

(Leap of faith)

5. Make money.

This is pretty much how Wall Street works. If everyone's committing the same kind of fraud, then the aggregate results are nobody's "fault."

[Bacon King]

Nate Cohn offers more criticism of PPP here:

http://www.newrepublic.com/article/114769/ppp-methodology-results-arent-defense

essentially arguing that it looks like PPP uses ad hoc weighting to get their results closer to the polling average, in cases where other pollsters have already polled the race.

What I gather from this is that the only difference between PPP's methodology and the hocus-pocus of traditional poll weighting practices is that PPP has the opportunity to correct their numbers into something somewhat sensible whenever they realize they have an outlier. Every pollster has an assumption about how they expect things will/should look, and that certainly comes into play when adjusting demographic ratios, or establishing likely voter screens, or merging their cellphone sample or internet panel with the rest of the poll.

It's not scientifically rigorous, no, but claiming to be so is a sham in the first place when you're dealing with response rates in the single digits.

[Mr. Morden]

Nate Cohn offers more criticism of PPP here:

http://www.newrepublic.com/article/114769/ppp-methodology-results-arent-defense

essentially arguing that it looks like PPP uses ad hoc weighting to get their results closer to the polling average, in cases where other pollsters have already polled the race.

I've had this thought before.

1. Start a fake polling company.
2. Release bogus polls that simply report results close to averages of previously published polls.
3. Be right most of the time (and when you're wrong, everyone else is too).
4. Establish a great reputation as the most accurate pollster.

(Leap of faith)

5. Make money.

You're not the first one to have thought of this:

https://uselectionatlas.org/FORUM/index.php?topic=153727.msg3300429#msg3300429

[Mr. Morden]

A few graphs in that latest Cohn piece jump out at me.  First, PPP has zero house effect, which is at least consistent with the idea that they're fitting towards the polling average from other polls in the final days before the election:



Second, when they do a statewide poll for both a gubernatorial/senatorial race and a presidential race at the same time, the "error" (how far off their poll is from the actual election result) depends on whether the state is a gubernatorial/senatorial battleground or a presidential battleground:



This suggests that they might be juicing the numbers to fit towards the overall polling average in competitive races, at the expense of creating a greater error in the same state's other races.  This phenomenon does not exist in polls from firms that do live interviews:



Finally, this is a graph that's not specific to PPP, but which covers all robopollsters….it shows that robopolls become more accurate in races where there are also polls from live interviewers:



Which, if taken at face value, would suggest that other robopollsters are juicing their results as well, in order to make themselves look like better pollsters than they really are.

[Mr. Morden]

Nate Cohn offers more criticism of PPP here:

http://www.newrepublic.com/article/114769/ppp-methodology-results-arent-defense

essentially arguing that it looks like PPP uses ad hoc weighting to get their results closer to the polling average, in cases where other pollsters have already polled the race.

What I gather from this is that the only difference between PPP's methodology and the hocus-pocus of traditional poll weighting practices is that PPP has the opportunity to correct their numbers into something somewhat sensible whenever they realize they have an outlier. Every pollster has an assumption about how they expect things will/should look, and that certainly comes into play when adjusting demographic ratios, or establishing likely voter screens, or merging their cellphone sample or internet panel with the rest of the poll.

It's not scientifically rigorous, no, but claiming to be so is a sham in the first place when you're dealing with response rates in the single digits.

The point is that the hocus pocus that a pollster performs on the data is something that they should establish (at least internally, to themselves) *before* they conduct the poll.  They shouldn't conduct the poll, take a look at the results, and then fudge the weighting around to make sure that the result "looks right", or lines up with other polls.  If they're fudging the methodology like that, which is what's at least being hinted at with the data that Cohn presents, then they're basically committing fraud.

They could well be a below average pollster that is making themselves look like an above average pollster by juicing the numbers like this.   And if you take the final graph I gave in my last post at face value, then it's possible that other robopollsters are doing this as well.

[FEMA Camp Administrator]

Nate Silver getting 50/50 states right isn't that impressive at all. You would have gotten exactly the same result - with Florida as the only possible exception - by doing a simple average of all poll, something not exactly requiring great mathematical skill.

For all the complicated prediction models of political scientists, there is no way to get a more accurate result than a simple aggregate of the polls.

Exactly.  Nate Silver is overrated.  The notion that President Obama ever had over an 80% chance of winning is ridiculous.  Granted, he was reelected, but his chances were never that high.

Given that Obama won, he had a 100% chance of re-election.

[Oldiesfreak1854]

I think it's also important to remember that even if PPP were the most accurate pollster in 2012, that doesn't mean they're always the most accurate.

[Mr. Morden]

In case it isn't obvious why juicing your results, as is being alleged here, is so bad, let me explain with this simplified thought experiment:

Let's suppose you have some hypothetical pollster called QQQ who's juicing their results.  That is, they conduct real polls, but the polls are pretty mediocre.  They're in fact below average pollsters in cases where they're just relying on their own numbers and letting the chips fall where they may.  However, it turns out that QQQ has been juicing their results in the following (exaggerated....not saying that PPP goes this far) way: Whenever there's a race that has been polled by at least three other pollsters in the last month, QQQ throws out their own numbers, and publishes a "poll" which is just the average of everyone else's polls for that race.

Since the average of all public polls is more often than not a decent predictor of the results, then QQQ's pollster ratings will actually look really good, since in most of the races where they're just copying everyone else, they'll be pretty accurate.  Sure, there will also be races where there aren't enough polls from other pollsters to meet their "at least three in the last month" criteria, so QQQ will just publish their own stuff, which isn't that accurate, but their overall "pollster rating" will look pretty good, because the good stuff outweighs the bad.

So isn't this "OK" for the poll consumer?  If QQQ is giving you numbers that, on average, are a pretty good predictor of the outcome of the race, aren't they "good" pollsters?

No, of course not.  What you, the poll consumer, want from a pollster is an indication of where the race stands.  You want things to work so that, when you see a certain result from a good pollster, it will cause you change your assessment of where the race stands.  But if that pollster is usually just giving you an average of everyone else's polls, then they're not giving you any new information that wouldn't already exist if they hadn't released the "poll".  And then the rest of the time, they're giving you a "real" poll which is of mediocre quality.

So again, if they're doing this without telling you, then they're basically committing fraud.

[they don't love you like i love you]

The states that Silver nailed better than any pollster through his demographic model were North Carolina and Indiana.

[Lurker]

The states that Silver nailed better than any pollster through his demographic model were North Carolina and Indiana.

Your claim that Silver made a statistical model "that completely ignored polling and was based only on demographics" is still wrong though.

[○∙◄☻¥tπ[╪AV┼cVê└]

I've had that thought as well. Do you want to make this a joint venture?

Well, just don't get busted like Research 2000.

Strategic Vision was another one accused of fraud.

[minionofmidas]

Nate Cohn offers more criticism of PPP here:

http://www.newrepublic.com/article/114769/ppp-methodology-results-arent-defense

essentially arguing that it looks like PPP uses ad hoc weighting to get their results closer to the polling average, in cases where other pollsters have already polled the race.

I've had this thought before.

1. Start a fake polling company.
2. Release bogus polls that simply report results close to averages of previously published polls.
3. Be right most of the time (and when you're wrong, everyone else is too).
4. Establish a great reputation as the most accurate pollster.

(Leap of faith)

5. Make money.

If you think there's anybody active in the polling field who's doing anything fundamentally different (though they also call real people), I have some prime Mongolian oceanfront properties to sell to you.

[Mr. Morden]

More Cohn criticism of the methodology of robo-polls:

http://www.nytimes.com/2014/05/29/upshot/when-polling-is-more-like-guessing.html?_r=0

[Amenhotep Bakari-Sellers]

I think that alot of pollsters like Rasmussen, have a GOP house effect. Or sometimes Gravis.

What PPP attempts to do, take out the GOP house effect.