Welcome, Guest. Please login or register.
Did you miss your activation email?
December 15, 2019, 10:32:06 am
News: 2020 Presidential Predictions (General) are now active.

  Atlas Forum
  Presidential Elections - Analysis and Discussion
  Presidential Election Process
  Polling (Moderator: muon2)
  PPP vs. Nate Silver and potentially cooked polls
« previous next »
Pages: [1] 2 Print
Author Topic: PPP vs. Nate Silver and potentially cooked polls  (Read 12953 times)
Queen Mum Inks.LWC
Inks.LWC
YaBB God
*****
Posts: 35,094
United States


Political Matrix
E: 6.19, S: -2.09


Show only this user's posts in this thread
« on: September 18, 2013, 12:03:33 pm »

Quote
You must be logged in to read this quote.

http://thefederalist.com/2013/09/18/did-bogus-methodology-enable-nate-silvers-perfect-2012-prediction/

I've always been a fan of PPP and didn't buy into the "They're a hack firm that just wants to help Democrats" crap that a lot of far right Republicans spewed.  But reading this was pretty interesting.  So does anybody know how much of the article is true and how much of it is not (or at least embellished)?
Logged
Beet
YaBB God
*****
Posts: 24,917


Show only this user's posts in this thread
« Reply #1 on: September 18, 2013, 12:40:27 pm »

Nate Silver wasnt the most accurate aggregator in 2012; Josh Putnam, Hugh Jackman, and Drew Linzer were all more accurate according to an analysis of Brier scores. He just got the most attention because his NY Times soapbox got him a lot of sh**t from conservatives. And it's not that he had some secret sauce that allowed him to be right on all 50 states; it's that he applied rigorous probability theory to generate precise forecasts. Him getting all 50 states right was the cherry on top. But it's true that if you believe what he was saying the whole campaign, even if he judged the probability right, he still could have ended up "predicting" wrong. A coin that has a 5 percent chance of landing heads will still land heads one in 20 times; it doesn't become impossible. Besides, even if the Florida poll was cooked, what does Nate Silver have to do with it? Garbage in, garbage out. His weighting of the various pollsters seemed to based on objective performance.

To me, the meat of this article is the criticism/accusation thrown at PPP is that they don't scientifically determine the white share of the vote. But the main evidence presented is that the white vote share decreased from 69 to 65 percent. To me it certainly seems unlikely that such a thing could have happened IRL; especially as you would have expected Romney's supporters, predominantly white, to have become more energized/likely to vote after the first debate. But pollsters arent supposed to make any assumptions so... this is circumstantial evidence at best. Is there a rule that says the demographic weightings can't change by more than x percent from poll to poll?
Logged
Lurker
YaBB God
*****
Posts: 706
Norway
Show only this user's posts in this thread
« Reply #2 on: September 18, 2013, 01:07:19 pm »

Nate Silver getting 50/50 states right isn't that impressive at all. You would have gotten exactly the same result - with Florida as the only possible exception - by doing a simple average of all poll, something not exactly requiring great mathematical skill.

For all the complicated prediction models of political scientists, there is no way to get a more accurate result than a simple aggregate of the polls.

A main reason Nate Silver's predictions appear so impressive are all the Republican pundits basically living in fantasy land. Rather than trusting the polls in November, they went with their "gut feeling" with the predictably laughable results (the "respected" Republican analysts of course being the worst of all).

What would be interesting to study is how many of Silver's long-term predictions prove true - does candidates who are given an 80% chance in the year before an election win 80% of the time?
Logged
Oldiesfreak1854
YaBB God
*****
Posts: 13,634
United States


WWW Show only this user's posts in this thread
« Reply #3 on: September 18, 2013, 01:18:24 pm »

Nate Silver getting 50/50 states right isn't that impressive at all. You would have gotten exactly the same result - with Florida as the only possible exception - by doing a simple average of all poll, something not exactly requiring great mathematical skill.

For all the complicated prediction models of political scientists, there is no way to get a more accurate result than a simple aggregate of the polls.

Exactly.  Nate Silver is overrated.  The notion that President Obama ever had over an 80% chance of winning is ridiculous.  Granted, he was reelected, but his chances were never that high.
Logged
Lurker
YaBB God
*****
Posts: 706
Norway
Show only this user's posts in this thread
« Reply #4 on: September 18, 2013, 01:40:30 pm »

Nate Silver getting 50/50 states right isn't that impressive at all. You would have gotten exactly the same result - with Florida as the only possible exception - by doing a simple average of all poll, something not exactly requiring great mathematical skill.

For all the complicated prediction models of political scientists, there is no way to get a more accurate result than a simple aggregate of the polls.

Exactly.  Nate Silver is overrated.  The notion that President Obama ever had over an 80% chance of winning is ridiculous.  Granted, he was reelected, but his chances were never that high.

Obama chances were that high on election day - in fact much higher if you'd based your prediction on the state polls. This is were Nate Silver is very useful: by looking at earlier elections/polls he can tell that a candidate with a 2% lead on election day tends to win 85% of the time, for instance.

(That number is just a guess on my part, but hopefully you see the point.)
Logged
Mr. Morden
YaBB God
*****
Posts: 40,358
United States


Show only this user's posts in this thread
« Reply #5 on: September 18, 2013, 08:51:32 pm »

Here's the original Nate Cohn takedown of PPP's methodology, btw:

http://www.newrepublic.com/article/114682/ppp-polling-methodology-opaque-flawed
Logged
Cigarettes & Saints
BRTD
YaBB God
*****
Posts: 88,088
Ukraine


Show only this user's posts in this thread
« Reply #6 on: September 18, 2013, 09:26:15 pm »

Nate Silver getting 50/50 states right isn't that impressive at all. You would have gotten exactly the same result - with Florida as the only possible exception - by doing a simple average of all poll, something not exactly requiring great mathematical skill.

For all the complicated prediction models of political scientists, there is no way to get a more accurate result than a simple aggregate of the polls.

Exactly.  Nate Silver is overrated.  The notion that President Obama ever had over an 80% chance of winning is ridiculous.  Granted, he was reelected, but his chances were never that high.

I should point out that Nate Silver's rise to fame was based on a statistical model he did during the 2008 primaries that completely ignored polling and was based only on demographics...and was actually more accurate than ANY polling company.
Logged
Lurker
YaBB God
*****
Posts: 706
Norway
Show only this user's posts in this thread
« Reply #7 on: September 19, 2013, 06:28:15 am »

Nate Silver getting 50/50 states right isn't that impressive at all. You would have gotten exactly the same result - with Florida as the only possible exception - by doing a simple average of all poll, something not exactly requiring great mathematical skill.

For all the complicated prediction models of political scientists, there is no way to get a more accurate result than a simple aggregate of the polls.

Exactly.  Nate Silver is overrated.  The notion that President Obama ever had over an 80% chance of winning is ridiculous.  Granted, he was reelected, but his chances were never that high.

I should point out that Nate Silver's rise to fame was based on a statistical model he did during the 2008 primaries that completely ignored polling and was based only on demographics...and was actually more accurate than ANY polling company.

This quote makes it appear that I'm agreeing with OldiesFreak, which is obviously not the case (other than on the fact that Silver's 50/50, so hailed by everyone, was not that impressive).

As to your assertion that Silver built a statistical model "that completely ignored polling and was based only on demographics" - please back this up with a source. I took a quick look at his 2008 Super Tuesday predictions, and that claim does not seem accurate.
http://www.dailykos.com/story/2008/02/05/450099/-Final-Super-Tuesday-Projection-2-5-08
Logged
Mr. Morden
YaBB God
*****
Posts: 40,358
United States


Show only this user's posts in this thread
« Reply #8 on: September 20, 2013, 02:48:24 am »

Nate Cohn offers more criticism of PPP here:

http://www.newrepublic.com/article/114769/ppp-methodology-results-arent-defense

essentially arguing that it looks like PPP uses ad hoc weighting to get their results closer to the polling average, in cases where other pollsters have already polled the race.
Logged
Gustaf
Moderators
YaBB God
*****
Posts: 29,570


Political Matrix
E: 0.39, S: -0.70

P

Show only this user's posts in this thread
« Reply #9 on: September 20, 2013, 04:05:05 am »

Nate Cohn offers more criticism of PPP here:

http://www.newrepublic.com/article/114769/ppp-methodology-results-arent-defense

essentially arguing that it looks like PPP uses ad hoc weighting to get their results closer to the polling average, in cases where other pollsters have already polled the race.


I've had this thought before.

1. Start a fake polling company.
2. Release bogus polls that simply report results close to averages of previously published polls.
3. Be right most of the time (and when you're wrong, everyone else is too).
4. Establish a great reputation as the most accurate pollster.

(Leap of faith)

5. Make money.

Tongue
Logged
Filuwaúrdjan
Realpolitik
YaBB God
*****
Posts: 62,798
United Kingdom


Show only this user's posts in this thread
« Reply #10 on: September 20, 2013, 12:12:12 pm »

I've had that thought as well. Do you want to make this a joint venture? Tongue
Logged
Beet
YaBB God
*****
Posts: 24,917


Show only this user's posts in this thread
« Reply #11 on: September 20, 2013, 12:12:38 pm »

Nate Cohn offers more criticism of PPP here:

http://www.newrepublic.com/article/114769/ppp-methodology-results-arent-defense

essentially arguing that it looks like PPP uses ad hoc weighting to get their results closer to the polling average, in cases where other pollsters have already polled the race.


I've had this thought before.

1. Start a fake polling company.
2. Release bogus polls that simply report results close to averages of previously published polls.
3. Be right most of the time (and when you're wrong, everyone else is too).
4. Establish a great reputation as the most accurate pollster.

(Leap of faith)

5. Make money.

Tongue

This is pretty much how Wall Street works. If everyone's committing the same kind of fraud, then the aggregate results are nobody's "fault."
Logged
Bacon King
Atlas Politician
YaBB God
*****
Posts: 18,364
United States


Show only this user's posts in this thread
« Reply #12 on: September 20, 2013, 02:20:03 pm »

Nate Cohn offers more criticism of PPP here:

http://www.newrepublic.com/article/114769/ppp-methodology-results-arent-defense

essentially arguing that it looks like PPP uses ad hoc weighting to get their results closer to the polling average, in cases where other pollsters have already polled the race.


What I gather from this is that the only difference between PPP's methodology and the hocus-pocus of traditional poll weighting practices is that PPP has the opportunity to correct their numbers into something somewhat sensible whenever they realize they have an outlier. Every pollster has an assumption about how they expect things will/should look, and that certainly comes into play when adjusting demographic ratios, or establishing likely voter screens, or merging their cellphone sample or internet panel with the rest of the poll.

It's not scientifically rigorous, no, but claiming to be so is a sham in the first place when you're dealing with response rates in the single digits.
Logged
Mr. Morden
YaBB God
*****
Posts: 40,358
United States


Show only this user's posts in this thread
« Reply #13 on: September 20, 2013, 08:21:08 pm »

Nate Cohn offers more criticism of PPP here:

http://www.newrepublic.com/article/114769/ppp-methodology-results-arent-defense

essentially arguing that it looks like PPP uses ad hoc weighting to get their results closer to the polling average, in cases where other pollsters have already polled the race.


I've had this thought before.

1. Start a fake polling company.
2. Release bogus polls that simply report results close to averages of previously published polls.
3. Be right most of the time (and when you're wrong, everyone else is too).
4. Establish a great reputation as the most accurate pollster.

(Leap of faith)

5. Make money.

Tongue

You're not the first one to have thought of this:

https://uselectionatlas.org/FORUM/index.php?topic=153727.msg3300429#msg3300429
Logged
Mr. Morden
YaBB God
*****
Posts: 40,358
United States


Show only this user's posts in this thread
« Reply #14 on: September 21, 2013, 02:13:34 am »

A few graphs in that latest Cohn piece jump out at me.  First, PPP has zero house effect, which is at least consistent with the idea that they're fitting towards the polling average from other polls in the final days before the election:



Second, when they do a statewide poll for both a gubernatorial/senatorial race and a presidential race at the same time, the "error" (how far off their poll is from the actual election result) depends on whether the state is a gubernatorial/senatorial battleground or a presidential battleground:



This suggests that they might be juicing the numbers to fit towards the overall polling average in competitive races, at the expense of creating a greater error in the same state's other races.  This phenomenon does not exist in polls from firms that do live interviews:



Finally, this is a graph that's not specific to PPP, but which covers all robopollsters….it shows that robopolls become more accurate in races where there are also polls from live interviewers:



Which, if taken at face value, would suggest that other robopollsters are juicing their results as well, in order to make themselves look like better pollsters than they really are.
Logged
Mr. Morden
YaBB God
*****
Posts: 40,358
United States


Show only this user's posts in this thread
« Reply #15 on: September 21, 2013, 02:20:55 am »

Nate Cohn offers more criticism of PPP here:

http://www.newrepublic.com/article/114769/ppp-methodology-results-arent-defense

essentially arguing that it looks like PPP uses ad hoc weighting to get their results closer to the polling average, in cases where other pollsters have already polled the race.


What I gather from this is that the only difference between PPP's methodology and the hocus-pocus of traditional poll weighting practices is that PPP has the opportunity to correct their numbers into something somewhat sensible whenever they realize they have an outlier. Every pollster has an assumption about how they expect things will/should look, and that certainly comes into play when adjusting demographic ratios, or establishing likely voter screens, or merging their cellphone sample or internet panel with the rest of the poll.

It's not scientifically rigorous, no, but claiming to be so is a sham in the first place when you're dealing with response rates in the single digits.

The point is that the hocus pocus that a pollster performs on the data is something that they should establish (at least internally, to themselves) *before* they conduct the poll.  They shouldn't conduct the poll, take a look at the results, and then fudge the weighting around to make sure that the result "looks right", or lines up with other polls.  If they're fudging the methodology like that, which is what's at least being hinted at with the data that Cohn presents, then they're basically committing fraud.

They could well be a below average pollster that is making themselves look like an above average pollster by juicing the numbers like this.   And if you take the final graph I gave in my last post at face value, then it's possible that other robopollsters are doing this as well.
Logged
Cath
Cathcon
YaBB God
*****
Posts: 24,431
United States


Show only this user's posts in this thread
« Reply #16 on: September 21, 2013, 12:04:18 pm »

Nate Silver getting 50/50 states right isn't that impressive at all. You would have gotten exactly the same result - with Florida as the only possible exception - by doing a simple average of all poll, something not exactly requiring great mathematical skill.

For all the complicated prediction models of political scientists, there is no way to get a more accurate result than a simple aggregate of the polls.

Exactly.  Nate Silver is overrated.  The notion that President Obama ever had over an 80% chance of winning is ridiculous.  Granted, he was reelected, but his chances were never that high.

Given that Obama won, he had a 100% chance of re-election.
Logged
Oldiesfreak1854
YaBB God
*****
Posts: 13,634
United States


WWW Show only this user's posts in this thread
« Reply #17 on: September 21, 2013, 07:19:02 pm »

I think it's also important to remember that even if PPP were the most accurate pollster in 2012, that doesn't mean they're always the most accurate.
Logged
Mr. Morden
YaBB God
*****
Posts: 40,358
United States


Show only this user's posts in this thread
« Reply #18 on: September 21, 2013, 11:14:25 pm »

In case it isn't obvious why juicing your results, as is being alleged here, is so bad, let me explain with this simplified thought experiment:

Let's suppose you have some hypothetical pollster called QQQ who's juicing their results.  That is, they conduct real polls, but the polls are pretty mediocre.  They're in fact below average pollsters in cases where they're just relying on their own numbers and letting the chips fall where they may.  However, it turns out that QQQ has been juicing their results in the following (exaggerated....not saying that PPP goes this far) way: Whenever there's a race that has been polled by at least three other pollsters in the last month, QQQ throws out their own numbers, and publishes a "poll" which is just the average of everyone else's polls for that race.

Since the average of all public polls is more often than not a decent predictor of the results, then QQQ's pollster ratings will actually look really good, since in most of the races where they're just copying everyone else, they'll be pretty accurate.  Sure, there will also be races where there aren't enough polls from other pollsters to meet their "at least three in the last month" criteria, so QQQ will just publish their own stuff, which isn't that accurate, but their overall "pollster rating" will look pretty good, because the good stuff outweighs the bad.

So isn't this "OK" for the poll consumer?  If QQQ is giving you numbers that, on average, are a pretty good predictor of the outcome of the race, aren't they "good" pollsters?

No, of course not.  What you, the poll consumer, want from a pollster is an indication of where the race stands.  You want things to work so that, when you see a certain result from a good pollster, it will cause you change your assessment of where the race stands.  But if that pollster is usually just giving you an average of everyone else's polls, then they're not giving you any new information that wouldn't already exist if they hadn't released the "poll".  And then the rest of the time, they're giving you a "real" poll which is of mediocre quality.

So again, if they're doing this without telling you, then they're basically committing fraud.
Logged
Cigarettes & Saints
BRTD
YaBB God
*****
Posts: 88,088
Ukraine


Show only this user's posts in this thread
« Reply #19 on: September 25, 2013, 12:10:07 am »

The states that Silver nailed better than any pollster through his demographic model were North Carolina and Indiana.
Logged
Lurker
YaBB God
*****
Posts: 706
Norway
Show only this user's posts in this thread
« Reply #20 on: September 25, 2013, 03:31:22 am »

The states that Silver nailed better than any pollster through his demographic model were North Carolina and Indiana.

Your claim that Silver made a statistical model "that completely ignored polling and was based only on demographics" is still wrong though.
Logged
○∙◄☻¥tπ[╪AV┼cVê└
jfern
YaBB God
*****
Posts: 46,710


Political Matrix
E: -7.38, S: -8.36

Show only this user's posts in this thread
« Reply #21 on: September 25, 2013, 04:10:00 am »
« Edited: September 25, 2013, 04:17:40 am by ○∙◄☻¥tπ[╪AV┼cVê└ »

I've had that thought as well. Do you want to make this a joint venture? Tongue

Well, just don't get busted like Research 2000.

Strategic Vision was another one accused of fraud.

Logged
minionofmidas - supplemental forum account
Lewis Trondheim
YaBB God
*****
Posts: 58,245
India


Show only this user's posts in this thread
« Reply #22 on: October 06, 2013, 06:28:44 am »

Nate Cohn offers more criticism of PPP here:

http://www.newrepublic.com/article/114769/ppp-methodology-results-arent-defense

essentially arguing that it looks like PPP uses ad hoc weighting to get their results closer to the polling average, in cases where other pollsters have already polled the race.


I've had this thought before.

1. Start a fake polling company.
2. Release bogus polls that simply report results close to averages of previously published polls.
3. Be right most of the time (and when you're wrong, everyone else is too).
4. Establish a great reputation as the most accurate pollster.

(Leap of faith)

5. Make money.

Tongue
If you think there's anybody active in the polling field who's doing anything fundamentally different (though they also call real people), I have some prime Mongolian oceanfront properties to sell to you.
Logged
Mr. Morden
YaBB God
*****
Posts: 40,358
United States


Show only this user's posts in this thread
« Reply #23 on: May 29, 2014, 09:19:29 pm »

More Cohn criticism of the methodology of robo-polls:

http://www.nytimes.com/2014/05/29/upshot/when-polling-is-more-like-guessing.html?_r=0
Logged
Cory Booker
olawakandi
YaBB God
*****
Posts: 29,722
United States


Political Matrix
E: -6.84, S: -0.17

P P P
Show only this user's posts in this thread
« Reply #24 on: July 15, 2015, 05:14:30 pm »

I think that alot of pollsters like Rasmussen, have a GOP house effect. Or sometimes Gravis.

What PPP attempts to do, take out the GOP house effect. 
Logged
Pages: [1] 2 Print 
« previous next »
Jump to:  


Login with username, password and session length
Logout

Terms of Service - DMCA Agent and Policy - Privacy Policy and Cookies

Powered by SMF 1.1.21 | SMF © 2015, Simple Machines

© Dave Leip's Atlas of U.S. Elections, LLC