PPP under fire from Cohn, Silver, etc.

[RogueBeaver]

In case anyone missed the spat between PPP and Silver, Blumenthal, Cohn and others on methodology arising from that Colorado recall poll, 'tis in the CO recall thread. Here Cohn writes a deep-dive on PPP's methodology.

[Lief 🗽]

Really starting to dislike Nate Silver.

He literally wouldn't have a job without accurate pollsters like PPP.

[windjammer]

If they succesfully the results, I don't see the problems...

[hopper]

The Colorado Recall was a low turnout election so anybody could get those 2 recall elections wrong. Even Sean Trende didn't write about those 2 recall elections pre-recall because he thought the Democrats would win them both. He wrote about them after the elections took place.

[DrScholl]

The margin was exact, the only thing was that they didn't release the poll before the election, which doesn't really have anything to do with methodology.

[Landslide Lyndon]

I love these numbers guys but it seems to me they have a little too  much free time in their hands. PPP has been accurate even in races where no other pollster dared to tread, so what's exactly their problem?

[Alcon]

What exactly is the problem?  Have any of you read the article?  It pretty clearly explains what he perceives the problem to be...

[RogueBeaver]

Cohn follows up, as did Silver on Twitter.

[Landslide Lyndon]

I tend to agree with David Nir from DKE:

http://www.dailykos.com/story/2013/09/12/1237429/-Daily-Kos-Elections-Live-Digest-9-12#20130912093008

Cohn's premise is a bit curious, since he begins by acknowledging PPP's accurate track record, but then insists: "Pollsters, though, tend to judge one another based more on methodology than record." That's a bit like saying meteorologists care more about putting together an elegant forecast model than getting tomorrow's weather right. Maybe that's true in certain quarters, but for polling clients, accuracy is undoubtedly paramount.

Cohn's arguments, which are laid out in detail, are not amenable to a quick summary, so you should read the piece yourself to judge whether you agree with the author that "n employing amateurish weighting techniques, withholding controversial methodological details, or deleting questions to avoid scrutiny, the firm does not inspire trust." Cohn claims that his concerns are not "abstract," citing the cases of disgraced pollsters R2K and Strategic Vision. Those firms, however, collapsed because they were accused of making up fake polls, which is an entirely different story. No one questions that PPP makes actual calls to actual humans and puts together actual polls.

[Lief 🗽]

Yeah, maybe this stuff is important for pollsters and statisticians, but all I and I suspect most people care about is results, and PPP has been consistently accurate and consistently willing to poll races that no one else is (coming up with good results in the process).

And again, Nate Silver criticizing them is a bit rich, since without them his precious model would be based much more on garbage like Mason-Dixon and Gallup.

[Maxwell]

lol this thread is great. Legitimate criticism of a pollster? HOW DARE YOU NATE SILVER? WITHOUT PPP YOU WOULD BE sh**t!!!

[Landslide Lyndon]

lol this thread is great. Legitimate criticism of a pollster? HOW DARE YOU NATE SILVER? WITHOUT PPP YOU WOULD BE sh**t!!!

I don't think anybody would have a problem if Cohn and Silver directed their criticism to crappy pollsters like Mason-Dixon and Rasmussen.
But trying to discredit PPP by ignoring it's record and focusing on technicalities that have little interest for non-statisticians, they both come off as petty. Frankly it looks as if they both have some personal grudge against the company and Tom Jensen.

[Beet]

Really starting to dislike Nate Silver.

He literally wouldn't have a job without accurate pollsters like PPP.

Really? He was a poll aggregator- the absence of one pollster, no matter how good, would not have prevented him from doing what he's doing. I remember him getting started as "poblano" at Daily Kos. If anything he owes his chance of getting noticed more to Markos Moulitsas. Andrew Gelman also seemed to give his work an official imprimatur.

I agree that, from a structural standpoint (as opposed to that of one firm) people like him need quality, public pollsters, and no one is going to pay for this stuff in the future as news companies collapse. But that's a larger issue.

[Oldiesfreak1854]

Finally, someone acknowledges the flaws in PPP's methodology.

[they don't love you like i love you]

Um, no. He didn't attack their methodology. Learn to read.

Silver's right. He's not challenging PPP's accuracy like those poll re-weighting morons or calling them discredited. He's just pointing out that yes disregarding a poll you conduct because you don't like the results is not scientific or correct. And worse of all this will just fill ammo for the critics, every PPP poll will get dismissed by right wing trolls in 2014 with this being the grounds (and perhaps ironically Silver's model as well due its inclusion of PPP.)

Cohn's criticism of the methodology is a bit more open to attack since yeah PPP is accurate despite all that. But I don't see anything wrong with criticizing PPP for this as it doesn't involve their accuracy.

[Oldiesfreak1854]

Um, no. He didn't attack their methodology. Learn to read.

Silver's right. He's not challenging PPP's accuracy like those poll re-weighting morons or calling them discredited. He's just pointing out that yes disregarding a poll you conduct because you don't like the results is not scientific or correct. And worse of all this will just fill ammo for the critics, every PPP poll will get dismissed by right wing trolls in 2014 with this being the grounds (and perhaps ironically Silver's model as well due its inclusion of PPP.)

Cohn's criticism of the methodology is a bit more open to attack since yeah PPP is accurate despite all that. But I don't see anything wrong with criticizing PPP for this as it doesn't involve their accuracy.

I don't know how it's accurate to have Obama leading by 8-10 points in Colorado and Virginia during the summer of 2012 when nearly every other poll had it much closer.

[Snowstalker Mk. II]

Um, no. He didn't attack their methodology. Learn to read.

Silver's right. He's not challenging PPP's accuracy like those poll re-weighting morons or calling them discredited. He's just pointing out that yes disregarding a poll you conduct because you don't like the results is not scientific or correct. And worse of all this will just fill ammo for the critics, every PPP poll will get dismissed by right wing trolls in 2014 with this being the grounds (and perhaps ironically Silver's model as well due its inclusion of PPP.)

Cohn's criticism of the methodology is a bit more open to attack since yeah PPP is accurate despite all that. But I don't see anything wrong with criticizing PPP for this as it doesn't involve their accuracy.

I don't know how it's accurate to have Obama leading by 8-10 points in Colorado and Virginia during the summer of 2012 when nearly every other poll had it much closer.

Because Obama had a strong lead during the summer and all the other pollsters were wrong?

[Small Business Owner of Any Repute]

I won't say anything about PPP's methodology, but their increased insistence on adding "gotcha poll questions" designed to show off how stupid and out of touch GOP voters are is kind of off-putting. It's not something a serious polling firm should be doing to the extent that PPP does it.

[Maxwell]

lol this thread is great. Legitimate criticism of a pollster? HOW DARE YOU NATE SILVER? WITHOUT PPP YOU WOULD BE sh**t!!!

I don't think anybody would have a problem if Cohn and Silver directed their criticism to crappy pollsters like Mason-Dixon and Rasmussen.
But trying to discredit PPP by ignoring it's record and focusing on technicalities that have little interest for non-statisticians, they both come off as petty. Frankly it looks as if they both have some personal grudge against the company and Tom Jensen.

They're statisticians. They have a right to do so. Cry me a finksing river.

[Alcon]

Dismissing criticism of a political pollster because it's "only important to statisticians" is like eating nothing but cotton candy because it tastes good and kills your hunger, and claiming it's reasonable because nutritional value is "only important to chemists."

You're effectively consuming insubstantial bullsh**t either way.  How satisfying and palatable it is doesn't matter, because it's not composed of anything meaningful.

PPP is a mostly sincere-seeming company, but totally deserves to be called out for fundamental logical errors like these.  If you don't agree, maybe you should stick to getting your "analysis" from people yelling RBI stats at each other on SportsCenter.

[Landslide Lyndon]

Dismissing criticism of a "political pollster" because it's "only important to statisticians" is like eating nothing like cotton candy because it tastes good and kills your hunger and nutritional value is "only important to chemists."

You're effectively consuming insubstantial bullsh**t either way.  How satisfying and palatable it is doesn't matter, because it's not composed of anything meaningful.

I'll let Taniel through his twitter account respond to that:

https://twitter.com/Taniel/status/378287794956677120

I guess I'm more interested in how Gallup+Mason Dixon ended up with terrible polls, than in the bad ways @ppppolls ended up with right ones.

[Alcon]

Dismissing criticism of a "political pollster" because it's "only important to statisticians" is like eating nothing like cotton candy because it tastes good and kills your hunger and nutritional value is "only important to chemists."

You're effectively consuming insubstantial bullsh**t either way.  How satisfying and palatable it is doesn't matter, because it's not composed of anything meaningful.

I'll let Taniel through his twitter account respond to that:

https://twitter.com/Taniel/status/378287794956677120

I guess I'm more interested in how Gallup+Mason Dixon ended up with terrible polls, than in the bad ways @ppppolls ended up with right ones.

No, you're evidently only interested in any explanation if it lasts less than 45 seconds and doesn't involve understanding how anything complicated works.

You seem to have treated my analogy the same way you're treating the criticism: "meh, I'd prefer something punchier and 140 characters or less."

[they don't love you like i love you]

Once again Silver is not criticizing PPP for their methodology. He's criticizing them for an action not too far off from what the poll reweighting idiots were doing last year, or J. J.'s Bradley Effect blather and Phil insisting we not listen to the polls and instead listen to his reports of who people told him they were voting for in 2008.

[Landslide Lyndon]

Dismissing criticism of a "political pollster" because it's "only important to statisticians" is like eating nothing like cotton candy because it tastes good and kills your hunger and nutritional value is "only important to chemists."

You're effectively consuming insubstantial bullsh**t either way.  How satisfying and palatable it is doesn't matter, because it's not composed of anything meaningful.

I'll let Taniel through his twitter account respond to that:

https://twitter.com/Taniel/status/378287794956677120

I guess I'm more interested in how Gallup+Mason Dixon ended up with terrible polls, than in the bad ways @ppppolls ended up with right ones.

No, you're evidently only interested in any explanation if it lasts less than 45 seconds and doesn't involve understanding how anything complicated works.

You seem to have treated my analogy the same way you're treating the criticism: "meh, I'd prefer something punchier and 140 characters or less."

I disagree with your analogy because eating cotton candy will eventually take a toll on your health.
What exactly is the bad thing that will happen to us or to politics if PPP continues to produce exact polls, despite using a "questionable" methodology?

Also, comparing PPP to fraudsters like Strategic Vision shows to me that these people aren't motivated by scientific integrity but rather that they have an axe to grind.

[Alcon]

I disagree with your analogy because eating cotton candy will eventually take a toll on your health.
What exactly is the bad thing that will happen to us or to politics if PPP continues to produce exact polls, despite using a "questionable" methodology?

Because what BRTD is saying.

Also, comparing PPP to fraudsters like Strategic Vision shows to me that these people aren't motivated by scientific integrity but rather that they have an axe to grind.

You may see people comparing the two, but do you see any of us equating the two?