PPP under fire from Cohn, Silver, etc.

[Landslide Lyndon]

I love these numbers guys but it seems to me they have a little too  much free time in their hands. PPP has been accurate even in races where no other pollster dared to tread, so what's exactly their problem?

[Landslide Lyndon]

I tend to agree with David Nir from DKE:

http://www.dailykos.com/story/2013/09/12/1237429/-Daily-Kos-Elections-Live-Digest-9-12#20130912093008

Cohn's premise is a bit curious, since he begins by acknowledging PPP's accurate track record, but then insists: "Pollsters, though, tend to judge one another based more on methodology than record." That's a bit like saying meteorologists care more about putting together an elegant forecast model than getting tomorrow's weather right. Maybe that's true in certain quarters, but for polling clients, accuracy is undoubtedly paramount.

Cohn's arguments, which are laid out in detail, are not amenable to a quick summary, so you should read the piece yourself to judge whether you agree with the author that "n employing amateurish weighting techniques, withholding controversial methodological details, or deleting questions to avoid scrutiny, the firm does not inspire trust." Cohn claims that his concerns are not "abstract," citing the cases of disgraced pollsters R2K and Strategic Vision. Those firms, however, collapsed because they were accused of making up fake polls, which is an entirely different story. No one questions that PPP makes actual calls to actual humans and puts together actual polls.

[Landslide Lyndon]

lol this thread is great. Legitimate criticism of a pollster? HOW DARE YOU NATE SILVER? WITHOUT PPP YOU WOULD BE sh**t!!!

I don't think anybody would have a problem if Cohn and Silver directed their criticism to crappy pollsters like Mason-Dixon and Rasmussen.
But trying to discredit PPP by ignoring it's record and focusing on technicalities that have little interest for non-statisticians, they both come off as petty. Frankly it looks as if they both have some personal grudge against the company and Tom Jensen.

[Landslide Lyndon]

Dismissing criticism of a "political pollster" because it's "only important to statisticians" is like eating nothing like cotton candy because it tastes good and kills your hunger and nutritional value is "only important to chemists."

You're effectively consuming insubstantial bullsh**t either way.  How satisfying and palatable it is doesn't matter, because it's not composed of anything meaningful.

I'll let Taniel through his twitter account respond to that:

https://twitter.com/Taniel/status/378287794956677120

I guess I'm more interested in how Gallup+Mason Dixon ended up with terrible polls, than in the bad ways @ppppolls ended up with right ones.

[Landslide Lyndon]

Dismissing criticism of a "political pollster" because it's "only important to statisticians" is like eating nothing like cotton candy because it tastes good and kills your hunger and nutritional value is "only important to chemists."

You're effectively consuming insubstantial bullsh**t either way.  How satisfying and palatable it is doesn't matter, because it's not composed of anything meaningful.

I'll let Taniel through his twitter account respond to that:

https://twitter.com/Taniel/status/378287794956677120

I guess I'm more interested in how Gallup+Mason Dixon ended up with terrible polls, than in the bad ways @ppppolls ended up with right ones.

No, you're evidently only interested in any explanation if it lasts less than 45 seconds and doesn't involve understanding how anything complicated works.

You seem to have treated my analogy the same way you're treating the criticism: "meh, I'd prefer something punchier and 140 characters or less."

I disagree with your analogy because eating cotton candy will eventually take a toll on your health.
What exactly is the bad thing that will happen to us or to politics if PPP continues to produce exact polls, despite using a "questionable" methodology?

Also, comparing PPP to fraudsters like Strategic Vision shows to me that these people aren't motivated by scientific integrity but rather that they have an axe to grind.

[Landslide Lyndon]

I disagree with your analogy because eating cotton candy will eventually take a toll on your health.
What exactly is the bad thing that will happen to us or to politics if PPP continues to produce exact polls, despite using a "questionable" methodology?

Because what BRTD is saying.

There is a little difference between people like JJ or the unskwedpolls crowd and PPP.
Those people were proven hilariously wrong. PPP is more often than not right, even in races that nobody else polls or where their results are contradicting the other pollsters.
Just ask non-senator Coakley for that.

You may see people comparing the two, but do you see any of us equating the two?

And it's an unfair and  malicious comparison that amounts into little more than a dog-whistle.
It's like defending the people who compare Obama to Hitler because they are just "comparing", not "equating" the two of them.

[Landslide Lyndon]

I don't know what to tell you at this point.  You continue to repeat "it's different because PPP isn't making polls up, and they're getting reasonable results." 

Isn't that the whole point of polling? Who cares if some company has a scientifically impeccable methodology if this methodology yields wrong numbers?
In sports too there are many examples of coaches who were criticized for their unorthodox tactics, yet they were eventually vindicated when their teams went on to triumph over their "orthodox" competitors.

[Landslide Lyndon]

If you're going to throw out the pretense of "science" from polling you might as well bring back the Literary Digest.

I think Gallup went down that route last year.

[Landslide Lyndon]

Well, now that we've established that PPP is somewhere between Charles Manson and Cory on a scale of villainy, I think it's appropriate to post their answer to one of the criticisms directed to them:

http://www.publicpolicypolling.com/main/2013/09/15-things-ppp-got-right-that-no-one-else-did.html

In the slew of critcism of PPP this week, the most insidious suggestion has been that we 'copy' our results off of other pollsters. The other criticisms are basically differences of opinion about our methodology, but this one goes to a whole different level because it implies seriously unethical behavior on our part.

That attack neglects the fact that we have polled more races that no one else wanted to poll than anyone else in the country over the last 5 years, and generally gotten it right. Here are 15 examples of where we were either the only pollster to look at a race, or the first to pick up on a surprising shift in a contest:

1) In late August of 2010, the Wall Street Journal reported that Mike Castle was an 'overwhelming favorite' in the Delaware Republican Senate primary against Christine O'Donnell. But we went in the field the weekend before the election and found O'Donnell up by a 47/44 margin, and she ended up winning 53/47. Our poll was the only one publicly released in that race.

2) When we went into the field for our final Minnesota poll last year, the most recent publicly released poll on the state's voter ID amendment had found it leading 53/41. That made it shocking when we put out our last poll before the election and found it actually failing by a 51/46 margin. But we were right- in the end it failed 54/46.

3) When there was a special election in California's 36th District in 2011 no one was really sure how it would turn out because only 3 points had separated Democrat Janice Hahn and Republican Craig Huey in the primary, and we weren't that far removed from the 2010 Republican wave election. We polled the race for Daily Kos and found Hahn up 52/44...she won 55/45.

...

[Landslide Lyndon]

From today's DKE Digest:

http://www.dailykos.com/story/2013/09/18/1239030/-Daily-Kos-Elections-Live-Digest-9-18#20130918102223

In an interesting op-ed in The Hill, pollster Mark Mellman takes polling firms to task for failing to distinguish between likely voters and what he calls the "likely electorate," which he considers to be far more important. He specifically criticizes polls from Siena and Marist that showed Eliot Spitzer with big leads in the Democratic primary for New York City comptroller.

By contrast, Mellman says his firm had Scott Stringer, their client, "ahead the entire time" because they assumed that a quarter of registered Democrats would turn out.

...

Mellman doesn't go into any detail about how he constructed his "likely electorate," but Mark Blumenthal suggested on Twitter that it involves "lists and vote history," to which Mellman replied "yep" and intimated that he wasn't about to share what someone else called his "secret sauce."


So let me get this straight. PPP is crucified for by the likes of Cohn and Silver for not being transparent enough, even though they cooperated with them and Cohn had a lengthy correspondence with Tom Jensen about the company's methodology.

But Mellman outright refuses to divulge any details about his own polling methodology and the polling nerds are nowhere to be found crying about "transparency" and "questionable scientific methods".

FTR, I'm with Mellman on that. If you have a "secret sauce" that helps you getting accurate results then why should you make it public and give a helping hand to your flailing competitors?

[Landslide Lyndon]

FTR, I'm with Mellman on that. If you have a "secret sauce" that helps you getting accurate results then why should you make it public and give a helping hand to your flailing competitors?

The problem is that this secret sauce, while apparently allowing PPP to get better topline numbers is also causing their crosstabs to go wacky at times.

Not really. Survey USA was the company where that phenomenon was most frequent.