PPP under fire from Cohn, Silver, etc.

[Landslide Lyndon]

I disagree with your analogy because eating cotton candy will eventually take a toll on your health.
What exactly is the bad thing that will happen to us or to politics if PPP continues to produce exact polls, despite using a "questionable" methodology?

Because what BRTD is saying.

There is a little difference between people like JJ or the unskwedpolls crowd and PPP.
Those people were proven hilariously wrong. PPP is more often than not right, even in races that nobody else polls or where their results are contradicting the other pollsters.
Just ask non-senator Coakley for that.

You may see people comparing the two, but do you see any of us equating the two?

And it's an unfair and  malicious comparison that amounts into little more than a dog-whistle.
It's like defending the people who compare Obama to Hitler because they are just "comparing", not "equating" the two of them.

[Tender Branson]

What are they up to ?

Discredit every pollster until no national pollster is left anymore ?

[Tender Branson]

Wasn't there a national poll earlier this year that PPP ended up never releasing?

Yes, but sometimes poll samples just produce very strange results. For example that 30% of Democrats back the Republican over Hillary Clinton.

That's why they are not releasing it to the public, because it's so off compared with previous polls.

The same happened in the CO recalls: 1/3 Democrats there supported the recall in the poll, while Obama didn't lose more than 5% of Democrats to Romney.

Therefore it's legitimate to hold back the results.

I don't know though why PPP released the poll at all after the recalls ...

[Zioneer]

This is the most boring political slapfight I have ever read about.

[publicunofficial]

If PPP gets good results, then screw tradition. Gallup is the most traditional pollster there is and their polls are slightly better than any man off the street making up numbers.

[Mr. Morden]

Wasn't there a national poll earlier this year that PPP ended up never releasing?

They've actually done that twice now.  There was the one in June that we discussed here:

https://uselectionatlas.org/FORUM/index.php?topic=175579.0

Both Tender Branson and I asked them about this one in blog comments and/or Twitter, but they ignored our questions.

Then last week, they mentioned that they did a national poll the preceeding week but weren't going to release it because "it is kind of dated now":

http://www.publicpolicypolling.com/main/2013/09/north-carolina-q.html

[badgate]

Maybe they polled Syria before opposition hardened and the numbers would look dumb now

[Alcon]

I disagree with your analogy because eating cotton candy will eventually take a toll on your health.
What exactly is the bad thing that will happen to us or to politics if PPP continues to produce exact polls, despite using a "questionable" methodology?

Because what BRTD is saying.

There is a little difference between people like JJ or the unskwedpolls crowd and PPP.
Those people were proven hilariously wrong. PPP is more often than not right, even in races that nobody else polls or where their results are contradicting the other pollsters.
Just ask non-senator Coakley for that.

You may see people comparing the two, but do you see any of us equating the two?

And it's an unfair and  malicious comparison that amounts into little more than a dog-whistle.
It's like defending the people who compare Obama to Hitler because they are just "comparing", not "equating" the two of them.

I don't know what to tell you at this point.  You continue to repeat "it's different because PPP isn't making polls up, and they're getting reasonable results."  My claim is not that they are making polls.  My claim is not that they're getting unreasonable results.  This conversation cannot continue unless you directly address what I'm saying.

Analogies can be dog-whistles, but you can also find false dog-whistles in analogies.  People frequently respond to analogies with "but those two things are different!"  Well, yeah, that's how analogies work.  They involve two different things.  Especially when your point is "who cares?", the analogy is probably going to draw parallels to something obviously worse; that doesn't mean the analogy is meant to show the two things are equally bad.  Here, the parallel is that all of these firms were disingenuous in order to produce the results they wanted.  It's a problem in both cases.  If you can dismiss statistics as extraneous in statistically scientific polling, surely you can ignore extraneous elements of analogies.

I'd just prefer you not ignore the point I'm trying to make.

[Landslide Lyndon]

I don't know what to tell you at this point.  You continue to repeat "it's different because PPP isn't making polls up, and they're getting reasonable results." 

Isn't that the whole point of polling? Who cares if some company has a scientifically impeccable methodology if this methodology yields wrong numbers?
In sports too there are many examples of coaches who were criticized for their unorthodox tactics, yet they were eventually vindicated when their teams went on to triumph over their "orthodox" competitors.

[True Federalist (진정한 연방 주의자)]

I don't know what to tell you at this point.  You continue to repeat "it's different because PPP isn't making polls up, and they're getting reasonable results." 

Isn't that the whole point of polling? Who cares if some company has a scientifically impeccable methodology if this methodology yields wrong numbers?

And how did they know the numbers were "wrong"?  In the case of the Colorado recalls,.it appears they weren't.  It appears that PPP was guilty of the same folly of massaging the data to obtain the desired results as some other companies did, they just got lucky and had their preconceptions be closer to reality than their competition in 2012.  But past performance is no guarantee of future results.

[I spent the winter writing songs about getting better]

PPP arbitrarily tossed a poll because they didn't think the results looked right. How is that different from what J. J. and co. do? Also just like the right wing trolls they were proven wrong later since the actual results did match the poll. They're methodology is irrelevant, that's not what's being attacked.

[Devils30]

There's really little chance PPP is just making up the numbers. No way could they have such a strong track record with random guessing. If Cohn and Silver are right they'll start to get more elections wrong in the future.

[Lurker]

PPP arbitrarily tossed a poll because they didn't think the results looked right. How is that different from what J. J. and co. do? Also just like the right wing trolls they were proven wrong later since the actual results did match the poll. They're methodology is irrelevant, that's not what's being attacked.

This is incorrect. You clearly haven't followed the debate/flame war between Silver/Cohn and PPP.

Only a small part of the discussion is about that single poll, most of the debate is about their general polling and methodology.

[Lief 🗽]

A few things. First of all, acting like any pollster is "scientific" these days with the pathetically low response rates, the rise of no-landline people, etc. is stupid. I think that the old assumptions about what makes a good polling methodology need to be seriously looked at, especially because PPP, despite not doing it the "correct" way, is getting accurate results, often in races where no one else is brave enough to poll, often before anyone else has released any polls. This is not dumb luck and it's not making up numbers. They're clearly very accurate, and at the end of the day, for their clients, for political enthusiasts, for journalists, etc. that's all that matters.

Second, PPP is a private company and they make their living from getting hired by politicians and other organizations to poll races, usually local or congressional races, and usually these polls aren't published. The polls that PPP does make public are essentially their advertisements for the accuracy of the service they provide. They have no obligation to release to us the details of every poll they do. If they do a poll of a race that no one else is doing and get a result that they think is garbage and they think would hurt their reputation (and thus their business) then it makes sense that they would choose not to release it.

[Alcon]

It's not "dumb luck" or "making out numbers."  These are smart people with excellent intuitions about what's probably right.  But that's not scientifically sound, and they weren't transparent about it.  You may like the product they supplied, but it wasn't what they claimed to supply.

[Filuwaúrdjan]

How truly shocking to discover that the polling industry is unethical. Again.

[Beet]

If you're going to throw out the pretense of "science" from polling you might as well bring back the Literary Digest.

[Landslide Lyndon]

If you're going to throw out the pretense of "science" from polling you might as well bring back the Literary Digest.

I think Gallup went down that route last year.

[Marokai Backbeat]

I feel like there are two or three different arguments going on here and I'm not entirely sure I grasp any of them or what everyone's issue with each other is.

[Linus Van Pelt]

To defend PPP based on its supposed accuracy involves a misunderstanding of Cohn's criticism. An average of polls is likely to be more accurate than any given single poll, just as often an average of measures is likely to be more accurate than a single measure. So if indeed PPP is tweaking its sample based on other results, that should, other things being equal, increase the accuracy of the published result. But pollsters should not be doing this inside the polls themselves, before publishing; if everyone did this, we'd have a weird circle of calibration with no-one doing the actual first-order polling.

Suppose I invent a new type of microscope and use the following method to argue that it is more accurate. First, I look through the microscope and take a measurement that's actually grainier and less accurate than existing results. But then - aha! - I, who have read the scientific literature and know the results from other microscopes, fudge the measurements a bit in the direction of my educated sense of how the results should be. I have indeed built a microscope and conducted an experiment with it, and the results I publish might well be very accurate! But that doesn't mean that I have invented a new, more accurate microscope; the microscope itself is less accurate.

Now, just to be clear, I don't think it's been clearly proven that PPP is actually doing this. But some of the patterns Cohn identifies are suspicious.

[7,052,770]

PPP could have avoided all this by simply releasing the poll.  

If it turned out to be wrong, so be it -- what is it, 0.3% of the time in a normal distribution that results will be more than 3 standard deviations from the mean?  That's going to happen every so often.

[Mr. Morden]

Not to pile on PPP too much, but on a completely separate tangent, Tom Jensen practices rather poor netiquette.  Posting a hate email that you've received on your public blog, without retracting the name and email address of the person who sent it?

http://www.publicpolicypolling.com/main/2013/09/crazy-e-mail-of-the-day.html

Yikes.

[Landslide Lyndon]

Well, now that we've established that PPP is somewhere between Charles Manson and Cory on a scale of villainy, I think it's appropriate to post their answer to one of the criticisms directed to them:

http://www.publicpolicypolling.com/main/2013/09/15-things-ppp-got-right-that-no-one-else-did.html

In the slew of critcism of PPP this week, the most insidious suggestion has been that we 'copy' our results off of other pollsters. The other criticisms are basically differences of opinion about our methodology, but this one goes to a whole different level because it implies seriously unethical behavior on our part.

That attack neglects the fact that we have polled more races that no one else wanted to poll than anyone else in the country over the last 5 years, and generally gotten it right. Here are 15 examples of where we were either the only pollster to look at a race, or the first to pick up on a surprising shift in a contest:

1) In late August of 2010, the Wall Street Journal reported that Mike Castle was an 'overwhelming favorite' in the Delaware Republican Senate primary against Christine O'Donnell. But we went in the field the weekend before the election and found O'Donnell up by a 47/44 margin, and she ended up winning 53/47. Our poll was the only one publicly released in that race.

2) When we went into the field for our final Minnesota poll last year, the most recent publicly released poll on the state's voter ID amendment had found it leading 53/41. That made it shocking when we put out our last poll before the election and found it actually failing by a 51/46 margin. But we were right- in the end it failed 54/46.

3) When there was a special election in California's 36th District in 2011 no one was really sure how it would turn out because only 3 points had separated Democrat Janice Hahn and Republican Craig Huey in the primary, and we weren't that far removed from the 2010 Republican wave election. We polled the race for Daily Kos and found Hahn up 52/44...she won 55/45.

...

[minionofmidas]

PPP arbitrarily tossed a poll because they didn't think the results looked right. How is that different from what J. J. and co. do?

In that PPP is a professional polling firm?

[Alcon]

Landslide Lyndon, I don't understand why the fact that they did this behavior arbitrarily tempers the critique.