PPP strikes out again

[CARLHAYDEN]

I remember a few years ago when I pointed out to Vorlon the weirdness of the R2K polls.

Well, PPP used to be pretty good -  then it took the R2K route.

Here's an illustration:

Poll                                 Date          Sample    Republicans      Democrats      Spread
Democracy Corps (D)  4/10 - 4/12   1000 LV         47                     45              R +2                        PPP (D)                         4/7 - 4/10     532 RV         41                     46              D +5             Rasmussen Reports      4/4 - 4/10   3500 LV         44                     38             R + 6
Quinnipiac                    3/22 - 3/28  2069 RV         40                     37              R +3

http://www1.realclearpolitics.com/epolls/other/generic_congressional_vote-2170.html#polls

[Holmes]

There is a polling board.

[All life is a blur of Republicans and meat.]

This hurts my eyes.

[CARLHAYDEN]

There is a polling board.

I cited four different polls on the congressional election preferences.

What part do you not understand?!?

[Dgov]

So?  There were weeks when Ras was vastly different from the rest of the other main pollsters.  It doesn't disqualify them any more than that does.

To put it simply, different pollsters have different methods for polling.  They are all imperfect, and can respond differently to different swings in the national electorate.  Add that on top of just regular old random variation and you can get results like this from time to time.

[CARLHAYDEN]

So?  There were weeks when Ras was vastly different from the rest of the other main pollsters.  It doesn't disqualify them any more than that does.

To put it simply, different pollsters have different methods for polling.  They are all imperfect, and can respond differently to different swings in the national electorate.  Add that on top of just regular old random variation and you can get results like this from time to time.

First, its the extent of the differences which is important (well outside MoE).

Second, its not the first time PPP has been well outside the MoE on this issue.

[Grad Students are the Worst]

Okay, I'm not saying that PPP doesn't appear like an outlier here, but you're being simplistic.

MoE applies to one perfectly random sample.  There are several variables -- time, sampling methodology, LV/RV -- that makes these not only random in the same way, but not directly comparable even if you assume them to be perfectly random in the same way.  You cannot calculate an MoE where sample type and time vary.  I mean, you can presume that they vary minimally enough to apply an MoE, but that's not strictly sound.

Also, even with a perfectly representative sample, the same methodology and no time variance (aka "magical perfect polling land"), 1-in-20 polls is out of MoE even when perfectly conducted (this is the [arbitrary] definition of 95% confidence rate.)  It's not even that you're conducting the best statistical analysis you can within the confines of practical reality.  This is a fancied version of, "this don't look right."

[CARLHAYDEN]

Okay, I'm not saying that PPP doesn't appear like an outlier here, but you're being simplistic.

MoE applies to one perfectly random sample.  There are several variables -- time, sampling methodology, LV/RV -- that makes these not only random in the same way, but not directly comparable even if you assume them to be perfectly random in the same way.  You cannot calculate an MoE where sample type and time vary.  I mean, you can presume that they vary minimally enough to apply an MoE, but that's not strictly sound.

Also, even with a perfectly representative sample, the same methodology and no time variance (aka "magical perfect polling land"), 1-in-20 polls is out of MoE even when perfectly conducted (this is the [arbitrary] definition of 95% confidence rate.)  It's not even that you're conducting the best statistical analysis you can within the confines of practical reality.  This is a fancied version of, "this don't look right."

Let's examine some of your quibbles.
First, while it is true that the PPP poll cited is a Registered Voter poll and the other three cited were Likely Voter polls, the historical difference between the two groups is relatively minor.  So, if PPP had suggested that the Approve/Disapprove was even, given the slightly greater tendency of Registered Voters who do not vote to favor Democrats, that would have been reasonable.  However, PPP indicated a five point advantage for the Democrats, which is, frankly, implausible.

Second, this is not the first instance of PPP producing numbers which are outside those of other survey research firms on the Generic Ballot this year.  In its poll for 2/11-14/11, PPP indicated a four point Democrat preference whereas Democracy Corps in its poll for 2/7-9/11 found a two point Republican advantage and Rasmussen, in its poll for 2/7-13/11 found a Republican advantage of six points.  So, no, it's not a ‘one-off.’

Third, with respect to you assertion that “MoE applies to one perfectly random sample, there are several problems with that assertion. : (a) there’s no such thing as a perfectly random sample. (b) the MoE is based on sample size (I believe I explained this to you once before) not time or type.

Fourth, there is no such thing as a perfectly conducted survey. 

[Grad Students are the Worst]

Okay, I'm not saying that PPP doesn't appear like an outlier here, but you're being simplistic.

MoE applies to one perfectly random sample.  There are several variables -- time, sampling methodology, LV/RV -- that makes these not only random in the same way, but not directly comparable even if you assume them to be perfectly random in the same way.  You cannot calculate an MoE where sample type and time vary.  I mean, you can presume that they vary minimally enough to apply an MoE, but that's not strictly sound.

Also, even with a perfectly representative sample, the same methodology and no time variance (aka "magical perfect polling land"), 1-in-20 polls is out of MoE even when perfectly conducted (this is the [arbitrary] definition of 95% confidence rate.)  It's not even that you're conducting the best statistical analysis you can within the confines of practical reality.  This is a fancied version of, "this don't look right."

Let's examine some of your quibbles.
First, while it is true that the PPP poll cited is a Registered Voter poll and the other three cited were Likely Voter polls, the historical difference between the two groups is relatively minor.  So, if PPP had suggested that the Approve/Disapprove was even, given the slightly greater tendency of Registered Voters who do not vote to favor Democrats, that would have been reasonable.  However, PPP indicated a five point advantage for the Democrats, which is, frankly, implausible.

Second, this is not the first instance of PPP producing numbers which are outside those of other survey research firms on the Generic Ballot this year.  In its poll for 2/11-14/11, PPP indicated a four point Democrat preference whereas Democracy Corps in its poll for 2/7-9/11 found a two point Republican advantage and Rasmussen, in its poll for 2/7-13/11 found a Republican advantage of six points.  So, no, it's not a ‘one-off.’

Third, with respect to you assertion that “MoE applies to one perfectly random sample, there are several problems with that assertion. : (a) there’s no such thing as a perfectly random sample. (b) the MoE is based on sample size (I believe I explained this to you once before) not time or type.

Fourth, there is no such thing as a perfectly conducted survey.  

Right, and none of this contradicts anything I said, does it?  Although you are wrong that there is no perfectly representative sample:  There is no perfectly representative sample in large-scale opinion polling, but there is elsewhere.

(MoE is based on sample size, but my complaint was that you were using MoE where other variables -- time, poll type, and sampling methodology -- vary, and presuming they have no effect.  If you're arguing this is formally sound, let me know, but otherwise you're not disagreeing with what I said.)

[CARLHAYDEN]

Okay, I'm not saying that PPP doesn't appear like an outlier here, but you're being simplistic.

MoE applies to one perfectly random sample.  There are several variables -- time, sampling methodology, LV/RV -- that makes these not only random in the same way, but not directly comparable even if you assume them to be perfectly random in the same way.  You cannot calculate an MoE where sample type and time vary.  I mean, you can presume that they vary minimally enough to apply an MoE, but that's not strictly sound.

Also, even with a perfectly representative sample, the same methodology and no time variance (aka "magical perfect polling land"), 1-in-20 polls is out of MoE even when perfectly conducted (this is the [arbitrary] definition of 95% confidence rate.)  It's not even that you're conducting the best statistical analysis you can within the confines of practical reality.  This is a fancied version of, "this don't look right."

Let's examine some of your quibbles.
First, while it is true that the PPP poll cited is a Registered Voter poll and the other three cited were Likely Voter polls, the historical difference between the two groups is relatively minor.  So, if PPP had suggested that the Approve/Disapprove was even, given the slightly greater tendency of Registered Voters who do not vote to favor Democrats, that would have been reasonable.  However, PPP indicated a five point advantage for the Democrats, which is, frankly, implausible.

Second, this is not the first instance of PPP producing numbers which are outside those of other survey research firms on the Generic Ballot this year.  In its poll for 2/11-14/11, PPP indicated a four point Democrat preference whereas Democracy Corps in its poll for 2/7-9/11 found a two point Republican advantage and Rasmussen, in its poll for 2/7-13/11 found a Republican advantage of six points.  So, no, it's not a ‘one-off.’

Third, with respect to you assertion that “MoE applies to one perfectly random sample, there are several problems with that assertion. : (a) there’s no such thing as a perfectly random sample. (b) the MoE is based on sample size (I believe I explained this to you once before) not time or type.

Fourth, there is no such thing as a perfectly conducted survey.  

Right, and none of this contradicts anything I said, does it?  Although you are wrong that there is no perfectly representative sample:  There is no perfectly representative sample in large-scale opinion polling, but there is elsewhere.

(MoE is based on sample size, but my complaint was that you were using MoE where other variables -- time, poll type, and sampling methodology -- vary, and presuming they have no effect.  If you're arguing this is formally sound, let me know, but otherwise you're not disagreeing with what I said.)

First, there is no perfectly represenative sample in the real world.

Second, if you want to get into sample size, it must be admitted that the PPP sample is rather small.  However, while that might explain a rather significance difference in on particular poll, it does NOT explain a consistent tendency to lean in one particular direction (and yes, I provided an example to slow it was not a 'one off,' example, as you implied).

Third, what you said and what you know say keeps changing.  I have already addressed the poll type matter in my prior post, which you are ignoring.

[Tender Branson]

There is a polling board.

I cited four different polls on the congressional election preferences.

What part do you not understand?!?

The place of these polls is valid here, but it doesn't mean anything at this point because support for both sides is very fluid right now and support won't be "hard" for at least another year or so.

So it's just MoE play ...

[Grad Students are the Worst]

First, there is no perfectly represenative sample in the real world.

Sure there is.  Not when it comes to a national poll, but they exist:  A mandatory-participation poll, or true census, of a sample generated from a full population list by random sample, would be perfectly representative.  Watch your precision, friend.

Second, if you want to get into sample size, it must be admitted that the PPP sample is rather small.  However, while that might explain a rather significance difference in on particular poll, it does NOT explain a consistent tendency to lean in one particular direction (and yes, I provided an example to slow it was not a 'one off,' example, as you implied).

I never complained you didn't account for the sample size.  You calculated MoE -- obviously you did.

Since I have not used the phrase "one-off," I don't know what you inferred, so I can't say if you are mistaken or not.

Third, what you said and what you know say keeps changing.  I have already addressed the poll type matter in my prior post, which you are ignoring.

That sentence makes no grammatical sense, but where has my claim changed?  My original claim was that your analysis was simplistic.  Now, you've conceded that variables exist (time, poll type, etc.) but just presume they're not significant enough to justify not just applying a standard MoE analysis.  You also ignored my 1-in-20 complaint; have you even demonstrated that PPP produces objectionable polls more often than that?  My complaint has consistently been that your analysis is based on intuition and laziness, and you'll have to let me know where that changed!

[CARLHAYDEN]

First, there is no perfectly Representative sample in the real world.

Sure there is.  Not when it comes to a national poll, but they exist:  A mandatory-participation poll, or true census, of a sample generated from a full population list by random sample, would be perfectly representative.  Watch your precision, friend.

Second, if you want to get into sample size, it must be admitted that the PPP sample is rather small.  However, while that might explain a rather significance difference in on particular poll, it does NOT explain a consistent tendency to lean in one particular direction (and yes, I provided an example to slow it was not a 'one off,' example, as you implied).

I never complained you didn't account for the sample size.  You calculated MoE -- obviously you did.

Since I have not used the phrase "one-off," I don't know what you inferred, so I can't say if you are mistaken or not.

Third, what you said and what you know say keeps changing.  I have already addressed the poll type matter in my prior post, which you are ignoring.

That sentence makes no grammatical sense, but where has my claim changed?  My original claim was that your analysis was simplistic.  Now, you've conceded that variables exist (time, poll type, etc.) but just presume they're not significant enough to justify not just applying a standard MoE analysis.  You also ignored my 1-in-20 complaint; have you even demonstrated that PPP produces objectionable polls more often than that?  My complaint has consistently been that your analysis is based on intuition and laziness, and you'll have to let me know where that changed!

Alcon,

Are you really so dense?

First, as the people who work in the Census Bureau will tell you, their "survey" is imperfect.  That you believe in perfection in surveys in the real world indicates you have absolutely no knowledge of how they work!

Second, the term "one-off" refers to your allegation that the difference between the surveys cited could be accounted for by the 1 in 20 change that a particular survey would have an error rate larger than the MoE.  However, as I noted (to put it in non-technical language), earlier this year PPP also had another case where it produced a pro-Democrat poll contradicted by other contemporaneous polls.

Third, you seem to have a problem with the English language. I noted that I had previously addressed the matter of poll type, I have not conceded to any of your allegations.

Fourth, if you look at the poll dates I cited, you will find they are contemporaneous, so your allegations of differences in dates is, well, pathetic.

[Grad Students are the Worst]

First, as the people who work in the Census Bureau will tell you, their "survey" is imperfect.  That you believe in perfection in surveys in the real world indicates you have absolutely no knowledge of how they work!

A census, not the Census.  "Census" is a statistics term for a sampling of an entire population -- it's what the U.S. Census attempts to do.  There are small populations for which censuses are possible, and there are larger populations for which perfectly random samples are possible.

Second, the term "one-off" refers to your allegation that the difference between the surveys cited could be accounted for by the 1 in 20 change that a particular survey would have an error rate larger than the MoE.  However, as I noted (to put it in non-technical language), earlier this year PPP also had another case where it produced a pro-Democrat poll contradicted by other contemporaneous polls.

Even ignoring that 1-in-20 is the rate with a perfectly representative sampling system (which you admit is functionally impossible), how about you tell us what -- considering the number of polls PPP produces -- is the chance that two of x polls will be biased Democratic outside the MoE, in a purely random sample.  Do you know the answer?  If not, this is a lazy, intuitive analysis.

Third, you seem to have a problem with the English language. I noted that I had previously addressed the matter of poll type, I have not conceded to any of your allegations.

Fourth, if you look at the poll dates I cited, you will find they are contemporaneous, so your allegations of differences in dates is, well, pathetic.

Are you arguing that the time variable can be assumed completely controlled for?  Be precise with your assertions here.

[CARLHAYDEN]

First, as the people who work in the Census Bureau will tell you, their "survey" is imperfect.  That you believe in perfection in surveys in the real world indicates you have absolutely no knowledge of how they work!

A census, not the Census.  "Census" is a statistics term for a sampling of an entire population -- it's what the U.S. Census attempts to do.  There are small populations for which censuses are possible, and there are larger populations for which perfectly random samples are possible.

Second, the term "one-off" refers to your allegation that the difference between the surveys cited could be accounted for by the 1 in 20 change that a particular survey would have an error rate larger than the MoE.  However, as I noted (to put it in non-technical language), earlier this year PPP also had another case where it produced a pro-Democrat poll contradicted by other contemporaneous polls.

Even ignoring that 1-in-20 is the rate with a perfectly representative sampling system (which you admit is functionally impossible), how about you tell us what -- considering the number of polls PPP produces -- is the chance that two of x polls will be biased Democratic outside the MoE, in a purely random sample.  Do you know the answer?  If not, this is a lazy, intuitive analysis.

Third, you seem to have a problem with the English language. I noted that I had previously addressed the matter of poll type, I have not conceded to any of your allegations.

Fourth, if you look at the poll dates I cited, you will find they are contemporaneous, so your allegations of differences in dates is, well, pathetic.

Are you arguing that the time variable can be assumed completely controlled for?  Be precise with your assertions here.

First, so your discussion of a "census" is a theoretical rather than practical one.  Hmm.

Second, you really need to drop your obsession with perfection.  It doesn't exist in the real world.  Likewise, there are no "purely random samples" in the real world.  Go talk some people actually working in the industry.  They will tell you that telephone surveys are defective (very low completion rates).  So, your question includes a number of assumptions which are simply, incorrect.

Third, the "time variable" is so minimal as to be inconsequential.  What we have in the examples I provided are surveys both closely before and after the dates of the PPP surveys.  And no, perfection is an illusion.

[Grad Students are the Worst]

First, so your discussion of a "census" is a theoretical rather than practical one.  Hmm.

It's a statistics term, and it's completely possible to take a census in real life.  For instance, a truly mandatory survey, or a survey in which membership in the population is contingent on participation in the survey (like, for an organization.)

As in, something that can happen in the "real world."

Second, you really need to drop your obsession with perfection.  It doesn't exist in the real world.  Likewise, there are no "purely random samples" in the real world.  Go talk some people actually working in the industry.  They will tell you that telephone surveys are defective (very low completion rates).  So, your question includes a number of assumptions which are simply, incorrect.

By "the real world," do you mean "the real world of political opinion polling"?  Because that's not what I've been arguing against, because it's not actually what you said.  Odd how you complain when people argue against reasonable inferences about what you say, and then also complain when they argue against a literal interpretation of your words.

Third, the "time variable" is so minimal as to be inconsequential.  What we have in the examples I provided are surveys both closely before and after the dates of the PPP surveys.  And no, perfection is an illusion.

"Perfection is an illusion" does not mean all known imperfection should be "adjusted" to be treated as perfect, but OK.

Care to respond to the part about you failing to account whether PPP's two strike-outs (or x strike-outs) were statistically significant even in a perfect world (i.e., even occur in more than 20 polls)?  Have you even calculated that?

[CARLHAYDEN]

First, so your discussion of a "census" is a theoretical rather than practical one.  Hmm.

It's a statistics term, and it's completely possible to take a census in real life.  For instance, a truly mandatory survey, or a survey in which membership in the population is contingent on participation in the survey (like, for an organization.)

As in, something that can happen in the "real world."

Second, you really need to drop your obsession with perfection.  It doesn't exist in the real world.  Likewise, there are no "purely random samples" in the real world.  Go talk some people actually working in the industry.  They will tell you that telephone surveys are defective (very low completion rates).  So, your question includes a number of assumptions which are simply, incorrect.

By "the real world," do you mean "the real world of political opinion polling"?  Because that's not what I've been arguing against, because it's not actually what you said.  Odd how you complain when people argue against reasonable inferences about what you say, and then also complain when they argue against a literal interpretation of your words.

Third, the "time variable" is so minimal as to be inconsequential.  What we have in the examples I provided are surveys both closely before and after the dates of the PPP surveys.  And no, perfection is an illusion.

"Perfection is an illusion" does not mean all known imperfection should be "adjusted" to be treated as perfect, but OK.

Care to respond to the part about you failing to account whether PPP's two strike-outs (or x strike-outs) were statistically significant even in a perfect world (i.e., even occur in more than 20 polls)?  Have you even calculated that?

Umm.

I have repeatedly tried to educate you, but you remain obstinant in thinking there is some "perfect world."  There isn't! 

Further, since there is no such thing as perfection, it can not be adjusted.

Odd, that you are incapable of comprehending what I have posted, or is it merely you deliberately choose to misrepresent?

[Grad Students are the Worst]

Explain which part of my post that rebutted.  Thanks.

[Yankees]

PPP hasn't had an accurate poll since the MA senate race.  I can give them credit for being the first pollsters to show Brown beating Coakly, but nothing more.

[Tender Branson]

PPP hasn't had an accurate poll since the MA senate race.  I can give them credit for being the first pollsters to show Brown beating Coakly, but nothing more.

You seem to have no clue.

[HappyWarrior]

Rasmussen is almost as far from the median as PPP

[King]

Rasmussen is almost as far from the median as PPP

But it's bias in the direction of the good guys.

[CARLHAYDEN]

Rasmussen is almost as far from the median as PPP

Yes, Rasmussen does tend to drift in the Republican direction, although somewhat less than PPP does in the Democrat direction.

However, there are a couple of things which Rasmussen does which most other survey research organizations generally do not do:

(a) it looks at the intensity of opinion, and
(b) it looks at changes in opinion (directionality).

[Joe Republic]

Interestingly, after an analysis of PPP and Rasmussen's final polls in each state ahead of last year's senatorial elections (and note that I'm only talking about the senatorial polls right now), both agencies overestimated the Republican candidates' eventual results.

In terms of percentage margin, PPP were on average 2.4 points more favorable to the Republicans, compared with 3.3 points from Rasmussen.

[CARLHAYDEN]

Interestingly, after an analysis of PPP and Rasmussen's final polls in each state ahead of last year's senatorial elections (and note that I'm only talking about the senatorial polls right now), both agencies overestimated the Republican candidates' eventual results.

In terms of percentage margin, PPP were on average 2.4 points more favorable to the Republicans, compared with 3.3 points from Rasmussen.

It is really kind of hard to evaluate your assertion as PPP did not release surveys for a lot of the Senate elections in 2010.

However, the last polls for Senate for Arizona, Florida, Iowa, Louisiana and New Hampshire showed a tighter race between the republican and democrat candidates than was actually the case.

Those polls can be reviewed at RealClearPolitics.

State   Poll   Date   R   D   Margin
               
AZ   Actual   2-Nov   59   35   24
   PPP   Oct 23-24   56   38   18
FL   Actual   2-Nov   49   20   29
   PPP   Oct 30-31   47   21   26
IA   Actual   2-Nov   64   33   31
   PPP   May 25-27   57   31   26
LA   Actual   2-Nov   57   38   19
   PPP   Aug 21-22   51   41   10
NH   Actual   2-Nov   60   37   23
   PPP   Oct 27-29   56   41   15