Nate Bashes Rassy

[Lunar]

http://www.fivethirtyeight.com/2010/07/is-poll-scientific-if-it-excludes-more.html

They don't elect to do these things, however, because it's not in their business model. In other words ... because they're cheap. That's not meant to be an ad hominem: it's the only way to accurately describe the problem.

Maybe Rasmussen's polling will happen to produce the right results in spite of all this. Historically, in fact, it's been about average -- a little better than average, actually, on balance. In the long-run, though, I think it's going to get them into some trouble, particularly as some of these issues -- fewer and fewer Americans have landlines each year -- are continuing to get worse.

...


But this discussion shouldn't be about the results that Rasmussen produces, either past, present or future. It should be about the state of the polling industry as a whole, an industry which is increasingly becoming a race to the bottom, and Rasmussen plays a big part in that.

[Lunar]

unrelated:

http://wonkette.com/416825/ooh-let-nate-silver-be-part-of-the-controversy-let-nate-silver-be-part-of-the-controversy

[J. J.]

So Basically Nate Silver is saying the 'bots do a better job than average, but he still doesn't like them.

[Cubby]

unrelated:

http://wonkette.com/416825/ooh-let-nate-silver-be-part-of-the-controversy-let-nate-silver-be-part-of-the-controversy

That was entertaining 

[President Marokai]

unrelated:

http://wonkette.com/416825/ooh-let-nate-silver-be-part-of-the-controversy-let-nate-silver-be-part-of-the-controversy

This sort of stuff basically sums up why I don't enjoy Wonkette.

[Former Moderate]

Oh, yawn, the "but what about cell phones" thing again?

Hasn't that been used on-and-off over the last what, ten years, to show that polling is faulty? Certain populations are harder to reach than others, of course, but that's why pollsters use weighting. Any idiot can call people out of a phone book. It's the weighting part that makes polling a "science."

For Rasmussen to be inherently faulty, you'd need to be able to show that people who shut off a landline in a certain demographic bracket are different from other people who keep a land line in the same demographic bracket. Given the past polling results, I'm not sure you can really say that there's much of a difference there.

If there was a huge problem here, polling would have come off the rails years ago.

[Lunar]

Oh, yawn, the "but what about cell phones" thing again?

I think the no-call-backs & single-day bits are relevant too, no?

[Former Moderate]

Oh, yawn, the "but what about cell phones" thing again?

I think the no-call-backs & single-day bits are relevant too, no?

So long as you can prove either would directly impact the data in a reliably biased way that is not fixable with weighting.

I'm not sure what same day polling would do to introduce game-changing error -- and would certainly not be something visible in their daily tracking poll. You could say that the people who answer phone calls at time x have different demographic data than those who would answer at time y -- and you'd be right, of course -- but again, that's the point of weighting. If you get an unusually high number of housewives and retirees over businessmen in their 40s, you use statistical methods to balance that out.

As for the call back issue -- again, can you say that there's a clear consistent ideological bias in people who answer phones when measured against those who do not answer phones? Or that there's a strong compelling reason why not asking for the person who has the next birthday introduces a statistically significant error?

Silver presents reasons why Rasmussen could be bad without actually showing that those reasons have significant statistical impact. He goes through a lot of (needless and nigh pointless) calculation of who may be reachable when, but he fails to do a comparison of value -- past results of pollsters who use method x against pollsters who use method y.

And I can understand why -- both Rasmussen and SurveyUSA (who uses automated polling very similar to Rasmussen) have fairly strong track records, especially when matched up against some of the "traditional" pollsters like Zogby.

[Lunar]

I'm not sure what same day polling would do to introduce game-changing error -- and would certainly not be something visible in their daily tracking poll.

Sorry for the cheapness of my response (at work) -- but I don't think Silver is arguing that Rasmussen is producing game-changing error. 

[muon2]

What I glean from Silver is that he has deduced the likely activities that Rassy's respondents are engaged in when they take the call. The undertone of the article seems to be that Rassy's sample set might have inherent bias, but there is no evidence of that. The only conclusion I can draw is that the sample does not have the same activity profile as the adult population as a whole.

On the surface I don't see the problem. I get the concept that by restricting call backs and time period, the number of unwanted variables is reduced. I presume that Rassy has appropriate weights to compensate for any skew in the sample due to these attempts to control unwanted variables. If the models that provide the weights are well tested, then the polls should be statistically sound.

[Former Moderate]

This bit may prove interesting to some:
http://www.pollster.com/blogs/surveyusa_polls_cell_phone_onl.php

"Their recent Washington poll, for example, shows Democratic Senator Patty Murray leading by a not-statistically-significant four-point margin (37% to 33%) over challenger Dino Rossi in a combined sample of landline and mobile phones. Murray's lead would have been a virtually identical five-point margin (39% to 34%) had they interviewed by landline phone only.

Similarly, a North Carolina survey released just yesterday shows Republican Richard Burr leading Democrat Elaine Marshall by ten points (46% to 36%) in the combined sample interviewed over both landline and cell phones. Burr would also have led by a 10-point margin (47% to 37%) had they interviewed all respondents via landline phones only."

[The Vorlon]

At it's core, there is almost a religious issue here.

Most pollsters, myself included, were raised in "The Church of the Random Sample" to use Mark Blumenthal's phrase.

Painfully, kicking and screaming, most pollsters have realized even a decent approximation of a true random sample is essentially impossible to achieve in the modern world.

Many phone pollsters (indeed just about everybody but Gallup and PEW) has essentially given up even trying to generate a random sample and has gone to some variation of quota calling. - Even Gallup is employing a lot more post sample weighting than they did before.

Approaches vary, but what ABC news (TNS is their actual call center) does is conceptually pretty typical.

The US census bureau divides the population into 48 census "cells" based upon age, income, race, education, geography, etc...  ABC "quota calls" - they screen respondents to see which "cell" they fit into and they then keep calling till they have filled their "quota" in all the cells...

What you end up with looks like a random sample, though if it actuially behaves like one is a matter of substantial mathematical debate....

ABC uses selection criteria when calling (ie asking for the youngest male initially) but this is done for economic reasons so as to try to fill their quota with the fewest possible calls, it is not part of the polling methodology per se.

I guess the question to be asked is once you have crossed the Rubicon (metaphorically speaking) does it really matter HOW you fill the quota? - It's not a random sample anymore, its a constructed stratified sample.

I was deeply skeptical of Rasmussen at the beginning, but the bottom line is his results, as defined by predicting actual outcomes in actual elections, are exceptionally good.

The "bot" also has the huge advantage of doing sooooo many surveys that he is at the point he has, in the gigantic aggregate, sooooo much information that his random error is basically zero, and all that is left is systemic error that he can (at least conceptually) weight away.  To quote "Uncle Joe Stalin..." - Quantity has a quality all it's own.

Rasmussen does +/- (if you count his separate economic and employment indexes) about half a million completed calls a year... he knows exactly how many people he actually reaches in every demographic, race, income, and other category... he can weight his data to fix it...

For those of us (Nate and Mark B are certainly on the list) raised in the" church of the random sample" Rasmussen is an atheist, and its hard for them to contemplate that maybe there actually is no God.

[The Vorlon]

Oh, yawn, the "but what about cell phones" thing again?

I think the no-call-backs & single-day bits are relevant too, no?

Actually... no....

If you use a stratified sample where you essentially build up your sample of people person by person so it looks like a true random sample, how you get there, assuming you do things correctly, really is unlikely to matter.

For example, lets say for my sample to be deemed representative, it needs 12 males between the ages of 45 and 54, who have a college degree, and live in the northeast (and in Rasmussen's case, because he weights by party ID, self identify as Democrats)

If I happen to find 12 males who meet all these criteria who are willing to pick up the phone on a tuesday evening, why should I presume them to be different than males who meet all the criteria except they are NOT willing (or not home) to pick up the phone?

Also....

Rasmussen buys his calling lists from a firm called "Survey Sampling International" - The exact same firm that Gallup (and Survey USA) uses.  

The difference (compared to Gallup) is that Rasmussen buys a variation of the calling lists called a "cluster sample" where groups that are known to be hard to reach are over-represented in the calling list so that after taking into account known patterns of actual response, the groups will be more or less correctly represented in the final sample.  

In the real world, what this means is, for example, old people (who answer the phone more often) are underrepresented in the raw calling list, while young people are over represented, so that after a typical response pattern, their representation in the final sample is roughly correct.

[Lunar]

Valid points all....what am I supposed to say? 

I think it's a good read, even if you end up disagreeing with it

[muon2]

At it's core, there is almost a religious issue here.

Most pollsters, myself included, were raised in "The Church of the Random Sample" to use Mark Blumenthal's phrase.

Painfully, kicking and screaming, most pollsters have realized even a decent approximation of a true random sample is essentially impossible to achieve in the modern world.

Many phone pollsters (indeed just about everybody but Gallup) has essentially given up even trying to generate a random sample and has gone to some variation of quota calling. - Even Gallup is employing a lot more post sample weighting than they did before.

Approaches vary, but what ABC news (TNS is their actual call center) does is conceptually pretty typical.

The US census bureau divides the population into 48 census "cells" based upon age, income, race, education, geography, etc...  ABC "quota calls" - they screen respondents to see which "cell" they fit into and they then keep calling till they have filled their "quota" in all the cells...

What you end up with looks like a random sample, though if it actuially behaves like one is a matter of substantial mathematical debate....

ABC uses selection criteria when calling (ie asking for the youngest male initially) but this is done for economic reasons so as to try to fill their quota with the fewest possible calls, it is not part of the polling methodology per se.

I guess the question to be asked is once you have crossed the Rubicon (metaphorically speaking) does it really matter HOW you fill the quota? - It's not a random sample anymore, its a constructed stratified sample.

I was deeply skeptical of Rasmussen at the beginning, but the bottom line is his results, as defined by predicting actual outcomes in actual elections, are exceptionally good.

The "bot" also has the huge advantage of doing sooooo many surveys that he is at the point he has, in the gigantic aggregate, sooooo much information that his random error is basically zero, and all that is left is systemic error that he can (at least conceptually) weight away.  To quote "Uncle Joe Stalin..." - Quantity has a quality all it's own.

Rasmussen does +/- (if you count his separate economic and employment indexes) about half a million completed calls a year... he knows exactly how many people he actually reaches in every demographic, race, income, and other category... he can weight his data to fix it...

For those of us (Nate and Mark B are certainly on the list) raised in the" church of the random sample" Rasmussen is an atheist, and its hard for them to contemplate that maybe there actually is no God.

It fascinates me that notions of weighting that have long been recognized in scientific experiments have been anathema to many in the polling biz. For instance, in particle physics there are way too many individual particle collisions to process in searching for rare phenomena. The experiment will set up the equivalent of a screen that biases the sample towards the events one is looking for, thus minimizing the processing time. Other control samples are used to determine precise weights for different parts of the screened sample so that a correct measurement results.

[The Vorlon]

It fascinates me that notions of weighting that have long been recognized in scientific experiments have been anathema to many in the polling biz. For instance, in particle physics there are way too many individual particle collisions to process in searching for rare phenomena. The experiment will set up the equivalent of a screen that biases the sample towards the events one is looking for, thus minimizing the processing time. Other control samples are used to determine precise weights for different parts of the screened sample so that a correct measurement results.

Have the folks over at CERN found the God Particle yet....?

[CARLHAYDEN]

As to the impact of cell phone only, please see:

http://www.pollster.com/blogs/surveyusa_olls_cell_onl.php

As to automated pollsters versus traditional pollsters, the evidence seems to be supporting automated pollsters as more accurate.

[muon2]

It fascinates me that notions of weighting that have long been recognized in scientific experiments have been anathema to many in the polling biz. For instance, in particle physics there are way too many individual particle collisions to process in searching for rare phenomena. The experiment will set up the equivalent of a screen that biases the sample towards the events one is looking for, thus minimizing the processing time. Other control samples are used to determine precise weights for different parts of the screened sample so that a correct measurement results.

Have the folks over at CERN found the God Particle yet....?

They had some problems with their magnets and will be taking a year and a half to fix them. Who knows, maybe we'll see it at Fermilab while they're shut down.

[The Vorlon]

It fascinates me that notions of weighting that have long been recognized in scientific experiments have been anathema to many in the polling biz. For instance, in particle physics there are way too many individual particle collisions to process in searching for rare phenomena. The experiment will set up the equivalent of a screen that biases the sample towards the events one is looking for, thus minimizing the processing time. Other control samples are used to determine precise weights for different parts of the screened sample so that a correct measurement results.

Have the folks over at CERN found the God Particle yet....?

They had some problems with their magnets and will be taking a year and a half to fix them. Who knows, maybe we'll see it at Fermilab while they're shut down.

Fermi Lab can do just under 2 TEV as I recall...? versus 7 (currently) at CERN?

The Higgs Bosom might be below the 2 TEV range as I recall..?  Been a while since I have looked at this area in any depth.

[muon2]

It fascinates me that notions of weighting that have long been recognized in scientific experiments have been anathema to many in the polling biz. For instance, in particle physics there are way too many individual particle collisions to process in searching for rare phenomena. The experiment will set up the equivalent of a screen that biases the sample towards the events one is looking for, thus minimizing the processing time. Other control samples are used to determine precise weights for different parts of the screened sample so that a correct measurement results.

Have the folks over at CERN found the God Particle yet....?

They had some problems with their magnets and will be taking a year and a half to fix them. Who knows, maybe we'll see it at Fermilab while they're shut down.

Fermi Lab can do just under 2 TEV as I recall...? versus 7 (currently) at CERN?

The Higgs Bosom might be below the 2 TEV range as I recall..?  Been a while since I have looked at this area in any depth.

The mass of the Higgs is well below 2 TeV if it follows anything like the Standard Model. The advantage of higher energy is a higher rate of production.

[The Vorlon]

The mass of the Higgs is well below 2 TeV if it follows anything like the Standard Model. The advantage of higher energy is a higher rate of production.

~~assuming~~ no intermediate particles 1.4 TEV is more or less the upper limit as I (very vaguely) recall otherwise electroweak symmetry breaking makes the math blow up beyond 1.4 tev.

Wouldn't it be amazing if the standard model ended up totaly wrong?

That that would be fun science!

[muon2]

The mass of the Higgs is well below 2 TeV if it follows anything like the Standard Model. The advantage of higher energy is a higher rate of production.

~~assuming~~ no intermediate particles 1.4 TEV is more or less the upper limit as I recall.

Wouldn't it be amazing if the standard model ended up totaly wrong?

That that would be fun science!

That's why we do the experiments.

Now if we could get the pollsters to appreciate variable control and cancellation of systematics in their sampling as well ...

[The Vorlon]

That's why we do the experiments.

Now if we could get the pollsters to appreciate variable control and cancellation of systematics in their sampling as well ...

I sometimes think what I do is hard, then I look over the math you guys do... and my brain hurts!