Nate Bashes Rassy

[The Vorlon]

At it's core, there is almost a religious issue here.

Most pollsters, myself included, were raised in "The Church of the Random Sample" to use Mark Blumenthal's phrase.

Painfully, kicking and screaming, most pollsters have realized even a decent approximation of a true random sample is essentially impossible to achieve in the modern world.

Many phone pollsters (indeed just about everybody but Gallup and PEW) has essentially given up even trying to generate a random sample and has gone to some variation of quota calling. - Even Gallup is employing a lot more post sample weighting than they did before.

Approaches vary, but what ABC news (TNS is their actual call center) does is conceptually pretty typical.

The US census bureau divides the population into 48 census "cells" based upon age, income, race, education, geography, etc...  ABC "quota calls" - they screen respondents to see which "cell" they fit into and they then keep calling till they have filled their "quota" in all the cells...

What you end up with looks like a random sample, though if it actuially behaves like one is a matter of substantial mathematical debate....

ABC uses selection criteria when calling (ie asking for the youngest male initially) but this is done for economic reasons so as to try to fill their quota with the fewest possible calls, it is not part of the polling methodology per se.

I guess the question to be asked is once you have crossed the Rubicon (metaphorically speaking) does it really matter HOW you fill the quota? - It's not a random sample anymore, its a constructed stratified sample.

I was deeply skeptical of Rasmussen at the beginning, but the bottom line is his results, as defined by predicting actual outcomes in actual elections, are exceptionally good.

The "bot" also has the huge advantage of doing sooooo many surveys that he is at the point he has, in the gigantic aggregate, sooooo much information that his random error is basically zero, and all that is left is systemic error that he can (at least conceptually) weight away.  To quote "Uncle Joe Stalin..." - Quantity has a quality all it's own.

Rasmussen does +/- (if you count his separate economic and employment indexes) about half a million completed calls a year... he knows exactly how many people he actually reaches in every demographic, race, income, and other category... he can weight his data to fix it...

For those of us (Nate and Mark B are certainly on the list) raised in the" church of the random sample" Rasmussen is an atheist, and its hard for them to contemplate that maybe there actually is no God.

[The Vorlon]

Oh, yawn, the "but what about cell phones" thing again?

I think the no-call-backs & single-day bits are relevant too, no?

Actually... no....

If you use a stratified sample where you essentially build up your sample of people person by person so it looks like a true random sample, how you get there, assuming you do things correctly, really is unlikely to matter.

For example, lets say for my sample to be deemed representative, it needs 12 males between the ages of 45 and 54, who have a college degree, and live in the northeast (and in Rasmussen's case, because he weights by party ID, self identify as Democrats)

If I happen to find 12 males who meet all these criteria who are willing to pick up the phone on a tuesday evening, why should I presume them to be different than males who meet all the criteria except they are NOT willing (or not home) to pick up the phone?

Also....

Rasmussen buys his calling lists from a firm called "Survey Sampling International" - The exact same firm that Gallup (and Survey USA) uses.  

The difference (compared to Gallup) is that Rasmussen buys a variation of the calling lists called a "cluster sample" where groups that are known to be hard to reach are over-represented in the calling list so that after taking into account known patterns of actual response, the groups will be more or less correctly represented in the final sample.  

In the real world, what this means is, for example, old people (who answer the phone more often) are underrepresented in the raw calling list, while young people are over represented, so that after a typical response pattern, their representation in the final sample is roughly correct.

[The Vorlon]

It fascinates me that notions of weighting that have long been recognized in scientific experiments have been anathema to many in the polling biz. For instance, in particle physics there are way too many individual particle collisions to process in searching for rare phenomena. The experiment will set up the equivalent of a screen that biases the sample towards the events one is looking for, thus minimizing the processing time. Other control samples are used to determine precise weights for different parts of the screened sample so that a correct measurement results.

Have the folks over at CERN found the God Particle yet....?

[The Vorlon]

It fascinates me that notions of weighting that have long been recognized in scientific experiments have been anathema to many in the polling biz. For instance, in particle physics there are way too many individual particle collisions to process in searching for rare phenomena. The experiment will set up the equivalent of a screen that biases the sample towards the events one is looking for, thus minimizing the processing time. Other control samples are used to determine precise weights for different parts of the screened sample so that a correct measurement results.

Have the folks over at CERN found the God Particle yet....?

They had some problems with their magnets and will be taking a year and a half to fix them. Who knows, maybe we'll see it at Fermilab while they're shut down.

Fermi Lab can do just under 2 TEV as I recall...? versus 7 (currently) at CERN?

The Higgs Bosom might be below the 2 TEV range as I recall..?  Been a while since I have looked at this area in any depth.

[The Vorlon]

The mass of the Higgs is well below 2 TeV if it follows anything like the Standard Model. The advantage of higher energy is a higher rate of production.

~~assuming~~ no intermediate particles 1.4 TEV is more or less the upper limit as I (very vaguely) recall otherwise electroweak symmetry breaking makes the math blow up beyond 1.4 tev.

Wouldn't it be amazing if the standard model ended up totaly wrong?

That that would be fun science!

[The Vorlon]

That's why we do the experiments.

Now if we could get the pollsters to appreciate variable control and cancellation of systematics in their sampling as well ...

I sometimes think what I do is hard, then I look over the math you guys do... and my brain hurts!