Polls

[CARLHAYDEN]

But he says its a Mason-Dixon poll.  You have a very high opinion of Mason-Dixon, I thought.

Mason-Dixon - absolutely a VERY good firm.  At the state by state level easily[/b] the best firm out there.  Frankly, compared to a lot of the other firms, it's not even close.

MD is the only firm in the public domain I actually trust from here to September that does state polls. (Maybe Ipsos too actually)

I was talking "generically" in this post.

Mason Dixon's record speaks for it's self.  In 2002 they publically polled 23 races and got 22 right.  Their average candidate error was 1.8%.  (If you limit it to polls published in the last 7 days of the campaign they did even better) The other 70 or so private polls they did in 2002 had a comparable success rate.

The only firm with lower average candidate error was Gallup, and there is certainly no shame in coming in 2nd to Gallup..

First, welcome back.  Hope you had a good vacation.

Second, thank you for your observations on the previous post to me.

Third, I agree on MD being a quality firm, I have my doubts about ISPOS (they seem to do ok in Canada but, IMHO they tend to overstate the liberal and Democrat support in this country).

Fourth, while the numbers aren't "hard," and aren't kept on official record, it appears that in the battleground states (which I define as being a state which either Bush or Gore carried with a margin of less than ten per cent) the "refuse" to participate in the surveys is highest among males whose age group is estimated to be 45-64 (without noticable accents).  

While you can (eventually) get enough 45-64 year old males to meet quota, I maintain that those responding do not necessarily reflect the views of those who refuse to participate.  

I believe that the exit polls should be funded by a consortium of existed reputable pollsters, who could use the data base to adjust their future polls.  This would probably be able to either confirm or deny my suspicion on this matter.

Fifth, also, please note that even when reading the questions verbatim to the respondents, the accent of the questioner appears to have some impact on the responses.  This is particularly important as firms do not pay particularly well for the questioners and tend to hire the less affluent (and more prone to regional accent) segment of the population.  

[The Vorlon]

But he says its a Mason-Dixon poll.  You have a very high opinion of Mason-Dixon, I thought.

Mason-Dixon - absolutely a VERY good firm.  At the state by state level easily[/b] the best firm out there.  Frankly, compared to a lot of the other firms, it's not even close.

MD is the only firm in the public domain I actually trust from here to September that does state polls. (Maybe Ipsos too actually)

I was talking "generically" in this post.

Mason Dixon's record speaks for it's self.  In 2002 they publically polled 23 races and got 22 right.  Their average candidate error was 1.8%.  (If you limit it to polls published in the last 7 days of the campaign they did even better) The other 70 or so private polls they did in 2002 had a comparable success rate.

The only firm with lower average candidate error was Gallup, and there is certainly no shame in coming in 2nd to Gallup..

First, welcome back.  Hope you had a good vacation.

Second, thank you for your observations on the previous post to me.

Third, I agree on MD being a quality firm, I have my doubts about ISPOS (they seem to do ok in Canada but, IMHO they tend to overstate the liberal and Democrat support in this country).

Fourth, while the numbers aren't "hard," and aren't kept on official record, it appears that in the battleground states (which I define as being a state which either Bush or Gore carried with a margin of less than ten per cent) the "refuse" to participate in the surveys is highest among males whose age group is estimated to be 45-64 (without noticable accents).  

While you can (eventually) get enough 45-64 year old males to meet quota, I maintain that those responding do not necessarily reflect the views of those who refuse to participate.  

I believe that the exit polls should be funded by a consortium of existed reputable pollsters, who could use the data base to adjust their future polls.  This would probably be able to either confirm or deny my suspicion on this matter.

Fifth, also, please note that even when reading the questions verbatim to the respondents, the accent of the questioner appears to have some impact on the responses.  This is particularly important as firms do not pay particularly well for the questioners and tend to hire the less affluent (and more prone to regional accent) segment of the population.  

You point regard respone rates is also valid, and touchs on a big question re sample stratification.

You eventually meet your quota, you have enough whites, blacks, young, old, rich poor, etc...

I looks like a ramdom sample.  Does that mean it is a random sample?
Regarding

[CARLHAYDEN]

I think you got cut off before you could complete your post.

I'd be interested in what else you have to say.

In addition to exit polls to serve as a partial corrective to existing problems, some firms on the qt also compare their data to University of Michigan SRC data (when it eventually, long after the fact becomes available).

[??????????]

Vorlon. Why doesnt some rich financier come up with the funds to have a 500,000 person sample? With todays modern technology that would be more then possible. Put some of those telemarkters who lost their jobs because they have been outsourced to work!


Vorlon, that wasn't a joke. It's a serious question.

[classical liberal]

Why does the poll page on this site show MI, WI, and NH as tossups when there is a 9-10 point Kerry lead according to the lastest polls posted for each state?

[CARLHAYDEN]

While waiting for Vorlon to reply, let me suggest two reasons.

First, sample size is not usually the biggest error built into polls.

For example, a poll with 900 respondents would have a sample size error of approximately 3.33% whereas a poll with 1600 respondents would have a sample size error rate of approximately 1.66%.  Even if you got up to a poll with 6400 respondets you would still have a sample size error of approximately 1.25%.  Increasing the sample size beyond a certain point yields verly little in reducing error.

Second, even thought there are a lot of people doing polls, the cost of doing a decent poll (one without a lot of inbuilt problems) is very high!

There is one advantage to doing a supersize poll , namely that (if correctly done) it would provide valid information on small subsets.

For example, the subset on a typical poll for Jewish voters is so small that the sample size error rate exceeds ten per cent.  This means that a poll that shows Jewish voters favoring Kerry over Bush  by say 65 to 35 could easily actually be 50-50, or 80-20, or anywhere in between.

The same holds true for other groups (farmers come to mind).

[??????????]

But a 500k person poll would give you a better idea of how the people actually felt, right? Much better then a 800 person poll. Meaning, I would put my faith in a 500k person poll more then a 800 person poll.

[CARLHAYDEN]

Lets take a case of an 800 respondent poll versus a 500,000 respondent poll.

If the 800 repondent poll said half the respondents prefer apple pie to cherry pie, and half prefer cherry pie to apple pie, its likely that (based on sample size) somewhere between 47-53 per cent prefer either.

While the 500,000 respondent survey would reduce this to 49.8-50.2% swing, the cost is not worth the small improvement.

The built in problems with polls (irrespective of sample size) would dwarf the sample size improvement resulting in negligible improvement for a vastly increased cost.

[The Vorlon]

Vorlon. Why doesnt some rich financier come up with the funds to have a 500,000 person sample? With todays modern technology that would be more then possible. Put some of those telemarkters who lost their jobs because they have been outsourced to work!


Vorlon, that wasn't a joke. It's a serious question.

A poll has two types of error - pure random error due to statistical fluctuation, and methodological error.

Unless you are talking a VERY good firm, methodological errors are typically larger than statistical ones.

A big sample will reduce the random statistical error, but does nothing about methodological error.

If you toss a coin 100 times, it might come up 50/50 heads/tails, but usually 51/49, or 52/48 or there abouts.  Once in a blue moon it comes up 80/20.  This is statistical or random error.  Nothing you can do about it.

To extend my example, let's say I have a problem with my coins and 80 of the have a head and a tail but 20 of them are two headed.

Over time, the 80 normal coins will tend towards 40/40, but the 20 two headed coins will always be 20 heads.

I can have an infinite sample and flip the coins a zillion times and I will typically get 60/40 rather than 50/50.  This is a methodological error.

For example if I was to make some crazy assumption and say weight my poll so that Republicans made up 50% of the sample or that there were 7% more Democrats than Republicans for example, I could talk to 10 million people and still get a garbage result.

The reduction in statistical error is proportional to the square root (not exactly but damn close) of the sample size, so you get to the point of diminish returns pretty fast.

For example, a sample of 1000 has an error of 3.1%
A sample size of 2000 has an error of 2.2%

You have doubled your costs to pick up 0.9%

A sample size of 10,000 has an error of 1.0%

I have made my sample 5 times bigger to get another 1.2%

A sample of 100,000 has an error of 0.31

Compared to 1000 size, we had to make our poll 100 times bigger to get 10 times more accurate.

A huge sample also does NOTHING to get rid of methodological errors - problems in how you have worded your question, how you select people, etc....

To take an extreme example, lets say I polled 10,000 "likely" voters.  In this case I defined "likely" as owning a BMW. (it's a silly example but it makes my point)

My sampling error on 10,000 is only 1%, but my methodological error due to a silly sampling method would render the poll likely to be waaaay out because 10,000 BMW owners are not a representitive sample of actual voters.

In 1936 Readers Digest (I think?) Magazine did a poll of 3,000,000 (?) Americans - all of whom susbscribed to Readers Digest and had telephones.  Sampling error should have been pretty close to zero.  Unfortunately in 1936 both telephones and magazine subscriptions were luxury items so the 3,000,000 polled looked very little like America.  In this poll "President" Alf Landon beat Rossevelt by about 15% if I recall corectly.

To use a recent example, here is Mr. Zogby's results from his 2002 senate tracking polls.



Clearly sample size is NOT the issue, the errors are just too many and too big to be statistical bad luck.

Something in they way he weighted/sampled/worded was just plain wrong (and hopefully corrected for 2004)

A bigger sample would NOT have corrected these problems.

Bottom line - a bad poll with a big sample is still a bad poll.

Regarding the cost of a poll.

A sample of 1500 from a good firm doing it totally absolutely by the book costs in the neighbourhood of $30,000 to $40,000.

If you poll is all of America or a state or just a city, the cost varies fairly little actually.

There are firms that change a heck of a lot less.

You get what you pay for.

[The Vorlon]

It was Liberty Digest and it went out of business shortly afterwards.

Thank-you.

 I knew it was xxx Digest, could not quite recall the name.

[??????????]

Thanks for the info. I am not to good at math but I get the jist of what you are saying.

[CARLHAYDEN]

It was Liberty Digest and it went out of business shortly afterwards.

Thank-you.

There is an interesting paper by Brown & Chappell (University of South Carolina) called Forecasting Presidential Elections Using History ahd Polls.

One of the interesting things they found looking at the Gallup Polls was that there was "a tendency for polls to overstate support for Democratic candidates" (page .

P.S. It was the Literary Digest.

 I knew it was xxx Digest, could not quite recall the name.

[Andrew]

It was Literary Digest.

[CARLHAYDEN]

Glad to see someone else caught this.

[CARLHAYDEN]

Hey, we all make mistakes.

Actually Vorlon was partially mistaken about the source for the Literary Digest poll.

They sent return mail ballots to subscribers, and to people who had telephones (even if they weren't subscribers).  

This was as much a gimmick to gain subscriptions as it is to being a genuine poll.

[The Vorlon]

Hey, we all make mistakes.

Actually Vorlon was partially mistaken about the source for the Literary Digest poll.

They sent return mail ballots to subscribers, and to people who had telephones (even if they weren't subscribers).  

This was as much a gimmick to gain subscriptions as it is to being a genuine poll.

Yes that sounds right.  I knew the sample was huge but really flawed.

Good memory you have there!

[CARLHAYDEN]

Hey, we all make mistakes.

Actually Vorlon was partially mistaken about the source for the Literary Digest poll.

They sent return mail ballots to subscribers, and to people who had telephones (even if they weren't subscribers).  

This was as much a gimmick to gain subscriptions as it is to being a genuine poll.

Yes that sounds right.  I knew the sample was huge but really flawed.

Good memory you have there!

I find that I can remember the substance of matters, but sometimes have trouble dredging up a citation of source.

[The Vorlon]

3. when the undecideds and leaners are asked to respond to a variety of issue questions they consistently score significantly closer to the core Bush supporters than they do to the core Kerry supporters (or more accurately anti=Bush people).

*************************

Unfortunately, the polls which have explicity differeniated between the beliefs of Kerry voters, Bush voters and undecided voters on the issues are not available for publication at this time.

The sources which have commissioned the surveys are more interested in using the data than in publishing it at this time.

Boy, theoretically speaking I suspect it's barely conceivable that a Public Opinion Strategies poll from May 17th-23rd of 1137 likely voters nationwide (which may have been oversampled in 7 key states to provide better breakouts) could have hypothetically shown something similar to that.  

But that's just a totally random guess of course.

[CARLHAYDEN]

I can give you one issue which can be fairly well checked, Kerry voters are pretty closely divided on the death penalty (which Kerry opposes), while both Bush and swing voters overwhelmingly favor it.

Notice how neither side has raised this issue yet?

Kerry is afraid of it (as well as others) while Bush is saving it.

BTW, Public Opinion Strategies is NOT, in MHP, one of the better survey research firms.

[Gustaf]

Hehe...Vorlon, once again I explained something in a much shorter post and without images...

A way bak on another thread, but I made teh same point on marginal returns.

[© tweed]

xxx Digest

That sounds like a good magazine

[Gustaf]

Hehe...Vorlon, once again I explained something in a much shorter post and without images...

A way bak on another thread, but I made teh same point on marginal returns.

Here we go:

It's not worth it. The increase in accuracy becomes less and less cost-effective, simply. So you'd pay a lot of money and get very little for it. You would always have the MoEs anyway.  

[The Vorlon]

Hehe...Vorlon, once again I explained something in a much shorter post and without images...

A way bak on another thread, but I made teh same point on marginal returns.

Here we go:

It's not worth it. The increase in accuracy becomes less and less cost-effective, simply. So you'd pay a lot of money and get very little for it. You would always have the MoEs anyway.  

ok, I admit it.  you are the literary xxx Liberty Reader Digest version of things.

But I make prettier diagrams than you do...

Damn swedish punks... always making fun of the dyslexic, forgetful, long winded old guys with bad reading and typing skills....

[Fmr. Gov. NickG]

The 2000 results on the Polls page for Mississippi are not correct...Bush won by 16%, not 8%.

[The Vorlon]

The 2000 results on the Polls page for Mississippi are not correct...Bush won by 16%, not 8%.

Corrected & Thank-you!