Which polls can we trust...?

[The Vorlon]

ALWAYS keeping in mind the margin of error, which firms can we trust...?

This is my personal rankings...

(Firms are NOT in any order within their categories)

The "Bias" I refer to is NOT ideological, but Methodological...

The Top Tier... Pretty darn good, most of the time...

Gallup (the real one, not the CNN/USAToday/Tracking version)
Battleground (Goaes/Lake/Perry) (mild GOP bias, maybe 1-2 points)
Tetter/Hart (Wall Street Journal/NBC News) (maybe a marginal GOP bias)
Mason Dixon - Damn good firm...
Fox News (Yes, liberals, check their historic record...)
Democracy Corps (Yes Republicans, James Carville's firm knows what they are doing.. marginal Dem bias)
TIPP/Infometrica (Investors Business Daily)
Snell/Perry/Lake
Yerxa/DDC
AP/Ipsos Reed
Harris

The Second Tier - Interesting, not worthless, but don't bet the farm....

Rasmussen (Misses the top tier, but not by much...still on "probation" after their 2000 presidential call)
ABC News (Strange weighting, but knocking on the top tier door)
Survey USA
ARG
Research 2000
PEW - very consistent, but has a structural 4 point Democratic bias.
Quinnipiac - Only University poll (I am familiar with) that's NOT totally $%$ing useless...

Third Rate...

CBS/NYTimes..

ok..they got LUCKY in 2000, but from 1980 to 1996 they AVERAGED missing the actually result by 2 and a half times the margin of error of their poll! - If they do ok this year, I might believe them.... maybe...

Newsweek..

(Historically waaaaay off the mark)

Time

(see Newsweek)

All off on his Own..

Zogby gets his own Category.  He is brilliant, and erratic.  

For the record, in 2002, he polled 17 races, he got 12 right, 5 wrong...  

Zogby is the "Drudge Report" of pollsters... he's often utterly wrong, but right just enough to keep you reading...

[Lunar]

Margin of error is pretty darn significant.  There is no probability given that the true value is anywhere within the margin, it's only known that it is probable that the true value is somewhere within the intervals.  So for one of these polls to be significant, it requires the candidates to be outside of the margin of error from each other, or that you spot a trend in many polls (which reduces the margin of error significantly).

I have no idea what the set confidence value is for a political poll, 95%?

[elcorazon]

lunar, you have the most unusual map I've seen for awhile.  Why do you think that Kerry will take Louisiana & Colorado?  I'm partly intrigued because I think Louisiana is Kerry's only shot in the south other than FL & WV and because I'm one of the few who think Colorado is in play.  I also think Virginia could be.  What's your line of thinking on this?

[Lunar]

There is a short description if you follow the link in my signature.  Lousiana has strong democratic trends and has a Catholic majority while Colorado has a growing Hispanic vote, is Kerry's birthplace, and could go under the radar.

Besides, it's more fun to predict an unusual scenario!  Anyway, we shouldn't hijack this thread, knock it over to the discussions thread if you want to continue, I'll read it.

[The Vorlon]

I have no idea what the set confidence value is for a political poll, 95%?

95% = two standard deviations from the norm, in an evenly divided bi-nomial expansion.

They picked it 'cause its a place where the math is easy..

[JohnFKennedy]

I have no idea what the set confidence value is for a political poll, 95%?

95% = two standard deviations from the norn, in an evenly divided bi-nomial expasion.

They picked it 'cause its a place where the math is easy..

i actually understood the maths talk lol, I am doing binomial expansions at the moment.

[angus]

ah, yes.  good ol' Louis Pascal and his triangle.  Useful in molecular spectroscopic line assignments too!  

Vorlon, thanks for this post.  It answers many questions.  Not why? or wherefore?  but which?  I think I'm going to cut and paste your post into a file for future reference.

[Huckleberry Finn]

There is a short description if you follow the link in my signature.  Lousiana has strong democratic trends and has a Catholic majority while Colorado has a growing Hispanic vote, is Kerry's birthplace, and could go under the radar.

Besides, it's more fun to predict an unusual scenario!  Anyway, we shouldn't hijack this thread, knock it over to the discussions thread if you want to continue, I'll read it.

Also New Mexico is very Catholic and Hispanic and you've put it for Bush.

In other hand your map is funnier that way.

[muon2]

In 2000 I followed the Battleground polling at their website. This year it seems to have stopped with their results from last September. Have they gone to private polling only?

[The Vorlon]

In 2000 I followed the Battleground polling at their website. This year it seems to have stopped with their results from last September. Have they gone to private polling only?

Muon2...

Didn't I see you violating matter/antimatter symmetry (or was that your evil twin Kaon..?) in another thread...?

Battleground is a joint venture between Terrance Group  and Snell/Lake/Perry (two excellent firms)

Terrance Group is polling the $%^# out of Ohio right now, so they are a tad busy..

[Nym90]

That's correct, if the poll is completely random and does not bias in favor of the supporters of any candidate in any way, then there is a 95% probability that the result for each candidate is within the margin of error. So even a perfect poll should still be wrong outside the MoE once every 80 years or so.

It is worth mentioning also that if a poll has a 95% chance of being right within +/- 4%, then it has a 68% chance of being within 2%, and a 99% chance of being within 6%. For most statisticians, though, a 95% probability is considered good enough to be called "statistically significant". Hence, if the result is within the Margin of Error, it is considered to be a statistical tie.

[WalterMitty]

why are most university polls unreliable?

[muon2]

In 2000 I followed the Battleground polling at their website. This year it seems to have stopped with their results from last September. Have they gone to private polling only?

Muon2...

Didn't I see you violating matter/antimatter symmetry (or was that your evil twin Kaon..?) in another thread...?

Battleground is a joint venture between Terrance Group  and Snell/Lake/Perry (two excellent firms)

Terrance Group is polling the $%^# out of Ohio right now, so they are a tad busy..

I was hoping they'd be doing a national tracking poll again this year.

[angus]

I thought it might be useful to remind us about polling.

[The Vorlon]

I thought it might be useful to remind us about polling.

And as an after thought....

Polls go really haywire in July and August when folks are on vacation.  The barely work now, they really do off the deep end when the kids are out of school.

The good firms will compensate, but from the 2nd tier and below ones... expect some truly strange results...

In August if you see Kerry with in the margin of error of Bush in Idaho, or Bush "closing fast" in Rhode Island... just remember this post...

[ilikeverin]

To the all great polling guru The Vorlon...

What's up with your projection... innovative setup (I wanna steal it! ), but why on earth is it called March 4.jpg?

Anyway, great info here... I wish it was stickied

[The Vorlon]

To the all great polling guru The Vorlon...

What's up with your projection... innovative setup (I wanna steal it! ), but why on earth is it called March 4.jpg?

Anyway, great info here... I wish it was stickied

Because it is a jpeg image, created on March4 originally...

Sometimes a cigar, is just a cigar...

I edit it as time goes on...

[ian]

I think Mason-Dixon is slanted to the right & even tho Rasmussen screwed up last election, I think their results lately are pretty accurate.  I agree that University polls are crap.  And Gallup is awesome.  SUSA is so far to the right.  As is ARG.  Keystone couldn't get any further to the right.  I think Quinnipac is quite good.

[angus]

ALWAYS keeping in mind the margin of error, which firms can we trust...?

This is my personal rankings...

(Firms are NOT in any order within their categories)

The "Bias" I refer to is NOT ideological, but Methodological...

The Top Tier... Pretty darn good, most of the time...

Gallup (the real one, not the CNN/USAToday/Tracking version)
Battleground (Goaes/Lake/Perry) (mild GOP bias, maybe 1-2 points)
Tetter/Hart (Wall Street Journal/NBC News) (maybe a marginal GOP bias)
Mason Dixon - Damn good firm...
Fox News (Yes, liberals, check their historic record...)
Democracy Corps (Yes Republicans, James Carville's firm knows what they are doing.. marginal Dem bias)
TIPP/Infometrica (Investors Business Daily)
Snell/Perry/Lake
Yerxa/DDC
AP/Ipsos Reed
Harris

The Second Tier - Interesting, not worthless, but don't bet the farm....

Rasmussen (Misses the top tier, but not by much...still on "probation" after their 2000 presidential call)
ABC News (Strange weighting, but knocking on the top tier door)
Survey USA
ARG
Research 2000
PEW - very consistent, but has a structural 4 point Democratic bias.
Quinnipiac - Only University poll (I am familiar with) that's NOT totally $%$ing useless...

Third Rate...

CBS/NYTimes..

ok..they got LUCKY in 2000, but from 1980 to 1996 they AVERAGED missing the actually result by 2 and a half times the margin of error of their poll! - If they do ok this year, I might believe them.... maybe...

Newsweek..

(Historically waaaaay off the mark)

Time

(see Newsweek)

All off on his Own..

Zogby gets his own Category.  He is brilliant, and erratic.  

For the record, in 2002, he polled 17 races, he got 12 right, 5 wrong...  

Zogby is the "Drudge Report" of pollsters... he's often utterly wrong, but right just enough to keep you reading...

have models been developed which accurately predict small changes due to projected voter registration trends?  say, based on studied movement in registration drive efforts.  are the accounted for in any polls?  I suppose that might contribute more uncertainty.

[The Vorlon]

I think Mason-Dixon is slanted to the right & even tho Rasmussen screwed up last election, I think their results lately are pretty accurate.  I agree that University polls are crap.  And Gallup is awesome.  SUSA is so far to the right.  As is ARG.  Keystone couldn't get any further to the right.  I think Quinnipac is quite good.

In 2002 Mason-Dixon polled 17 Senate races.  They got every single one right.

Hard to argue with that.


The "real" Gallup (not the cnn/usatoday version) is still the gold standard. - Polling just does not get any better.

SUSA is not "right" they are just bad in a non-partisan way.
I disassembled to SUSA polls recently, the Kerry+10 in Michigan and the Bush +9 in Arizona, and both were just brutally unbalanced.

[agcatter]

Survey USA is not to the right.  See Kerry +10 in Michigan.

[The Vorlon]

You must be logged in to read this quote.

There are two "religious faiths" in polling.

There are those who weight their samples for things like party Id, and there are those who don't.

Which faith is the correct one is unclear based upon the historic record.

Gallup, which does not weight, can go toe to toe with anybody and over the long term look pretty good...

Those who DO weight for things such as party id need to make darn sure their weighting assumptions are correct, because if they are wrong, the poll goes right off the rails.

To directly answer your question, Yes, changes in demographics, party Id, etc are incorporated into the polling models used.

Like any field of endeavor, there are some firms that are just a lot better at getting it done right.

I guess my take is that if you look over the long, long term both "religious faiths" have similar track records if you compare equal quality firms.

Those who weight will usually be a bit more accurate, but on the occassions when they do miss, they miss by a lot more...

Gallup, by contrast will usually miss the mark by a bit more than the others, but will rarely have the huge misses that the "weighters" now and then run into..

[The Vorlon]

Survey USA is not to the right.  See Kerry +10 in Michigan.

The Arizona Bush+9 from SUSA and the Kerry+10 in Michigan were both just waaaay out there...

[CARLHAYDEN]

Vorlon,

Very interesting thread.

Might I suggest two factors (other than bias) which impact results.

First, a large percentage of the population now refuses to answer the questions or talk to pollsters (I know from experience as I at one time worked for an/major survery research firm).  The assumption that their voting behavior is mimicked by those who do answer such questions is questionable to say the least.

Second, there is the little problem of turout.  In 1994 the voters were skewed to the right while in 1998 and 2000 the actual voters were skewed to the left (the Democrats had a highly effective voter participation drive the black community).  Zogby is a Democrat politician (he ran for Mayor as a Democrat) with links to the Clinton administration (his brother worked there) and was aware of the turnout effort in 2000, which allowed him to more accurately forecast the result.

I would suggest that non-poll data (voter registration by party, primary election results, etc.) is a good cross check for polling.

I suspect

[The Vorlon]

Vorlon,

Very interesting thread.

Might I suggest two factors (other than bias) which impact results.

First, a large percentage of the population now refuses to answer the questions or talk to pollsters (I know from experience as I at one time worked for an/major survery research firm).  The assumption that their voting behavior is mimicked by those who do answer such questions is questionable to say the least.

Second, there is the little problem of turout.  In 1994 the voters were skewed to the right while in 1998 and 2000 the actual voters were skewed to the left (the Democrats had a highly effective voter participation drive the black community).  Zogby is a Democrat politician (he ran for Mayor as a Democrat) with links to the Clinton administration (his brother worked there) and was aware of the turnout effort in 2000, which allowed him to more accurately forecast the result.

I would suggest that non-poll data (voter registration by party, primary election results, etc.) is a good cross check for polling.

I suspect

You are correct about response rates.  Currently only about 1 in 4 actually complete the survey when requested.

The advent of things like Caller ID is making things in this area even worse.

Your comment regarding party ID changes is also valid.

In 1994 when the GOP took over Congress, most pollsters (those who weighted their results by party ID) missed it.

Actual GOP vs Dem turnout in 1994 was almost exactly even, but back then most pollsters like Zobgy weighted at something like 40/35/25 Dem/GOP/Ind so the "extra" 5% GOP, while showing up in the raw data got "lost" in the weighting process.

In an election like 1994 the Gallup method of NOT weighting for Party ID was clearly superior.

Weighting for things like party ID, turnout levels among various groups, etc is a set of ASSUMPTIONS.

To the degrees these assumptions are correct, your poll gets more accurate, to the degree they are wrong, you get a less accurate poll.

Zogby is a love/hate pollster to me.  

He weights the %^%$ out of his polls, far more aggresively than any other major firm.

Many times he is right, many times he is wrong.  

Zogby is the undisputed champ in both spectacular successes AND spectacular failures.

2004 looks more volitile to me than most years, I think the "purests" - Folks like Gallup, TIPP, Fox and Ipsos are the ones most likely to get it right this year.