Non-Gallup/Rasmussen tracking polls thread

[Lunar]

[Lunar]

538 stipulates:

What are the odds, given the parameters above, that a random sampling of 98 voters aged 18-24would distribute themselves 74% to McCain and 22% to Obama?

Using a binomial distribution, the odds are 54,604,929,633-to-1 against. That is, about 55 billion to one.

So, there is an 0.000000002% chance that IBD/TIPP just got really unlucky. Conversely, there is a 99.999999998% chance that one of the following things is true:

(i) They're massively undersampling the youth vote. If you only have, say, 30 young voters when you should have 100 or so in your sample, than the odds of a freak occurrence like this are significantly more likely.
-or-
(ii) Something is dramatically wrong with their sampling or weighting procedures, or their likely voter model.

My guess is that it's some combination of the two -- that, for instance, IBD/TIPP is applying a very stringent likely voter model that removes you from the sample if you haven't voted in the past two elections, which would rule a great number of 18-24 year olds out.

[Lunar]

Lunar, the subset problem has been discussed.

Sigh.


Subsets are allowed to fluctuate, say, McCain winning 15% of African-Americans, but if McCain is winning 80% of African-Americans, are we allowed to question the all-mighty poll?

The math clearly takes these previous "discussions" into account.

[Lunar]

Lunar, the subset problem has been discussed.

Where, and to what conclusion?

This thread, IIRC, about 4-5 days ago.  It was just too small a sample size to be able to generalize.

The math takes the small sample size (98) into account.

[Lunar]

To be honest, the error is more likely than the raw calculation.  Why?  Because the error wasn't predetermined, we were looking at ALL of their data, which means we could find various age-sample outliers, various race outliers, etc.  We just happened to grasp on the one that is off.

That plus the small sample-size are ONLY two reasons to discard subsamples.  

Discarding subsamples is not a rule and it's not universal, J.J., it's just a rule of thumb.  There's a line somewhere (we could calculate it, based on the number of crosstabs available and known sample sizes, but it'd be way too much work) where the poll crosses over into the unacceptable line and becomes DISQUALIFIED.  If some poll of Iowa shows McCain winning 51% of AA voters, that's not that bad.   If some poll shows McCain within a couple points or whatever on youth voters, that's not that bad.  Once it crosses that line, which is fuzzy at this point, but can be finely calculated, the poll becomes ridiculous.  If some poll showed McCain winning 100% of AA's in Georgia, that's ridiculous.  And this poll, which shows McCain winning youth voters by a 50% margin, becomes disqualified from legitimacy in a simpler manner.

Especially because the topline result is also an outlier.

I don't see how you can logically argue otherwise.

[Lunar]

If a pollster is more likely to be hit by a meteor and killed than get the result they release, then their model is bad.

[Lunar]

I'm saying the subsample is bad, but a bad subsample doesn't invalidate the poll.

So, in other words, you are maintaining that it's a one-in-several-million event, and not a flaw in the methodology.  Why?

No, I'm saying that the poll isn't primary designed to accurately measure a subsample.

Yes, but at some level you have to question the methodologies, especially if the topline result isn't in line with other polls.

The subsample can reveal the methodological weaknesses of a poll.  If a subsample shows 500% growth among elderly voters, then perhaps the poll is doing something funky.  It's wrong to pick apart moderate subsample weaknesses (say McCain winning 15% of blacks, a common subsample weakness with about the same sample size as Youth), but at some point we got to take a step back and say that this poll is probably doing something wrong.  The odds are simply one in a trillion that this poll doesn't have an overarching flaw.

At what point do you, J.J., question a sub sample?  Is a one and a gazillion bazillion chance not enough for you?  Does McCain need to win 100% of Blacks in Mississippi?

[Lunar]

Only those three options are logically possible.  There is no option #4.

If Gallup had a poll with a subsample as wildly off as this poll, then yes, we would presume that either there had been a huge swing in the subsample or that Gallup had a flawed methodology.

[Lunar]

J.J. is saying that it all averages out Alcon.  

So, yes, this mistake was just because of randomization.

He believes in the  0.000000002%

[Lunar]

J.J. is saying that it all averages out Alcon.  

So, yes, this mistake was just because of randomization.

He believes in the  0.000000002%

Think about it Alcon, let me explain it to you.

Sure, while this poll, assuming it had perfect methodology, would be 99.999999998% not to get this result.

But did you know that atoms are 99.9999999999% empty?  That's only one more 9 than this poll has.  Punch your wall ten times, did your hand go through it?  WHOOAAH you just defied statistics, it's not that hard, these things HAPPEN ALL THE TIME.

[Lunar]

Zogby did pretty good in 2000.

[Lunar]

J.J. is saying that it all averages out Alcon.  

So, yes, this mistake was just because of randomization.

He believes in the  0.000000002%

Think about it Alcon, let me explain it to you.

Sure, while this poll, assuming it had perfect methodology, would be 99.999999998% not to get this result.

But did you know that atoms are 99.9999999999% empty?  That's only one more 9 than this poll has.  Punch your wall ten times, did your hand go through it?  WHOOAAH you just defied statistics, it's not that hard, these things HAPPEN ALL THE TIME.

Let me explain this better to you Alcon.  You don't seem to get it.  

How many times have you been struck by lightning?  Has it been more than 1,250 times?  I thought it would be.  You'll get struck by lightning about that many times every time a poll showing McCain winning by this margin has good methodology (minus the subsample repitition issue I've mentioned earlier, so more like 700-800 times).

[Lunar]

So, how many times has lightning struck you this year, J. J.?

This year?  That's not fair, he has a 5.6% chance of not being struck by lightning this year under his system (~18 strikes a year).

And if something that is 0.000000002% is an acceptable random occurrence in J.J.'s world, 5.6% is twenty-eight million times more acceptable.

[Lunar]

Subsamples have to abide by the same laws of statistics as the overall sample.  Subsamples are not immune from criticism.

They are, however, numerous (creating higher likelihood of error) and with a higher MoE, but this result clearly shows that the poll has flawed methodology or is a once-in-a-century type of poll. I don't think you would be defending the poll so aggressively if it wasn't so favorable to McCain (and you could critique me on the inverse, but it doesn't make me wrong).

[Lunar]

Subsamples have to abide by the same laws of statistics as the overall sample.  Subsamples are not immune from criticism.

If this just a pole of the 18-24 year olds, I'd probably agree.  Newsflash, it isn't.

If you've paid attention, newsflash, I'm the only guy that defends you somewhat when it comes to this issue.  Polls with dozens and dozens of samples make outliers d
ozens and dozens of times more likely.

it *IS UNFAIR* to pick through a poll and find things that seem a little off to validate pre-conceived beliefs about the poll.  Every poll with have something a little off.  No poll with a statistically sound methodology will have McCain leading the 18-24 year olds like this poll has, given the 100 person sample size.  Well, maybe one poll every thousand years.

I'm not a fan at looking at subsamples either.  But if they show something that is basically impossible for them to show, it reveals a methodological weakness.  If McCain was winning 5 of the 8 black people (63%) in an Iowa poll or whatever, that'd be no big deal.  But we can look at sample size and account for the normal error that is involved with looking at subsamples and adjust our criticism to account for the reasons why we shouldn't critique subsamples under most situations.  

I agree with you that, in general, criticizing subsamples is bad.  Of course they average out in any good poll and some in every poll with appear off.

But this poll is one in a gazillion or it has flawed methodology.  You can't dismiss this fact under the above rule-of-thumb.  Subsamples are generally useless, but rarely they are incredibly revealing, and this is one of those cases.  This is partly why polls publish their subsamples, so we can understand the inner workings of the poll a little bit better.

[Lunar]

Doesn't J. J. frequently argue polls don't matter because they supposedly oversample Democrats or whatever?

Although he is hypocritical, I think it's fairer to critique weighting samples (where you ignore your SRS and weight  it to pre-conceived beliefs about what the election will actually look like) than randomized selections of subsamples.

I mean, he's still wrong times five trillion, but that's not why.

[Lunar]

ZOGBY SUNDAY: 'In single day of polling, it was 49% to 46% in favor of Obama'... Developing...

He doesn't usually do single days, does he?

What a buffoon.  But his margins are about the RCP average, which isn't coincidental in my book.  He won't have an awful year in his final prediction.

[Lunar]

Daily Tracker Table - October 26, 2008

Poll Name	Obama	McCain	Margin	Change	Observation
Zogby	49.4%	44.1%	O+5.3%	M+4.3%	Who knows - it's Zogby!  Seriously though, a huge Obama sample did fall off today.
Rasmussen	52.46%	44.00%	O+8.46%	O+0.57%	Another good Obama sample.
Battleground	49%	46%	O+3%	NC	No polling done today.
Hotline	50%	42%	O+8%	O+1%	A good Obama sample should fall off tomorrow.
R2000/DKos	51%	40%	O+11%	M+1%	Ditto.
Gallup
Expanded	52%	43%	O+9%	O+1%	I have some real commentary on Gallup that I may post later.  It's now been over a week that the so-called "Expanded" model picks up less likely voters than the "Traditional" model.
Traditional	50%	45%	O+5%	M+2%
IBD/TIPP	46.5%	43.3%	O+3.2%	M+0.7%	Hard to tell.
NOT RELEASED
ABC/WP	52%	45%	O+7%	M+2%	I looked at the samples here yesterday, and it was pretty clear a strong Obama sample moved off today.
POLLS AVERAGE	50.17%	43.55%	O+6.62%	M+0.82%	Note: I do include Battleground in this average, even though it didn't poll yesterday or the day before.  If you wish not to, just remove and re-round.

In all honesty, but for Rasmussen, I'd be saying McCain is closing.

Average of Rasmussen, Hotline, Gallup traditional, and IBD is McCain %+0.2825 today.  How are you viewing this, honestly?

[Lunar]

I'll wait for Gallup and TIPP, but this have been the tightening I was talking about yesterday.  In terms of trends, McCain supporters should be very happy about any shift to McCain on R2K, because of the weighting.

So now you start looking at R2k as if it has value?  Because I tried to include it in my daily update just for that reason (trends) and you forced me to stop by aggravating me about it every day.  I think that's highly hypocritical and if R2K showed a trend in the opposite direction, you wouldn't give it the time of day.

[Lunar]

I'll wait for Gallup and TIPP, but this have been the tightening I was talking about yesterday.  In terms of trends, McCain supporters should be very happy about any shift to McCain on R2K, because of the weighting.

So now you start looking at R2k as if it has value?  Because I tried to include it in my daily update just for that reason (trends) and you forced me to stop by aggravating me about it every day.  I think that's highly hypocritical and if R2K showed a trend in the opposite direction, you wouldn't give it the time of day.

This is J.J.

Anyway, he fought tooth and nail to make me remove R2K from my list in the same manner that I removed Zogby and today it provides valuable insight to him?! AHGHUAG

[Lunar]

IBP/TIPP has, um, 'corrected' their process:

You must be logged in to read this quote.

Clicky

I personally think he's making some amateur pollster mistakes in 'correcting' his poll.  It's a little Zogbyian, but make your own decisions.

So J.J. can stop hiding from lightning and admit he was wrong and we were right?

[Lunar]

Another question is what TIPP is doing to CAUSE this systemic error.  Could this mistake be something that impacts other parts of the poll that might not show up in the crosstabs?  Could the "fixing" bleed over into other crosstabs?

[Lunar]

We said there was a methodological error and you consistently, throughout many posts, continued to deny this.  You were wrong on this fact (even if you maintain that it "doesn't matter").  There was an error, the poll corrected it through amateur means, and then they started to become more favorable to Obama.  You defended the poll's legitimacy even though their own pollster can't defend their model that they were using at the time.  You refused to believe that the poll had an inherent flaw, insisting that we shouldn't look at such things.  C'mon man!

[Lunar]

I gave it 9:1 odds of showing a lead, Sam only said it'd be at least tied.

F***cking Drudge giving it the headline treatment when it's a ZOGBY, SINGLE-DAY poll...  I swear, sometimes HuffPost is less ridiculous.

[Lunar]

F***cking Drudge giving it the headline treatment when it's a ZOGBY, SINGLE-DAY poll...  I swear, sometimes HuffPost is less ridiculous.

Sometimes?

Huffington Post is infinitely more reasonable than Drudge Report.

Yeah their in-depth researched article analyzing whether Palin's lipstick is tattooed on was really groundbreaking.