Assessing 538's performances in 2018

[Antonio the Sixth]

Part 3 of a 5-part series of post-election analysis:

Part I: Composition of the House by Region, 1912-2018
Part II: Senate Elections Model - Post-2018 Update
Part III: Assessing 538's performances in 2018
Part IV: Forthcoming
Part V: Forthcoming


Hey all. As the new Congress begins, I'm trying to wrap up the various aspects my analysis of 2018 results. I hope this analysis doesn't come too late, but I had to wait for official results to be finalized before starting to crunch the numbers since I didn't want to have a map full of errors (in truth, there will be one error, since I'm going to rely on the likely fraudulent NC-9 results, but that one won't be resolved for another 6 months, so I have little choice).

Anyway, some of you might remember my previous post-election assessments of 538.com's accuracy (warning: the earlier posts in this thread contain some very broken English - enter at your own risk and peril). Naturally, I decided to do the same for this year. 538 themselves did a very thorough postmortem about their forecast's performance, which I encourage you to check out. But I think there are a couple thing I can add - chief among which, maps, which I know are dear to this forum.


The House

First, let's take a look at how far off 538's forecasts were for the House. The House forecast was very much the site's flagship this year, and, as Nate noted, it did remarkably well. Still, it had some pretty large misses. Below is a map showing where, to what extent, and in which direction the forecasts missed. I'm only including districts that were assumed to be competitive (ie, those where the forecast showed less than a 95% chance of victory for either party) or those that turned out to be competitive (ie, those where the margin was within 10 points). Red indicates districts where Democrats outperformed their forecast, and blue districts where Republicans did. The shade indicates the amount of the overperformance (with increments of 1 percentage points until 6, and of 2 above).



Overall, Democrats very slightly beat expectations in these competitive districts, doing 1.1 points better on average than the model assumed. This is all the more significant because Democrats' national popular vote margin was actually 0.2 points behind 538's predictions, which suggests that Democrats' structural disadvantage wasn't as big as initially thought. Either way, we are talking about minor difference, and the main takeaway here is that 538 came remarkably close to being perfectly unbiased.

There are some interesting regional differences as well: 538 underestimated Democrats most in the South (where they did 2.3 points better than expected on average, largely thanks to Texas), while being almost perfectly on the mark in the Midwest (overestimated Democrats by 0.1 points), the Northeast and West being on par with the country as a whole. However, as the map shows, there is still considerable diversity from district to district. Despite their upsets in the South, for example, Democrats had serious disappointments in most Florida districts. Even within some States, Democrats had both major over- and underperformances: see NJ, MN, IA, and even CA, where Ami Bera's race was way closer than it was supposed to be. In sum, it's hard to make out a clear pattern, although the biggest Democratic upsets tend to be in suburban and/or minority-heavy districts, while their biggest failures are generally in whiter areas.

How did specific components of the model perform? The 538 model is made up of 4 main building blocks: polling averages, CANTOR (an algorithm that uses polls in similar places), fundamentals (all the non-polling factors like PVI, incumbency or fundraising), and expert ratings. Which component of the forecast came closest to nailing each district? That's what the next map shows (using yellow for polls, blue for CANTOR, red for fundamentals, green for experts).



I honestly don't know what to make of this map - I really can't find any clear pattern. Overall, polls were the most unbiased nationwide, underestimating Democrats by 1.2 points. They are closely followed by fundamentals (overestimated them by 1.4), with CANTOR and experts further behind (both underestimated Democrats by 3 points on average). Still, it is remarkable that each component proved to be the closest call in a significant number of districts (I'm especially surprised to see that CANTOR, which was designed as a mere complement for underpolled districts, ended up being spot-on in so many races, including closely watched ones like ME-2, WV-3, or TX-23). So, 538's method of averaging them ultimately proved to be the best call.

Here is the full breakdown of average forecasts by region and by component of the model, for those who want to dig into it:

[Antonio the Sixth]

The Senate

Let's now take a look at 538's Senate model. Here, the picture is extremely clear from the outset when you look at the over/underperformance map:



Across the board, Democrats did significantly worse than the model assumed they would. Overall, they underperformed by 2.2 points, nearly twice the amount by which they overperformed in the House (if you want to laugh a bit, my guess before the election was that either Democrat would overperform in the Senate or they'd underperform in the House, so feel free to throw an egg on my face). This discrepancy between House and Senate is strange, but it's interesting to note that it's the mirror image of 2010, when Republicans had flopped massively in the Senate while winning beyond their wildest dreams in the House. Maybe there's something about voters in a President's first midterm splitting their vote between House and Senate? Let's see if this pattern holds going forward.

There's also a clear and obvious regional bias to how this Republican overperformance panned out. In the Midwest, Democrats were a whopping 3.9 points below their expected scores on average, a serious and systematic miss. In 5 of the 9 races in this region, the underperformance exceeded 5 points, and, in Indiana and Michigan, it reached almost 8 points. Clearly, there is something about the Midwest (or at least the "near" Midwest or "Rust Belt") that pollsters, experts and traditional political metrics all seem to have missed, both in 2016 and in 2018. Until they figure it out, Democrats should feel nervous about these States even when polls show them ahead.

The Senate model overestimated Democrats in the other regions as well, but by more reasonable amounts (only 1.1 in the South and 1.5 in the Northeast, nothing to panic about). There were major Republican upsets in each of these regions (TN in the former, RI and DE in the latter, WA and HI in the West), but there also were some solid Democratic overperformances to counterbalance them (NY, the MS special election, and to a lesser extent a trio of Latino-heavy Southeastern States). Still, the only seriously competitive race that Democrats beat their polls on were Texas and Nevada. The latter has a decade-long history of Democratic overperformance, while the latter marks its second cycle of the same pattern. If it continues, it should be a source of concern for Republicans. Finally, Kevin De Leon outperformed his polls by almost 7 points, although there are many different ways to interpret that.

What component of the model performed better? Here as well, the answer is a lot simpler than for the House:

Senate polls were remarkably good this cycle, providing the best estimate of 20 of the 35 races. In fact, the polling component of the model had no bias at all: Republicans did on average only 0.1 points better than it would have assumed (although they still did 2.9 points better in the Midwest). Where the polls failed, expert ratings provided the best alternative, providing the best estimate for 10 more races. Those include some "T***py" states where polls were too friendly to Democrats, like WV, IN and MO, but also some where the opposite was true, like NY, NM and NV. That said, experts as a whole were severely biased toward Democrats, by a full 2.8 points. CANTOR was actually much less biased, with only 0.9, but it only got it closest in 3 States. This is a great example to show that lack of bias and accuracy are two very different things. Finally, one component of the model that performed terribly on both accounts: fundamentals. Aside from only providing the closest estimate in Mississippi (for both the regular and special election), they were ridiculously biased toward Democrats - by a full 6.4 points. In the Midwest (but also the Northeast), this bias reached a whopping 9(!!) points. There are a full 8 States where Republicans did over 10 points better than fundamentals said they would. In short, there is no question that throwing fundamentals out of the model completely would have made it a lot more accurate across the board. As the 538 people themselves pointed out in their postmortem, they have some serious thinking to do about how they calibrate their fundamentals, perhaps especially the incumbency advantage. It's clearly not what it used to be.

Full breakdown again:

[Antonio the Sixth]

Governorships

I can't go into as much detail with 538's governors model, because unlike the other two it was pretty minimalistic in its presentation - you can't go in and look at the details of each component of the model. Still, let's take a look at the full model and what it missed:



Here we see something we hadn't quite seen before: lots of pretty serious misses in both directions without a clear pattern. Out of 36 gubernatorial races, there are 7 races where Democrats beat expectations by over 4 points, and 7 races where Republicans did the same (compare with the Senate models, where Democrats did so in only 2 races but Republicans in 10). In fact, these errors are so well-balanced that the model as a whole comes out looking perfectly unbiased: in the average gubernatorial race, Republicans did just a statistically insignificant 0.1 points better than the model assumed they would. This sheds a very different light on gubernatorial results that otherwise seemed pretty disappointing for Democrats. The reason for this disappointment is that three of the Republicans' overperformances proved to be decisive, including 2 of them in the two biggest competitive States (Florida and Ohio). This alone explains why the share of the population governed by Democrats ended up being so much lower than expected (53% instead of 60%). Meanwhile, the only error that made the difference for Democrats was in the small Kansas, barely enough to make up for the loss of Iowa. Democrats also did slightly better than expected in Georgia, but still fell short.

Looking at it by region, the Governors is the only one that didn't overestimate Democrats in the Midwest - in fact, it overestimated them by 0.4 points. This makes the losses in Ohio and Iowa all the more painful, since they seem to have been due to bad luck rather than anything systematic. Not only did Democrats pull the upset in Kansas, they also far outran the model in IL and NE, and were almost exactly on the mark throughout the Upper Midwest. It was really only those two States where they bombed. Democrats also very slightly overperformed in the Northeast, which was the only region without a serious miss for the model. The only noticeable Democratic underperformance was the South, where they were dragged down by Florida but also by a trio of State in this "Outer South" region where Democrats have collapsed in the past decade. In the West, finally, Republicans did ever so slightly better than predicted (mainly due to being massively underestimated in Wyoming), but the bottom line is that the expected winner won by a bigger margin than expected everywhere except in Hawaii.

Unfortunately, as I said, I have no way to break down the various components of this model. Still, I decided to show you what the "lite" version of the forecast was predicting (I've labeled it as "Polls" here but it should really be "Polls+CANTOR"). As you can see, there isn't much difference between this and the full model, or between either model and the eventual results.

[Antonio the Sixth]

Is there interest in this? I don't want to do the work for the whole series (this is several days' worth for every installment) if nobody's reading it.

[Politician]

Why were 538's Senate fundamentals so bad?

[Antonio the Sixth]

Why were 538's Senate fundamentals so bad?

I sure wish I knew. It's especially bizarre because fundamentals seem to have done decently in the other two models. Fundamentals in general were the more pro-Democratic component in every model (so it's not just incumbency screwing things around, since most incumbents in competitive House and Governor races were Republicans). It's just for some reason they were way further off in the Senate model. Clearly incumbency advantage needs to be reassessed, but there might be other issues to address, like whether a party's generic ballot advantage matters as much for the Senate as it does for the House. It's possible that a lot of traditionally disgruntled Republicans decided to vote Democrat in the House, but for a Republican Senator. I find this logic ridiculous, but voters are dumb as we all know.

[Xeuma]

Is there interest in this? I don't want to do the work for the whole series (this is several days' worth for every installment) if nobody's reading it.

I'm reading! I just can never think of anything intelligent enough to say in response.

[OneJ]

Is there interest in this? I don't want to do the work for the whole series (this is several days' worth for every installment) if nobody's reading it.

I'm reading! I just can never think of anything intelligent enough to say in response.

Same.

EDIT: As a matter of fact, since the model had turnout models for the House and Senate races, I was thinking of doing a comparison between the model’s projections and the actual results. Btw, great work!

[Pericles]

Is there interest in this? I don't want to do the work for the whole series (this is several days' worth for every installment) if nobody's reading it.

I'm reading! I just can never think of anything intelligent enough to say in response.

Same.

[Antonio the Sixth]

Reading without adding something is totally fair! Just a heads up that you're interested every once in a while helps me stay motivated. But not I know, so no worries, I'll keep it up.

EDIT: As a matter of fact, since the model had turnout models for the House and Senate races, I was thinking of doing a comparison between the model’s projections and the actual results. Btw, great work!

Feel free to add your analysis to this thread! I haven't looked at turnout but I'd love to see what patterns you find. I'm especially curious if there's a correlation between where turnout was higher than expected and which party did better.

[TheSaint250]

Nice work!

I just plugged in the numbers from the Deluxe Model for the 21 Trump-Obama districts and 3 Trump-Romney districts that elected Democrats in 2016. Republicans did better in 18 of the 24 and Democrats in the remaining 6. The results are below, with the numbers representing the D-R/R-D margins compared to the D-R/R-D margins in the FiveThirtyEight Deluxe Model.

Where Republicans were underestimated:

AZ-01: +1.0
IA-01: +4.6
IA-02: +5.7
IA-03: +1.5
IL-12: +1.8
IL-17: +6.4
ME-02: +3.1 (1st Preference)
MN-01: +1.8
MN-02: +0.8
MN-07: +8.6
NH-01: +0.2
NJ-02: +7.0
NJ-03: +0.5
NJ-05: +3.5
NY-18: +8.9
NY-21: +2.6
PA-08: +1.8
WI-03: +4.8

Where Democrats were underestimated:

MN-08: +0.9
NV-03: +2.1
NY-01: +7.2
NY-02: +2.0
NY-11: +11.6
NY-19: +2.8

[JG]

Is there interest in this? I don't want to do the work for the whole series (this is several days' worth for every installment) if nobody's reading it.

I'm reading! I just can never think of anything intelligent enough to say in response.

[100% pro-life no matter what]

Polling always seems to heavily underestimate Republicans in Tennessee, and I'm not really sure why.  In 2016, Tennessee actually had a bigger polling miss than the Midwestern states that got all the attention, with most polls suggesting a Blackburn-sized Trump win, when he actually won by 26.  Additionally, all polls I had seen had a controversial pro-life ballot initiative down in 2014, when it won by 6 points on Election Day.  Likewise, heading all the way back to 2006, every Senate race viewed as a tossup in polling went Democratic except for Tennessee.

This year, some people even thought Bredesen had a chance, and only a few people thought Blackburn would win by more than 5.  People laughed at a FOX News poll for having Lee up by 19 when most polls had him up low-double digits.  He won by 21.

Any ideas why it seems to be impossible to poll TN?

[Antonio the Sixth]

Nice work!

I just plugged in the numbers from the Deluxe Model for the 21 Trump-Obama districts and 3 Trump-Romney districts that elected Democrats in 2016. Republicans did better in 18 of the 24 and Democrats in the remaining 6. The results are below, with the numbers representing the D-R/R-D margins compared to the D-R/R-D margins in the FiveThirtyEight Deluxe Model.

Where Republicans were underestimated:

AZ-01: +1.0
IA-01: +4.6
IA-02: +5.7
IA-03: +1.5
IL-12: +1.8
IL-17: +6.4
ME-02: +3.1 (1st Preference)
MN-01: +1.8
MN-02: +0.8
MN-07: +8.6
NH-01: +0.2
NJ-02: +7.0
NJ-03: +0.5
NJ-05: +3.5
NY-18: +8.9
NY-21: +2.6
PA-08: +1.8
WI-03: +4.8

Where Democrats were underestimated:

MN-08: +0.9
NV-03: +2.1
NY-01: +7.2
NY-02: +2.0
NY-11: +11.6
NY-19: +2.8

I'll try to do the same for Clinton-Romney and other Clinton-Republican districts and post the results here.

Thanks! So yeah, this seems to confirm my intuition that Republicans generally overperformed predictions in T***p-friendly areas, and (presumably) underperformed in Clinton-friendly ones. This also emerges clearly from the Senate map (but, interestingly, not from the gubernatorial map. Anyway, I'll have more on recent presidential trends and how they affected 2018 results very soon.

Any ideas why it seems to be impossible to poll TN?

I really don't know. There are a few States that seem to be Just Like That, with polls always missing and always in the same direction. Nevada has received a lot of attention on this forum, but California and New York have even more egregious histories of polling misses on the Democratic side. On the Republican side, Tennessee and Kentucky seem particularly problematic. Maybe it has something to do with polls not getting their racial subsamples right? Like, the States where Republicans tend to overperform are typically whiter, and vice versa. But this is far from a universal rule, so I can't give you a definitive answer.

[Associate Justice PiT]

Polling always seems to heavily underestimate Republicans in Tennessee, and I'm not really sure why.  In 2016, Tennessee actually had a bigger polling miss than the Midwestern states that got all the attention, with most polls suggesting a Blackburn-sized Trump win, when he actually won by 26.  Additionally, all polls I had seen had a controversial pro-life ballot initiative down in 2014, when it won by 6 points on Election Day.  Likewise, heading all the way back to 2006, every Senate race viewed as a tossup in polling went Democratic except for Tennessee.

This year, some people even thought Bredesen had a chance, and only a few people thought Blackburn would win by more than 5.  People laughed at a FOX News poll for having Lee up by 19 when most polls had him up low-double digits.  He won by 21.

Any ideas why it seems to be impossible to poll TN?

     TN also seems to be remarkably inelastic in its voting habits. I remember in 2006 it was the only competitive Senate seat that the GOP won. Sure there was controversy over the "call me" ad, but this was a huge national wave. So big in fact that the GOP failed to pick up a single CD. That one advertisement could overpower such a headwind on its own is an absurd concept. Tennessee just really isn't open to voting for Dems.

[wesmoorenerd]

Reading without adding something is totally fair! Just a heads up that you're interested every once in a while helps me stay motivated. But not I know, so no worries, I'll keep it up.

EDIT: As a matter of fact, since the model had turnout models for the House and Senate races, I was thinking of doing a comparison between the model’s projections and the actual results. Btw, great work!

Feel free to add your analysis to this thread! I haven't looked at turnout but I'd love to see what patterns you find. I'm especially curious if there's a correlation between where turnout was higher than expected and which party did better.

Top notch analysis! 4 realz. We need more effortposts like this.

I think Iowa is one of the more interesting cases of the year. The fact that Finkenauer underperformed so dramatically combined with Scholten's outstanding overperformance suggests some interesting patterns regarding candidate quality. I think this lies more with the GOP than the Dems on both counts. Finkenauer and Scholten were both good candidates, but Blum was notoriously underestimated in every race he ran in and King got the worst press of pretty much any candidate in the cycle.

Candidate quality matters fam

[Antonio the Sixth]

Reading without adding something is totally fair! Just a heads up that you're interested every once in a while helps me stay motivated. But not I know, so no worries, I'll keep it up.

EDIT: As a matter of fact, since the model had turnout models for the House and Senate races, I was thinking of doing a comparison between the model’s projections and the actual results. Btw, great work!

Feel free to add your analysis to this thread! I haven't looked at turnout but I'd love to see what patterns you find. I'm especially curious if there's a correlation between where turnout was higher than expected and which party did better.

Top notch analysis! 4 realz. We need more effortposts like this.

I think Iowa is one of the more interesting cases of the year. The fact that Finkenauer underperformed so dramatically combined with Scholten's outstanding overperformance suggests some interesting patterns regarding candidate quality. I think this lies more with the GOP than the Dems on both counts. Finkenauer and Scholten were both good candidates, but Blum was notoriously underestimated in every race he ran in and King got the worst press of pretty much any candidate in the cycle.

Candidate quality matters fam

Yeah, candidate quality is a tempting explanation for several of these upsets (other examples that jump out are WI-1, NY-27 and CA-48). However, there are almost as many cases were the "stronger" candidate (or at least the one that was widely perceived as such) severely underperformed, or conversely a "weak" candidate beat the odds: see CA-21, TX-23, NJ-2 and WV-3. So it's hard to tell an overarching story.

[wesmoorenerd]

Reading without adding something is totally fair! Just a heads up that you're interested every once in a while helps me stay motivated. But not I know, so no worries, I'll keep it up.

EDIT: As a matter of fact, since the model had turnout models for the House and Senate races, I was thinking of doing a comparison between the model’s projections and the actual results. Btw, great work!

Feel free to add your analysis to this thread! I haven't looked at turnout but I'd love to see what patterns you find. I'm especially curious if there's a correlation between where turnout was higher than expected and which party did better.

Top notch analysis! 4 realz. We need more effortposts like this.

I think Iowa is one of the more interesting cases of the year. The fact that Finkenauer underperformed so dramatically combined with Scholten's outstanding overperformance suggests some interesting patterns regarding candidate quality. I think this lies more with the GOP than the Dems on both counts. Finkenauer and Scholten were both good candidates, but Blum was notoriously underestimated in every race he ran in and King got the worst press of pretty much any candidate in the cycle.

Candidate quality matters fam

Yeah, candidate quality is a tempting explanation for several of these upsets (other examples that jump out are WI-1, NY-27 and CA-48). However, there are almost as many cases were the "stronger" candidate (or at least the one that was widely perceived as such) severely underperformed, or conversely a "weak" candidate beat the odds: see CA-21, TX-23, NJ-2 and WV-3. So it's hard to tell an overarching story.

Who exactly was the weak candidate in TX-23? GOJ was pretty strong, Hurd was just stronger.

[Antonio the Sixth]

Reading without adding something is totally fair! Just a heads up that you're interested every once in a while helps me stay motivated. But not I know, so no worries, I'll keep it up.

EDIT: As a matter of fact, since the model had turnout models for the House and Senate races, I was thinking of doing a comparison between the model’s projections and the actual results. Btw, great work!

Feel free to add your analysis to this thread! I haven't looked at turnout but I'd love to see what patterns you find. I'm especially curious if there's a correlation between where turnout was higher than expected and which party did better.

Top notch analysis! 4 realz. We need more effortposts like this.

I think Iowa is one of the more interesting cases of the year. The fact that Finkenauer underperformed so dramatically combined with Scholten's outstanding overperformance suggests some interesting patterns regarding candidate quality. I think this lies more with the GOP than the Dems on both counts. Finkenauer and Scholten were both good candidates, but Blum was notoriously underestimated in every race he ran in and King got the worst press of pretty much any candidate in the cycle.

Candidate quality matters fam

Yeah, candidate quality is a tempting explanation for several of these upsets (other examples that jump out are WI-1, NY-27 and CA-48). However, there are almost as many cases were the "stronger" candidate (or at least the one that was widely perceived as such) severely underperformed, or conversely a "weak" candidate beat the odds: see CA-21, TX-23, NJ-2 and WV-3. So it's hard to tell an overarching story.

Who exactly was the weak candidate in TX-23? GOJ was pretty strong, Hurd was just stronger.

Wasn't Hurd considered to be Unbearable Titan caliber? I never heard GOJ being considered particularly weak or particularly sttong.

[Skye]

Keep this up. Make sure to finish it and send it to 538 so they can see it. This is the kind of content this site needs.

[IceSpear]

Why were 538's Senate fundamentals so bad?

Some of us called them out as trashdamentals and were ridiculed for it.

MUH incumbency! But Nevada polls overestimating Republicans 10000000000 times doesn't matter!

[Antonio the Sixth]

Why were 538's Senate fundamentals so bad?

Some of us called them out as trashdamentals and were ridiculed for it.

MUH incumbency! But Nevada polls overestimating Republicans 10000000000 times doesn't matter!

You were rightly ridiculed for being overconfident about your predictions, as you still are today. Just because you ended up being right doesn't automatically mean your thinking was sound.

[IceSpear]

Why were 538's Senate fundamentals so bad?

Some of us called them out as trashdamentals and were ridiculed for it.

MUH incumbency! But Nevada polls overestimating Republicans 10000000000 times doesn't matter!

You were rightly ridiculed for being overconfident about your predictions, as you still are today. Just because you ended up being right doesn't automatically mean your thinking was sound.

I'm not sure what you're referring to here. I was "overconfident" that 538's fundamentals were trash because I KNEW they were trash and it was blatantly obvious they were trash. I was "overconfident" that Marsha Blackburn, Kevin Stitt, and Jacky Rosen were going to win because some of us actually learned a few things from the past couple election cycles. Like that factors such as polarization, the partisan lean of a state/district, the overall political environment, and the unreliability of early polls need to be given heavy weight rather than lazy outdated archaic analysis about "MUH incumbency!" or "MUH all politics is local!" or the people who go back and forth like a pendulum depending on whatever the latest poll taken 6 months before the election says. It's not like I got these predictions from throwing darts at a board and simply got lucky. The trends were pretty clear after 2014 and 2016, it's not that complicated. But people are already are making the same mistakes for 2020 (and, as my username alludes to, 2019 as well )

So considering Oklahoma and Tennessee proved my thinking right that they won't elect Democrats except in the most extreme of circumstances (i.e. pedophilia or something similar) due to how polarized and partisan they are, I don't see how my thinking wasn't sound. My theory was completely accurate, as partisan lean and polarization were clearly the ONLY reasons Republicans won those races, and nothing was going to change that. All the other stuff was just window dressing that would end up deciding what the margin of victory for the Republican would be.

[Antonio the Sixth]

I don't deny that some Atlas posters were deluded about States like Tennessee and Nevada throughout the campaign (in fact I pretty strongly and vocally agreed with you on Tennessee and and thought that Rosen would probably win), but that doesn't mean that you refusing to admit the possibility of any uncertainty about these elections outcomes is reasonable. 538 rating of Blackburn having an 80% chance or so was reasonable - maybe a better-calibrated model could have gotten to 90%, but that's still a far cry from the 100% chance that you assumed. It's even more ridiculous for Nevada, where Rosen only won by 5 points at the end of the day.

So yeah, your predictions can keep being usually factually correct, but are still bad predictions regardless on account of you underestimating the amount of uncertainty.

Anyway, I invite you to check out the new thread I made, since this sounds like something you'd have Strong Thoughts about as well.