Senate Elections Model - Post-2018 Update

[Antonio the Sixth]

Hi all,

With the Midterms and many close Senate elections coming up, I've been looking for a way to handicap those races based on something more concrete than polls (which tend to be pretty unreliable until Labor Day). I figured that, through my research methods, I actually had the tools to do the sort of statistical analysis people at 538 do. So I went and collected data for all regularly scheduled Senate elections since 1990. Then, I built a logit model with this data, seeking to predict the probability of Democratic victory (note that this is not a prediction about the margin but only about win/lose) with a few simple variables:

• The State's PVI based on the most recent Presidential election prior to the Senate election in question. So, for example, 2016 Senate results are compared to 2012 Presidential results (there are two good reasons for this: the first is that this allows me to make ex-ante predictions even for Presidential-year Senate elections; the second is that, to my own surprise, this model actually does a better job of predicting Senate election results than the ones that use contemporaneous PVIs).
• Whether it's an open seat or has a Democratic or Republican incumbent running for reelection.
• A dummy for each election year, to capture the effect of the national environment ("dummy" means that it's not really a variable: rather than having an a-priori measure of the national environment, I'm letting the model estimate its effect by looking at who won and who lost on that year).
• An interaction term allowing for the predictive power of PVI to change over time (you can interpret it as a measure of growing partisanship in Senate voting patters).
• An interaction term allowing for the predictive power of PVI to vary depending on whether it's an open seat or an incumbent running for reelection.

This model ends up correctly predicting (meaning, ascribing a >50% probability to the eventual winner) 432 out of the 482 elections in my sample - a success rate of 89%. This isn't actually as good as it sounds, since it means the model misses on average 3 or 4 races every cycle, but it still means that it gets the big picture largely right. On average, the eventual winner is given a probability of success of 84.5%.

So, what does the model say? First, yes, PVI is a big deal and has become even more of a big deal over the past quarter century. In 1990, on average, if a State with a 5-point PVI for one party had an open seat, you could give that party's candidate a 61% chance of winning the seat in a neutral year (basically a tossup). The probability rose to 71% with a 10-point PVI, and to 86% with a 20-point one. Today, however, these figures have risen to 74%, 89% and 98% respectively. In other words, you'd be safer running for an open seat that leans 5 points toward your party today than you'd be running for one that leaned 10 points in 1990. It's a pretty striking trend.

As strong as PVI is, though, incumbency is still an even more powerful force. The numbers, here, really speak for themselves: the probability for an incumbent to win reelection is an even-PVI State in a neutral year is a whopping 94%. In other words, even in 2018, you should feel safer if you're running for reelection in a neutral State than if you're running for an open seat in a State that favors your party by 10 points. Yet, these figures might still understate just how resilient incumbents are. Indeed, the interaction term between PVI and incumbency shows that, when an incumbent is running for reelection, the effect of PVI is significantly weaker. In other words, this means that incumbents in "hostile territory" get a much bigger boost than those in favorable grounds. If you're a Democrat running in New York or a Republican running in Montana, it doesn't make that big a difference if you're an incumbent or not - but if you're a Democrat running in Montana or a Republican running in New York, it makes a huge one. This means that incumbents can still be the favorites to win reelection even in places where their party is otherwise completely uncompetitive. To quantify this, we can look at what PVI you need a State to have in order for an opposite-party incumbent to have exactly 50% chance of reelection in a neutral year. The answer, even in 2018 with the effect of PVI stronger than ever before, is 18.2 points (if you go back to 1990, it was apparently 84.9(!!!!) points, but I have reasons to believe this part of the model is a bit skewed, so I wouldn't put too much stock in that).

What is the effect of the national environment, then? This is a slightly tricky question, since the year dummies don't have a directly quantifiable interpretation. One thing we can calculate is the predictions the model makes, for each year, for a hypothetical open-seat race in an even-PVI State. When we do that, the national political environment emerges as a powerful variable. In their worst year on record, 2010, Democrats had a paltry 14% chance of edging it out in an even-PVI open seat. In their best, meanwhile, they were basically a shoo-in, at 95%. There is a one very big picture amid these massive variations, though: Democrats really do seem to have a built-in advantage in Senate elections that can't be entirely explained by PVI, incumbency, or even the mood of the country. In 7 of the 14 election cycles I have data for, Democrats were clear favorites to win an even-PVI open seat, and in 3 others they were about even money. These include years where Democrats had no business being in the lead. Take an unquestionably GOP-favorable year, like 2004 - the only time in the past 26 years in which a Republican won the Presidential popular vote. On first glance, Republicans seem to have done very well in Senate elections that year, picking up 4 seats. However, when you take into account the roles played by PVI and incumbency, the probability for a Democrat to win a seat where these factors did not come to play was almost exactly 50%. In other words, although the country as a whole was R+2, the national political environment in the Senate specifically was perfectly neutral. In 2016, the Democratic candidate was expected to have a 3-to-1 shot to win an even-PVI open seat (shame there weren't any ).

There is a more sophisticated way to assess the national political environment in Senate elections, and especially to compare it to what we observe at other levels of election. To employ it, we first need to make an assumption that I hope is straightforward enough: when the national political environment is perfectly neutral, the two parties are equally likely to win an even-PVI open seat. This seems straightforward enough, although it's possible the assumption is not literally true (for example, if Democrats systematically run more high-quality candidates, we could see them win more seats even when the partisan climate favors Republicans - but since the model can't measure that it's a moot point). If we make this assumption, we can then give a point-measure of the national environment by looking how "far" you need to go from an even PVI in order to have an open seat where the odds of victory are a coin flip. For example, if, for a given year, we find that Democrats have a 50% chance to win an open seat with a R+10 PVI, we can conclude that it was a D+10 year (and vice versa). When we do that, we find that Democrats do indeed do better in the Senate than at other levels of government. For example, in 2006, while Democrats won the House popular vote by 8 points, the calculated political environment in the Senate turns out to be a whopping D+18. In 2000, with the national popular vote almost tied, the Senate environment reached D+12. In 2016, which ended as we all know, it was still D+6. Admittedly, very large Republican landslides also seem to be magnified: 1994 was R+14 and 2010 was R+10. Still, in almost all years, Democrats do better than you'd expect looking at other elections. I put all of this information (along with corresponding figures for seats with incumbents running, and dashed lines to indicate what we should see if the partisan environment were neutral) in the chart below:




For those who want to get deeper into the weeds, here are charts showing how exactly the Democratic candidate's probability of winning increases with PVI for every year and every type of seat:








Stay tuned for a prediction of 2018 races based on this model.

[Antonio the Sixth]

So, what does this model have to say about 2018?

To be able to make an actual forecast, we'd need to know what the national political environment is going to be this year - but of course, that's exactly the question we're waiting for the election to answer. Still, we have a good sense of what the plausible range of possibilities is for next year's elections, and we can use this to calculate various scenarios about where things stand. I'll focus on three of them, that I think reflect the pessimistic, neutral and optimistic visions of Democrats' prospects.

For a lower bound, we might simply make the assumption that Democrats do as well this year as they did two years ago. While 2016 was a disappointment compared to Democratic expectations, it was still a year where Democrats stood at a significant advantage, so it seems realistic that a repeat of 2016 would be their floor (no one in their right mind would, I think, claim that Democrats will do worse this year than in 2016). So, what do Democrats' chances look like in this scenario? I've mapped them below, using the standard tilt/lean/likely/safe color scheme (the cutpoints are 67%, 80% and 95% respectively).

As you can see, even in this underwhelming scenario, things don't look too bad for Democrats. Most of their Red State incumbents (including embattled ones like McCaskill and Donnelly) are clear favorites, to speak nothing of Nelson and Baldwin, who seemingly have nothing to worry about. Sinema is also already neck-to-neck with McSally for the Arizona seat. Still, this map would nevertheless entail major gains for Republicans: they seem heavily favored to unseat Manchin and Heitkamp, and have a good shot at three more incumbents. Overall, this map indicates that they could plausibly net anywhere between 1 and 5 seats (the average outcome has them net 2.2), expanding their Senate majority enough to have an easy time confirming T***p's nominees for the remainder of his term.

Of course, most of the data we've seen so far indicates that Democrats will do considerably better than in 2016. What if we assume a more markedly Democratic partisan environment? A good baseline for a solid but not overwhelming Democratic partisan climate is 2012. That year, Obama won reelection by 4 points - however, the partisan climate of Senate elections implied by the model is a robust D+12. So, if Democrats have wind in their sails as much as they did with Obama's reelection, we should expect a similar performance. Here is what the map looks like then:

All but two Democratic incumbents now seem heavily favored, as is Sinema. Heller is now just barely ahead, his race basically a coin flip (as is Heitkamp's). Manchin, however, still seems in a tough spot. Finally, two new Atlas-blue States begin entering this election's radar as potential longshots: Texas (unsurprisingly) and Utah (obviously a fluke due to the weird 2016 results - thanks McMullin!). Overall, the realm of realistic possibility ranges from Republicans netting 1 seat to Democrats netting 2, with an average outcome of no change. In other words, Republicans are slight favorites to retain their Senate majority, but not much beyond that.


Finally, what if this is the blowout that many special election results (and some generic ballot polls) suggest? The most Democratic election year for the Senate in my sample, according to the method used earlier, was 2006, with a D+18 apparent national environment. 2006 is also an obvious parallel, as a the most recent example of a midterm election featuring an unpopular Republican President, so a repeat of it seems like a pretty conceivable scenario. What would 2018 prospects look like then?

Now, no Democratic incumbent would even need to break a sweat about their reelection except for Heitkamp and Manchin (who, amazingly, even in this landslide scenario is still nothing more than even money to survive). Heller would also turn into a heavy underdog, while Sinema would consolidate her advantage. Cruz also starts needing to be concerned about his prospects (as does, apparently, freaking Mitt Romney - again, thanks McMullin!). The situation is so good for Democrats that they just barely have a shot in titanium-R States like TN, MS and NE. Literally only Wyoming remains in the Safe R column. Overall, if this is the partisan climate we find ourselves in, it is still a distinct possibility that Republicans net 1 seat. Their downside, however, is far bigger: they could plausibly lose as many as 4 and be reduced to a 47-seat minority. The average outcome now has Democrats netting 2.1 seats, exactly what they would need to take control of the chamber.


The limits of this model are obvious. Since it includes only "fundamentals", it has no way of capturing factors specific to each race, such as candidate quality or the peculiarities of a particular state. This is why the model gives almost no chance to Scott and Bredesen, for example, while seriously overestimating those of Morrisey and of whatever sacrificial lamb is running against Romney. However, I still think this perspective is incredibly valuable. As I said at the beginning, polls before Labor Day don't have a very good track record, and afterwards they tend to converge to where the fundamentals are. Thus, even if the polls might show Tennessee and Florida to be tossups right now, we can make an educated guess that partisan gravity will eventually pull them back in opposite directions. I'd be curious to revisit these maps in a couple months and see if the polls will still contradict them as much as they do right now.

[Zaybay]

Very good work! The map is almost the exact same as the one counting for approvals. And Heller has finally become the Unbeatable Titan we always knew he was.

[Antonio the Sixth]

Very good work! The map is almost the exact same as the one counting for approvals. And Heller has finally become the Unbeatable Titan we always knew he was.

I guess. Personally I still think he goes down, though (and it's rational to think so if you assume 1. it's probably going to be a 2012-like year than a 2016-like one, and 2. Heller is weaker than your average incumbent because of all the baggage he carries).

Any other thoughts or questions?

[Antonio the Sixth]

Bump in case this actually gets noticed this time.

[Antonio the Sixth]

Final bump...

[America Needs a 13-6 Progressive SCOTUS]

Interesting

[mencken]

I told y'all that if you crunch the numbers, Arizona and Florida should be much safer for the Democrats than they currently are.

I take slight issue with North Dakota and West Virginia. I do not think we can obtain meaningful predictions regarding these races, unless there is historical precedent in your model for incumbents in states won by the opposite party President by >35 points running for re-election?

[Amenhotep Bakari-Sellers]

This scenario with TX or TN can happen

[Antonio the Sixth]

I told y'all that if you crunch the numbers, Arizona and Florida should be much safer for the Democrats than they currently are.

I take slight issue with North Dakota and West Virginia. I do not think we can obtain meaningful predictions regarding these races, unless there is historical precedent in your model for incumbents in states won by the opposite party President by >35 points running for re-election?

The problem might be that, by its very nature, the model assumes a linear effect of PVI. That is to say, moving a State's PVI from Even to R+10 shifts the odds by the same amount (in the logit scale) as moving it from R+30 to R+40. There's good reason to believe that that's not the case, and that there's really not much difference between an R+30 State and an R+40 one. I could test for that by adding a PVI-squared variable, but I doubt it improves the model all that much. I can still try it if anyone's curious.

[💥💥 brandon bro (he/him/his)]

Very cool model! Quite impressive work.

A couple thoughts/ideas:

- Do you think it's worth adding in a covariate for Class (i.e., Class I, Class II, Class III) races? For example, this year's races are predominantly in R-leaning states, which may confound your measurement of national environment. So, it seems a little difficult to compare national environment in, e.g., 2018 with 2016 because the races are taking places in different states and have different voter pools.

- I know you're dealing with limited sample sizes here, but it seems like it would be helpful for the model to include covariates for which party is controlling the white house (or I guess in your dataset, whether a candidate is in the same party as the President) and whether or not it's a midterm race. If you looked at an interaction of these two you'd probably get an estimate of how much we can expect the national environment to turn against incumbent Presidents in midterms.

- When plotting the response curves of the win probabilities against PVI for each year, it's a little difficult to pick out a trend. It may be more helpful if you use a continuous color scale instead of the color key you're using (e.g., ranging from light to dark red; it looks like you're using R, you can probably get these colors using the rainbow() function). I guess you don't really need to do this because you already have a time series pot of the incumbent advantage, but it may make those response curves easier to digest.

- Kind of a weird question... your model is logistic, so for each race, you'll get a probability of winning for each candidate. Is there any trend over time in how certain these victories are? (e.g., is there a trend where 20 years ago your model had 51% Dem. win probabilities, which is still a highly competitive race, but something like an 80% Dem. win probability in later years, which is a less competitive race?) Sorry this is kind of a weird question and I'm probably not asking it well, so feel free to ask me for clarification.

I told y'all that if you crunch the numbers, Arizona and Florida should be much safer for the Democrats than they currently are.

I take slight issue with North Dakota and West Virginia. I do not think we can obtain meaningful predictions regarding these races, unless there is historical precedent in your model for incumbents in states won by the opposite party President by >35 points running for re-election?

The problem might be that, by its very nature, the model assumes a linear effect of PVI. That is to say, moving a State's PVI from Even to R+10 shifts the odds by the same amount (in the logit scale) as moving it from R+30 to R+40. There's good reason to believe that that's not the case, and that there's really not much difference between an R+30 State and an R+40 one. I could test for that by adding a PVI-squared variable, but I doubt it improves the model all that much. I can still try it if anyone's curious.

I'm curious and definitely think this is worthwhile. I think linear effect of PVI is probably too simple of an estimate -- squaring seems like a good idea. For one thing, even-PVI races will have a lot more resource investment than states with high PVI, so you'd expect things like national environment (and maybe incumbency?) to have a stronger effect in even PVI races than in high PVI ones.

[Co-Chair Bagel23]

Final bump...

Dale Bumpers!!

[mencken]

Also, interesting that you also picked up on an overall Democratic advantage in Senate elections. When I did the numbers, I assigned Florida 2006, Nevada/Colorado/Delaware 2010, Missouri/Indiana 2012, and Alabama 2017 to a special category. Having a gaffe-magnet for a Republican nominee cost ~9% overall, but Democrats' built-in advantage persisted (it was ~2%, small but it makes the difference with close races)

[Antonio the Sixth]

Thanks for all the comments!

- Do you think it's worth adding in a covariate for Class (i.e., Class I, Class II, Class III) races? For example, this year's races are predominantly in R-leaning states, which may confound your measurement of national environment. So, it seems a little difficult to compare national environment in, e.g., 2018 with 2016 because the races are taking places in different states and have different voter pools.

- I know you're dealing with limited sample sizes here, but it seems like it would be helpful for the model to include covariates for which party is controlling the white house (or I guess in your dataset, whether a candidate is in the same party as the President) and whether or not it's a midterm race. If you looked at an interaction of these two you'd probably get an estimate of how much we can expect the national environment to turn against incumbent Presidents in midterms.

In both those cases, these variables would be redundant given that I already have fixed-effects for each year. For example, all a Class term would do is average the effects of 1994, 2000, 2006 and 2012 for Class I, the effects of 1990, 1996, 2002, 2008 and 2014 for Class II, etc. Conversely, a midterm vs presidential term would simply average 1990, 1994, 1998, 2002, 2006, 2010 and 2014 together and vice versa. I don't think we'd learn much of interest, and it would throw off the individual year estimates completely. I don't think it would make much sense theoretically either, since there are a lot of factors that influence how a particular year goes beyond simply who controls the White House (1998 and 2002 are both years that bucked the trend in this respect, for example). I also don't understand why you're saying I need class effect to account for the fact that some classes are more Republican-leaning than others. That's what the PVI measure is for (and sure, PVI is an imperfect measure, but class is even more so.

You must be logged in to read this quote.

Thanks for the tip! I actually wasn't familiar with the rainbow() function, but I'll look it up and see if I can make something nicer with it. That said, those charts are probably too oversaturated to ever be easy to read, honestly.

You must be logged in to read this quote.

Actually, yeah, I'm not entirely sure I understand your question. As I said in my write-up, the effect of PVI has definitely become stronger, and that means that, yeah, holding PVI constant, in an open seat during a neutral year, a race in 2018 is going to be less competitive than a race in 1990 (for example, to reuse the probabilities I cited in my writeup, a D+5 State went from 61% chances of Democratic win in 1990 to 74% in 2018, and a D+10 one from 71% to 89%). However, the opposite is true if you consider races where an incumbent is running in a State with a hostile PVI - those races have become more competitive, not less.

You must be logged in to read this quote.

You're right, it's definitely worthwhile. I'll get to work on it!

As for incumbency, I've already interacted it with PVI and what I find is that it has a stronger impact the more hostile PVI is (meaning, for example, Manchin and Heitkamp get a bigger incumbency boost than Schumer or Harris, while conversely Collins and Heller get a bigger boost than Barrasso and Shelby).

[Attorney General, LGC Speaker, and Former PPT Dwarven Dragon]

How does the model perform in 2014? I ask because that was the year that all the pundits said to disregard PVI because MUH SUPER POWERFUL DEMOCRATIC INCUMBENTS, but then incumbency ended up being pretty meaningless for Democrats.

[Antonio the Sixth]

How does the model perform in 2014? I ask because that was the year that all the pundits said to disregard PVI because MUH SUPER POWERFUL DEMOCRATIC INCUMBENTS, but then incumbency ended up being pretty meaningless for Democrats.

Well, as it turns out, even the model overrated incumbents. It had Hagan winning (although it was basically a coin-flip, so I'm not really counting that against it) and it gave Udall a full 75% chance of hanging on. Still, it never gave any of the other red-State incumbents more than a 20% chance of surviving.

Interestingly, it also had Land a modest favorite in Michigan (68%). It might seem excessive, but keep in mind that Michigan is not very far from Iowa in terms of PVI, and Ernst won it easily.

[Antonio the Sixth]

Hi all. So, now that we have results for 2018, it's time to revisit this model and see what we can learn from this year's results going forward. Plus, some more tidbits from my model that I didn't elaborate on before.

All right, let's get started.


2018 Post-Mortem

2018 was not a very good year for the model. If you take the first prediction map out of the 3 I posted (which is the closest to the 2018 in terms of overall results since it involves an average Republican gain of about 2 seats, as actually happened), you'll see quickly that it missed 5 races: FL, IN, MO, NV and WV. That in itself is not too alarming: the average number of missed races has been 3 or 4 per cycle so far, so missing 5 is not particularly far off. What's more concerning is that some of these misses were really big ones. Florida, in particular, was a race the model refused to believe could be remotely competitive - even in the direst circumstances for Democrats, Nelson was supposed to win easily. That certainly didn't turn out that way. Conversely, the model saw Manchin as almost doomed unless Democrats were having a really good night, so him hanging on is a serious miss as well. And to a lesser extent, as noted, the model was pretty bullish on Heller's Unbeatable Titan status unless things became really dire for Republicans nationwide. So those are all notable errors.

Does that mean the model is broken? Of course not. It's just incomplete, but that's by design. The point of the model is not to maximize predictive power (that's what 538 is for, and it's doing a decent job at it). The point is to provide a baseline based on the most fundamental, macro factors of American politics. Deviations from that baseline are supposed to happen, because every race is different and every candidate is different. In other words, by seeing how often these deviations happen, we find out to what extent #CandidateQualityMatters. Clearly Manchin gets his State a lot better than most incumbents do, clearly Nelson was rusty and unprepared for Scott's challenge, and clearly Heller's sucking up to T***p in a State that voted against him wasn't such a smart strategy after all.

That said, there were some ways in which this model wasn't optimally calibrated. When I ran the calculations again, I saw that the overall accuracy measures from my previous model had deteriorated slightly, and therefore a slight retooling was needed. The main change I've ended up making was to add an interaction term between incumbency and year (ie, to allow for the incumbency advantage to change over time). Initially, this appeared superfluous, because I had captured the declining effect of incumbency through the increasing effect of PVI. However, after 2018, this appears not to be sufficient anymore. Not only has incumbency advantage declined relative to PVI, it has also declined relative to the national environment. Therefore, the bottom line of this retooling is that incumbency still matters, but less and less so.


Updated Model

So, what do we make of this updated model? The bottom line is the same, but some numbers have shifted at the margins. The model now called 462 out of 519 races right (still an 89% success rate) and gave on average a 84.3% chance of success to the eventual winner, virtually unchanged from before. Of course, that's with one parameter added to the model, so the point still is that it takes more modeling to get to it.

The model now estimates slightly larger effects of PVI earlier in the period and slightly smaller ones at the end of it. An open-seat candidate running in a neutral year in a seat with a 5-point favorable PVI had a 62% chance of victory in 1990 and a 72% one in 2018. With a 10-point favorable PVI those chances grew from 73% to 87%, and for a 20-point favorable PVI from 88% to 98%. So straight-ticket voting still has increased considerably. The incumbency advantage, meanwhile, has diminished too (as mentioned earlier, that's the main addition of the new model). Now, an incumbent running in an even-PVI State in a neutral year had a 95% chance of being reelected in 1990, but only an 88% one in 2018. The combined effect of this growing power of PVI and shrinking incumbency advantage is that the PVI that an incumbent needs to have a 50/50 shot at reelection in a hostile territory dwindled from 74.7 points(!!) to just 14.8. Note that that latter number is 3 points lower than it was in the estimate I came up with pre-election, which does indeed suggest that my previous model overrated the incumbency advantage.

Estimates of the effects of the national environment of a given election cycle haven't changed too much, except from the last one. The model I calculated earlier had interpreted 2016 as a pretty good year for Democrats, one where they had a 75% chance of winning an open seat in an even-PVI State and where the Democratic partisan advantage was about 5.8 points. My new estimate still suggests that Democrats had an advantage that year, but a much slighter one. The Democratic chance of victory in neutral conditions was down to 65%, and the advantage just 3.4 points. This estimate is a lot more in line with the Presidential popular vote, so I'm inclined to believe it over my previous one.

How did 2018 turn out? According to the model, it was still a solid year for Democrats: they would have had an 81% chance of winning an even-PVI open seat, and enjoyed a 7.6 advantage. That's a strong result by any standard, again more or less in line with the House popular vote. However, it is slightly less than the House popular vote, which is surprising since Democrats tend to overperform the House popular vote in Senate elections, especially in good years for them (the only times Republicans did better in the Senate than in the House were 1990, 1994, 1996, 2010 and 2014). So this does suggest that Democrats' 2018 performance was slightly below what one might have guessed. Democrats' performance was worse than in 2012, 2006, and even 2000 (when they won fewer seats but had to face off against many Republican incumbents). Still, apart from those years, it was better than any other except 2008 or (depending on the metric used) 1992. In addition, because Democrats had so many incumbents up for reelection, they bore the brunt of the shrinking incumbency advantage.

Here's what the partisan advantage chart looks like now:

(black=open seat, red=dem incumbent, blue=rep incumbent)


And here's what the model ended up "post-dicting" for 2018:

So the model has managed to more or less justify IN, MO and NV ex post facto (those are all tossups) but is still baffled that Manchin survived and Nelson went down.


More to come...

[Xeuma]

This is seriously impressive, good work!

Can I ask how you go about the actual predicting? Like with an Excel sheet, pen and paper?

[Antonio the Sixth]

This is seriously impressive, good work!

Can I ask how you go about the actual predicting? Like with an Excel sheet, pen and paper?

Thanks!

But no, this goes way beyond what Excel can do, let alone what I could calculate with pen and paper. I use a statistics program called R to run a logit model and then use it to make predictions about all races.

[Gustaf]

Interesting. I never got into R myself (I used Stata).

An interesting thing you might consider is to assume your residuals reflect candidate quality and see if they seem to be steady over time or not when the same candidates run again.

[Antonio the Sixth]

Interesting. I never got into R myself (I used Stata).

An interesting thing you might consider is to assume your residuals reflect candidate quality and see if they seem to be steady over time or not when the same candidates run again.

I used Stata at Sciences Po too, but then at UCLA they had us move to R. Maybe it's a US/Europe thing (although Nate Silver said he uses Stata too, to my surprise).

And I was just about to look at residuals! So without further ado, let's look at...


The Biggest Upsets (1990-2018)

Which Senate candidates of the past decades kept beating their odds, either by overcoming a hostile national environment, capturing unfriendly territory, beating an incumbent, or any combination of these? As Gustaf suggested, the way to find out is to look at the largest residuals (ie, the difference between the probability of winning and who actually won). In the context of this model, all this means is finding candidates who were predicted to lose but won.

Specifically, I looked at residuals higher than .667 - ie, candidates who won despite having less than a 1 in 3 shot. There are quite a few of those, 33 in total (17 Democrats and 16 Republicans, which makes sense since the model is designed to be unbiased between parties). Looking through the list give us the races where #CandidateQuality actually seems to have mattered.

Here's the list, ranked from the mildest to the biggest upset:
- MI 2000: Debbie Stabenow (D), 33.1%
- RI 2000: Lincoln D. Chafee (R), 32.3%
- WA 2000: Maria Cantwell (D), 32.0%
- TN 2006: Bob Corker (R), 29.5%
- CO 2014: Cory Gardner (R), 28.5%
- GA 2002: Saxby Chambliss (R), 27.2%
- IN 1998: Evan Bayh (D), 26.5%
- NH 2016: Maggie Hassan (D), 23.2%
- MO 2000: Mel Carnahan (D), 22.9%
- CO 2004: Ken Salazar (D), 22.1%
- WI 1992: Russ Feingold (D), 22.0%
- TN 1994: Bill Frist (R), 22.0%
- ME 2006: Olympia Snowe (R), 21.4%
- AR 1996: Tim Hutchinson (R), 20.8%
- IL 2010: Mark Kirk (R), 20.6%
- WI 2010: Ron Johnson (R), 19.8%
- MT 2006: Jon Tester (D), 19.1%
- ME 2008: Susan Collins (R), 18.8%
- PA 1994: Rick Santorum (R), 18.3%
- CA 1992(S): Dianne Feinstein (D), 16.8%
- MO 2002(S): Jim Talent (R), 16.1%
- NE 2000: Ben Nelson (D), 12.3%
- AK 2008: Mark Begich (D), 12.1%
- MN 1990: Paul Wellstone (D), 10.1%
- WV 2018: Joe Manchin (D), 8.3%
- NC 1998: John Edwards (D), 5.4%
- FL 2018: Rick Scott (R), 4.9%
- SD 1996: Tim Johnson (D), 3.1%
- VA 2000: George F. Allen (R), 3.0%
- AR 2002: Mark Pryor (D), 2.5%
- NC 1992: Lauch Faircloth (R), 2.4%
- IL 1998: Peter G. Fitzgerald (R), 1.7%
- WV 2010(S): Joe Manchin (D), 0.7%

There are a few obvious patterns that stand out here. The first is that the majority of those upsets (22 out of 33) consisted in challengers unseating incumbents. While at first glance this might suggest that the model overrates the incumbency advantage, the truth is more complicated. Because the vast majority of incumbents do win reelection, the event of an incumbent losing is a somewhat chaotic process, where a lot of factors besides state PVI and national environment comes to play. Since the model can't account for these factors, it is reduced to slightly underrating the vast majority of incumbents who can probably survive even in very dire conditions (like Chafee or Manchin) while vastly overrating a few incumbents who are uniquely weak. We've seen it this year not only with Nelson, but also with McCaskill, Donnelly and Heller, who were all expected to hang on.

Still, it is quite clear that some candidates simply played their cards right, or had uniquely weak opponents, or benefited from an idiosyncrasy of their race. Jon Tester, Joe Manchin, Susan Collins and Olympia Snowe were uniquely good fits for their States, Paul Wellstone and Ron Johnson were uniquely good at generating grassroots enthusiasm, Mark Begich and Peter Fitzgerald faced embattled incumbents. There are some I am less sure of, but I'm sure someone more familiar with the specific race would be able to give a plausible explanation.

Finally, it's fun to see how certain seats keep coming up in this list, sometimes with the same candidate seeing the tables turned on them. Lauch Faircloth pulled a major upset in 1998 by beating Terry Sanford, then was the victim of an almost equally big one when he lost to John Edwards six years later. Similarly, Russ Feingold saw his Senate career begin with an upset in 1992, and end with another in 2010. And in Missouri, dead candidate Mel Carnahan unexpectedly beat an incumbent in 2000, only for his subsequently-appointed widow to even more unexpectedly go down in defeat a mere two years later. This goes to show that #CandidateQuality is not so much an intrinsic attribute of the person running, but more often a conjunction of circumstances that require the right person at the right time.

[Antonio the Sixth]

The Partisan Structure of the Senate

And finally, let's conclude this overview by taking a hard look on what the model can tell us about partisan advantage in the Senate. It's common knowledge at this point that the Senate generally favors Republicans, since they do better in the small States that it overrepresents. However, quantifying this advantage is tricky and the few attempts I've seen so far were pretty rough. I'll try to be a bit more systematic here.

One thing the model allows us to do is to go back to each Senate election and "decompose" it, taking away the effects of various components and comparing them to the actual results. In the following chart, I used the model to "predict" how many seats the Democrats would have been expected to win in a neutral year, first based solely on PVI (that is, assuming that every seat was open), and then based on both PVI and incumbency. I compared these figures with how many seats Democrats actually won.



This chart tells us what we've already seen before: Democrats tend to do better than the "fundamentals" would imply, even in relatively neutral years like 1990, 1998 or 2012. But it also quantifies just how big this effect is in terms of raw seats. In 2012 for example, Democrats won 5 more seats than they were "supposed" to. Even in 2016, they won 1 more than expected, and 3 more in 2018. Of course, they did worse than expected in Republican wave years, but even then, not as much as one might think: 3 less in 2014, 4 in 2010, and 5-6 in the "catastrophic" 1994 cycle. Compare with the 8-seat overperformances in 2006 and 2008 that paved the way for their supermajority.

And here, I'm still talking about the Democrats' overperformance compared to a baseline based on PVI and incumbency. But incumbency itself is usually a boon for Democrats. In every year except 1996, 2000 and 2016, Democrats' position improved when the incumbency advantage was factored into the prediction. This makes perfect sense, since overperforming in one election cycle means that the party will have more incumbents six years later (and it's no accident that two of the exceptions came six years after the two worst Democratic years). This is a piece of good news for Democrats: their overperformance is, to some extent, self-sustaining. This self-sustaining effect is rapidly waning with the incumbency advantage, but even in 2018 it was still worth a whopping 5 seats.


Now for the bad news. What does it say about the Democrats' Senate position that, with all the advantages I just discussed, they only managed to control the Senate for 8 out of 26 years in the recent period? What is says is, to put it mildly, really dire. I can honestly say I was surprised when I realized how bad the situation is, especially if 2016 is the new baseline for State political alignments going forward.

To fully measure it, let's forget about classes and election cycles for a second, and take a look at the Senate as a whole. What would happen if every single Senate seat was suddenly up for reelection at the same time, with no incumbents running? This is so simple to calculate that I really should have thought of it before. Since every State has exactly 2 Senators, this is incredibly simple to calculate: just take the predicted probability of winning a seat in a given State and multiply it by 2. So, for example, if a State has a 75% chance of electing a Democrats, this means it will provide on average 1.5 seats to Democrats and 0.5 to Republicans.

In the chart below, you can see the results of this calculation using State PVIs for every year since 1960. Now, a slight complication is that the effect of PVI itself has changed over time (that's one of the features of the model, you'll remember). So, for each PVI alignment (which corresponds to each Presidential year), the final outcome also varies depending on the PVI's effect (which varies monotonously upwards over the period 1990-2020). The chart below tries to display the two variables together as clearly as possible: each cluster of bars represents the PVI alignment in a given Presidential year, while, within each cluster, bars are colored differently based on the PVI's effect at various intervals during the period. You'll see however than PVI alignment is a bigger factor than PVI effect.



So, here we get to the bottom line, and it's not good for Democrats. Under no modern political alignment, going as far back as JFK's days, were they favored by the structure of the Senate. In this sort of blind, universal Senate election, they would be expected to come out with somewhere between 40 and 45 seats out of 100, depending on the year and the strength of PVI. Toward the higher end of this range, you find Presidential elections like those of JFK, Carter and Clinton, which, if they are used as baselines, show Democrats doing respectably. On the lower end, you find downright catastrophic alignments like 1972, 1984, 2000, or, indeed 2016.

Just how bad is 2016 in terms of Democrats' Senate prospects? Well, as you can see, it's quite simply the worst political alignment in recent history for Democrats. If all the Senate seats were up for election without incumbent and a neutral partisan environment, based on the 2016 political alignment of States and the current strength of PVI, we would expect Democrats to win just 38 seats. That's right, Republicans would win a filibuster-proof majority with the American electorate split evenly. This has to be alarming to anyone who is a big- or small-d democrat. With the effect of PVI rising, it does portend a future where a permanent Republican majority is a realistic possibility (although a lot of things can and will change, of course).

You can see how much things can change by looking at previous years. In 2008, Democrats could still hope to win 43 seats in a neutral year. In 2012, that number had only shrunk slightly to 42. It's only in 2016 that, as we saw, it plummeted brutally to 38. It was suggested in another thread that adding DC and Puerto Rico would help rectify the balance, and that's true. But even if Democrats netted all 4 of those seats (and there's some doubt over Puerto Rico), that would only bring the balance back to 42-62, still less than the 42-58 it could be if we could just back to 2012's alignment. No one can know for sure what the future holds, but those who advocate for Democrats to abandon the White working class completely should be aware that one of the consequences would
probably be to turn Senate Democrats into a permanent rump.

[Antonio the Sixth]

Any thoughts?

[Xeuma]

Any thoughts?

Democracy always wins: attempts to suppress the Will of the People will always end in revolution. A permeant minority-rule in any other country we'd call autocratic, but in our own it becomes a restriction placed upon us by the eternal wisdom of the Founding Fathers (which is why the 13th has to go, right?). Its time to stop glorifying the Constitution, it was a very, very good model for its time, but we live in a radically different time. The only solutions I see are either to redraw the states based on communities, to abolish the Senate, or to functionally abolish the Senate by limiting its powers a la the UK House of Lords.