Yes, it was a mistake - but perhaps one very few people truly could have made by virtue of knowing they were making it at the time. Nevertheless, the Kerry campaign was probably aware of the dynamics.
Before I say what I have to say and some people start getting cynical or crappy about it, I feel it is important to simply emphasize the following: the Kerry campaign never pulled out of North Carolina. Now that we have that out of the way, let's proceed.
Unfortunately, Kerry played into the same tragic game that most in his position do: incumbency comes with a lot of privileges. One of them is the ability to play the "expanding the map" game; wherever the incumbent decides to invest, the challenger feels obligated to "defend", bogging down resources. When there isn't an incumbent, this rule applies to whoever has the most/least money. Kerry could have been a wildcard here and put Bush on defense (requiring him to pull investments out of less sure states like NM & IA), but the campaign chose not to (and instead, try to make their wildcard state a ridiculous one like NC).
[...] I would say Virginia flipped on the national level at some point in 2006 or 2007 and has been moving further into the Democratic column ever since [...]
Now I understand that state-level politics (particularly in states that were still somewhat Southern at the time) are different, but the undercurrents of the Republican Party weren't strong enough to overcome the whole "Virginia elects a Governor opposite of the party of the President" and thus the state had Democratic Governors throughout the 2000s.
Furthermore, "Macaca" and all of that, but a
literal loser-nobody with hardly any financing or organization won a Senate race just two years later. Even if it was a wave year and the win was slight, that fact is balanced out by the notion that turnout was lower in 2006 and much more favorable in 2004 in racial terms to Dems:
The sheer fact that "Macaca" was what really caused the Allen campaign to begin unraveling to a large degree tells you all you need to know about how much VA had already changed. Now, the rate of change as I described in my full quote above was very rapid - one could perhaps argue that the rate of change was so fast that the state was still a backwater hellhole ideologically in 2004 but had progressed enough in just two years to refute such nonsense and actually make it matter - but I think the reality is that the basis for such a climate was largely there in 2004. In other words, someone like Kerry could have played very well in NoVA at the very least, had he tried. That, combined with the fact that the more people heard about Bush, the less they liked him, could have produced some very interesting results.
Furthermore, take a look at those exit polls one more time, and compare 2004/2008. The white/black make-up of the electorate changed surprisingly little between those two elections - Georgia diversified twice that much in the voting booth between the same two elections. Asians going from 2% to 3% and Latinos going from 3% to 5% - with their support likely shifting by no more than 10-15 points in all - closes the margin by about 2.2 points. So Obama went from Kerry's baseline to 46.6% because of that. The black vote shift between the two cycles is literally 0.1 point of performance. It was white voters that did it, and Kerry could have played here.
A substantial power-play by the Kerry campaign in VA and its subsequent effects would have likely forced Bush to divert resources being invested in IA & NM. Considering both of those states were won by fractions of a percentage point, it's likely that even small diversions of investment would have tilted these two states back to Kerry. Combined with a narrow win in VA, Kerry would have 276 EVs while losing the PV by more than two points (wouldn't that have been a sweet ending for Bush?).
TL;DR: Virginia would have been a nail-biter in 2004 but it also could have been a life-raft as it would have forced Bush to divert funds from states like NM & IA in order to defend it. The dynamics that were in play in 2005 & 2006 in VA elections were already there in 2004. The electorate didn't change wildly despite demographic shifts between 2004 & 2008; generating support among whites and actually campaigning in the state was what gave Obama a victory, as the swings were solid statewide. Minority shifts in the electorate accounted for less than 1/3 of the swing in VA between 2004-2008. 
