No way, Walker would unify the Republican party, and would appeal greatly to the midwestern swing states.
Not sure why so many people think this. His first election was in 2010, which was a landslide year for the GOP in every state but the solid Democratic voting ones. In the recall, he won a lot of voters over on the very idea that he should be able to finish out his term. He had been in office for a very short amount of time before the recall protests and then the election and most people thought that he ought to have his term.
We'll see how well he fares in 2014, but right now he is only polling a couple of points ahead of a Democrats with almost no recognition.