All this said, punching Richard Spencer and his ilk aren't going to solve, or even seriously address the problem. If we set aside the hyperbole, overt neo-Nazis aren't going to take over or even seriously intimidate the body politic like early 30's Germany. We need to be far far more concerned with their sympathizers in high positions whose veneer of respectability insulates them to varying degrees. Assaulting these meaningless peons of the hate movement in anything other than legit self-defense only drags the assaulters and their movement down to the Nazis level, both in an objective moral sense, as well as a subjective judgment of public opinion.
Keep calm and fight racists, but not with violence. It's worse than useless; it's downright counterproductive.