"Racist" isn't really anything that can be defined in black-and-white, either-you're-one-or-you're-not terms anyways. At least not unless you want to limit it to the handful of kooks who actually talk about and think in terms of White Supremacy as an ideology, but then that's a meaningless non-debate.
Also,
I think being called a racist has become such a negative thing in America that people react as we see above if they get even an inkling they might be accused as such.
And not just in America. It's very un-pc to talk about it.