It's not like you could say "I'm not racist, I just don't want black people in the front of the bus."
I don't think that's a very great metaphor, honestly; a more adroit one would be if, say, some random black man wanted to join the Polish-American Club. Pre-existing members of the club might naturally be outraged and oppose it on the basis that it renders the very basis for the club meaningless - it redefines Polish-Americanness as something else (say, affinity for Polish culture). However, the wrinkle in the metaphor is that members of the Polish-American club get special legal rights and privileges that non-members don't get, and so there is an inherently discriminatory legal regime so long as the club leadership can exclude people. The natural solution is to extend any such rights and privileges to anyone who requests them while simultaneously getting the government out of the business of deciding who is and who isn't a Polish-American.
