Freedom of association is a two way street. People are free to boycott this store, and I am sure the public outrage and typical hysterical reaction of the media will be more than enough to take down this sign.
Are you being serious?
Partly. I don't want to see the Civil Rights Act repealed, though in an ideal world, people should be free to discriminate in their personal lives if they choose to live life like an ass-wipe. CNN will talk about this story for a week, people will shame the owner, and the sign will come down. It's a pretty simple chain of events.
You could also legalize punching your customers in the face under the same theory.
As for freedom of association, that's not under threat here. This person opened a store that caters to anyone who walks by. So, he chose to associate with anyone who walks by, aka the general public. Running a store is not "your personal life" at all. It's public.