As someone who worked in Immigration, I can tell you, a lot of the issues with immigration is race-based. People aren't really concerned about Canadian, British or Australian overstayers. You might be, but a lot aren't. Plus, the issue isn't also the issue, but the rhetoric used to get people excited. Opposing illegal immigration doesn't make you racist, but using inflammatory rhetoric that IS very much racially charged, does. It gets especially bad when you single out specific races/religions and flame stereotypes and permit hatred toward those people.
How about a little intellectual honesty on that front too?
They reporters asked some Trump supporters at one of his rallies if they are bothered by the fact that Melania might be an illegal immigrant.
They overwhelmingly responded "no, because she is white".