tbh I disagree with this (as with the ban on Farakhan and non-violent Islamists as well). I think the Home Office has too much power in general. I'm also not sure why it really matters that they are foreign agitators; are they so much worse than homegrown fascists?
Homegrown fascists are our problem; we don't need to multiply that problem.
As long as you are in the EU you shouldn't ban EU citizens from entering due to their political beliefs. You can do that after Brexit but until then an Austrian citizen should have the same rights as a British, "fascist" or not.