I have to trust Nate Silver not to weight these polls high. However, considering how high he weighted the Big Ten polls, I can't predict.
538 shouldn't (and doesn't) weight polls based on how reasonable they look, or based on whether or not a pollster seems to be coming out with garbage results. Selecting your own outliers and giving them a lower weight them is a recipe for confirmation bias.
He does give them a lower weight than most polls (and in fact a lower weight the Big Ten polls, which were given the weight of a completely unknown pollster coming out with their first polls), but he doesn't discount them entirely.
