Uh, we already have plenty of people as crazy as the Libertarians or Greens in Congress, look at the Tea Party and people like Sheila Jackson-Lee. Not to mention those fringe parties would probably only win about 2-3% of the vote anyway, and thus might not even make it into Congress depending on the threshold.
Right now, the House functions effectively as a partisan body. The swing vote is a member of the majority party who may represent an atypical district for his party; or there is no swing vote at all because of party discipline and the size of the majority.
I really, really, really don't want a situation where the House is split 216-215 and the balance is held by LaRouchites and Ron Paul as an independent.