It's not the rules, it's the message. Your problem with TRUMP appears to be that he's clearly articulating the parts of the Republican message that until now major figures in the party have chosen to say in coded language. At least with TRUMP there is no need for a secret decoder ring which is a large part of his appeal. You say you don't want the crazies in the party but you haven't said how you'd do that other than procedurally.
At this point, I don't know that it feasibly can be done (or rather undone). You need elected officials to start saying, "I don't care if I get primaried. I'm not going to spout extremist stances so that I can win a primary." The problem is that it's a positive feedback cycle. People who previously weren't political decided to get involved in politics, and while both sides have radicals, the radicals at issue now joined to GOP--not because there was anything wrong with the GOP at that time, but because that party more closely aligned with those radicals' views. Some officials decided to tap into those new supporters and pandered to them. This made people inside the party shift right as well; I'm not sure what the reason for that was. I've seen people completely reverse positions and become more radical, and these people are not elected people, just party participants. I don't know if it's because they want to stay in what they feel is the "majority" opinion-wise or what the reason is. But as the grassroots shifted right, this caused elected officials to shift right in order to stay in office. This in turn brought in new radicals and caused former sane party participants to also shift right.
I can tell you how to avoid it: don't pander, and when the bad apples come in in the first place, make it known that their radical ideas aren't welcome. But we're past that now. I don't know how you undo that without pissing off the radicals in order to drive them from the party.