I think the Libertarian party could get serious influence in certain parts of the country, in certain states, but probably not surpass the GOP in popularity in every part of the country. Just like the Green party will never become a giant in West Virginia, the Libertarian party will struggle in gaining adherents in deeply religious and conservative states like Mississippi. Maine could perhaps be the first state where the Libertarian party becomes a major one, thanks to its independent nature, it being one of the three near-complete-non-religious states and due to Libertarian thoughts already infiltrating both local parties there and in particular the Democratic party (I'm sure many people already think of Shenna Bellows as more of a Libertarian rather than a Democrat for instance).
So, the Green party might become massive in California and Washington one day, just like the Libertarian party might see the same rise in Maine, and possibly neighbouring New Hampshire and Vermont too. There are way too many religious people in the US for the GOP to start moving in a less theocratic, more pluralistic and non-religious way any time soon. Remember that according to a recent Pew poll, a clear majority of adult Americans actually want to see the major parties move in a more religious direction, not in a less one.