Well, for one I'm not an American, and for another if asked my honest-to-God most personal opinion, I wouldn't agree that any position as powerful as the current American presidency ought to exist, and that direct popular vote would be an improvement on the situation but not a great one.
But Americans do tend to answer your question with stuff like "imagine the chaos and controversies of a nationwide recount" which, really, presupposes that rules remain odd and at variance even under direct popular vote - most don't tend to imagine that that might be changed. As it would need to be. So it's sort of a major psychological barrier.
And the other one's of course that the US Constitution is just so ridiculously hard to amend and the issue just doesn't seem worth fighting such an epic battle over to most people.