I've crossdressed more than once and to assume that all men who crossdress are doing it for sexual or perverted reasons is kind of insensitive, to be flat honest.
Men wearing dresses, bras, etc is extremely creepy regardless of their reasons for doing so, imo.
Well dresses and bras are entirely different kettles of fish. A bra is specifically designed for women's bodies (or for very fat men) and so would only be worn by men for a joke or for some sexual reason. A dress is only feminine in as far as contemporary Western society says so. If you look in other contexts (both temporal and geographic) it would be completely normal for men to wear their hair long, use ostentatious jewellery and cosmetics, clothe themselves in skirts/tights/heeled shoes/dresses and other such clothing. Heck it used to be commonplace to dress prepubescent boys in "girls clothes" in the early 20th century for example (if you've ever seen those pics of presidents as children dressed in outrageously feminine outfits.
The question is, then, is it "creepy" to merely transgress a social norm?