Also, "buses are for poor people" is bullsh*t. Buses can, and should, be for everyone. (And there are already apps which help with the problem you mention, which I agree is a real one.)
I take the bus. About half the time I work standard hours and get to take an express bus. It's only 5 minutes longer than driving and cheaper too. The other half, I work odd hours and take a normal bus. It takes me a full 30 minutes longer to get home and has a higher proportion of "problem passengers". I imagine your typical bus in a mid-sized American city is like my "normal bus" rather than my "express bus".
If I had to take the normal bus all the time, I'd drive... and I'm someone with a relatively low income and a somewhat progressive attitude to transportation. What's your average Joe going to do when faced with the same situation?
Part of the challenge for cities will be to make the bus more appealing to middle class commuters.