I think very hot weather is worse. In very cold weather, it's easy to get warm with shelter and enough clothes. It's not as easy to get cool in very hot weather, even if you have air conditioning.
That's definitely not my experience. I've been freezing inside buildings (and even f**king buses) even when it was over 90F outside.
I find that most building are far too cold when it's hot outside. It's like someone decided that indoor areas need to be at 60F at all times, regardless whether it's 100F outside or 20F outside.
I like the building cool (55-60 degrees) when it is hot outside (to instantly stop sweating if I am) and nice and toasty (70 degrees?) if it is cold outside. I guess that's an expensive taste. But, constant cold is simply depressing, and heat is not. This week is going to have three straight days in the 40s here, the coldest stretch in a long time. It looks like the 70s should at least return by Sunday.
See, I'm actually kind of the opposite. I find drastic differences in outdoor and indoor temperatures to be annoying, especially when it's warn enough that I'm doing a lot of walking around outside. I'm outside, it's warm enough for me to sweat, I walk into a building with AC that would be a bit too cool for my tastes anyway, and then I get really cold becasue I'm also kind of wet from the sweat.