I think the mistake the white/gold people are making is to not take the background into account. Given the bright yellowish glare you see there, your eyes should be adjusting the brightness of the dress down, not up. I could understand the dress being seen as white/gold if it was clearly taken in a dark room with a blue-ish light source, but looking at the background it obviously isn't.
Aren't you overthinking this? It's not like people spend a couple of minutes assessing the image only to then reach the conclusion "ah yes, the background is quite clearly ___ so it should be ____, now I see it." My brain is telling me it's gold and white and who am I to argue with my brain?