"Do you want to come up to my place for a drink?" = "do you want to have sex?"
Everyone knows that because it's been repeated in every piece of media ever.
Anyone who doesn't know that is either a vegetable or being disingenuous.
No, it isn't consent. Maybe they just really want to visit and have a drink.
The fact that people can interpret such a statement differently is significant in itself though. The rare false accusation must be a much smaller issue than just two people having different understandings of what was consented to and what wasn't.
