Doesn't it depend on the region? To my understanding, in the north, the upper classes were with the British. In the South and Mid-Atlantic, it was the slave-owning plantation elites (ironically enough) who were the most in league with the Patriot cause, with the lower classes being more divided in their loyalties.
Though I confess I am not sure why they cast their lot with the Patriot cause -they had the most to lose knowing the British would (and certainly did) use their slaves against them.
In the Southern colonies, the local elites were simultaneously the most affected by the British taxes and the most offended by the British elites not treating them like equals. If it weren't for going out of their way to offend the locals, the British Southern strategy might have worked as the upstate colonists were initially glad to be rescued from the control of the coastal elites until the overseas elites proved even worse. (To be fair, using Northern Tory troops looking for any excuse to treat someone like Rebel scum and get revenge for how they'd been treated up north was a major part of the problem.)
