his racial problems were societal problems.
No, Lee's position can't be equated with that of with free soil society. Case in point:
https://ballotpedia.org/Wisconsin_Suffrage_for_African_Americans_Referendum,_Question_1_(1857)Yes, it still failed, and doubtless most of the people that voted for black suffrage had racial prejudices, but its clear that many thousands (if not millions) of white people in the North were further ahead on race than Lee or even Lincoln.
True. Most people forget, if they ever knew it, that abolitionists were but a fringe part of the Republican coalition. For most Republicans, they just wanted to ensure that white workingmen didn't have to compete with blacks, be they slave or free, either in the north or in the western territories. Keeping down competition for jobs and land was also a factor in the nativist Know-Nothing Party becoming part of the Republican coalition as well.
While the Radicals weren't a majority of the party, they were hardly "fringe" and made up at least a third of the party.