Yes, I know that. If anything it's strengthens my argument. Abolitionists, to some degree, were motivated by pity and paternalism. A right-libertarian would have suggested that property rights trump these feelings.
Do you have any historical examples of a right-libertarian contemporary to the Civil War who was not anti-slavery? Proto-libertarian thought, like individualist anarchism and libertarian-style classical liberalism, was not common at all in the South at the time, and I'm having trouble thinking of anyone who might fit your description.
It's pretty easy to think of historical examples of anti-slavery, pro-Northern classical liberals, though. Herbert Spencer was as much of a classical liberal as it gets, and he was pro-North (although his enthusiasm cooled through the war, believing that the North mistrusted Britain too much). There's a good argument to be made that
Frederick Douglass was a classical liberal, and he was, of course, pro-North. John Stuart Mill, although a much poorer approximation of a libertarian than Spencer or even Douglass, was nonetheless a classical liberal who was very anti-slavery and pro-North.