Technically, every president before Lincoln would be worse than him, as they did nothing about the legalized human bondage of millions of Americans.
Good point that I had overlooked. After all, justice delayed is justice denied.
The question becomes how much can we attribute to the fact that the times dictated a different view of civil liberties. I mean, the only reason torture is looked down upon today is because we have decided that "harsh interrogation techniques" are wrong. Back then, until a certain point, slavery was not viewed as a civil liberties issue.
Doesn't excuse what they did, but you can't consider them terrible people. It's a hard matter to clear up.