There are many miles of distance between the "not really a fan, as such, though it may look like it to outside observers" attitude of Fillmore or Buchanan, and Van Buren's who was, it is safe to say, the most opposed President pre-Lincoln.
That may have been true after his presidency, but during his administration:
I guess the main difference between Martin Van Buren and Fillmore, Pierce, and Buchanan is that he was opposed to the expansion of slavery into the western territories. He did vote against the admission of Missouri into the Union as a slave state when he was in Congress, after all.
His is essentially the same position of Abraham Lincoln up until he abolished slavery in the District of Columbia in early 1862, and issued the Emancipation Proclamation later that same year.