There are many miles of distance between the "not really a fan, as such, though it may look like it to outside observers" attitude of Fillmore or Buchanan, and Van Buren's who was, it is safe to say, the most opposed President pre-Lincoln.
John Quincy Adams?
And Taylor, while a slaveholder, was quite opposed to its expansion outside the West, something that Pierce and Buchanan were willing to entice.
Adams. Good point. Though he evolved on the issue as well as "evolved", I think.
I'm well aware of Taylor, of course. Not being a political man, he had the right instincts - that further expansion of slavery beyond Texas was a nonstarter for a number of reasons (geographic, demographic - look at the northwest's growth in the era), international, and that the only way to end the debate was to create Free States in the South West now.
And he didn't share the just-as-pervasive-if-less-dangerous anti-Mexican racism of the age. That's another huge part of why his scheme to admit what would have been a huge Mestizo State of New Mexico was unthinkable to Congress.