According to some sources, the party which Clinton ran under is disputed, but according to most, he ran as a Federalist. Skeptical of this, Rufus King also ran as a "Straight" Federalist. He however had very limited ballot access and only got 2% of the popular vote.
It seems that Rufus King was supported by the state Federalist party in Virginia, so the votes for the Federalist electors there are that 2%. Elsewhere the state parties either supported Clinton or perhaps didn't specify, and some anti-Madison Republican electors also supported Clinton.