Certainly don't know everything about this, but he brought in outsiders to his Presidential Office who knew nothing of the Congressional legislative process and he was uninterested in developing a relationship with House Speaker Tip O'Neill.
I'm less certain of this, but, despite the revisionist history of Carter as some far left liberal, he was probably easily the most conservative Democrat since before FDR and the new guard New Deal Democrats who were elected as a result of Watergate in 1974 and the traditional New Deal Democrats who were around before them, were disappointed that Carter wasn't interested in helping them pass much of their agenda.
Edit to add: This article goes into more detail and backs up most of what I wrote. I'd say it's a genuinely fair and balanced article. (It's a shame that Fox 'News' turned the phrase 'fair and balanced' into a joke)
https://carterthepeanutfarmer.wordpress.com/anti-carter/relationship-with-congress/