People should only follow Jesus Christ. Not a man.
Ok then, don't "follow" your pastor or minister. Stop going to those services. Who cares what he says? He is a man. He isn't Jesus Christ. Therefore, you shouldn't "follow" him.
None of the pastors at my church, (for which a majority are women by the way) claim to be God's infallible representative on Earth. In fact they've even done joint teaching or round-table discussion sermons where they've admitted to having different views on some things...and that that's OK and that we do not need to agree with everything they preach either.
Now I think this Pope isn't a bad guy, in fact I even like his Twitter feed. But he is just a man, like me.
As it must always be pointed out, papal infallibility is a very rare occurrence, happening only "[w]hen, in the exercise of his office as shepherd and teacher of all Christians, in virtue of his supreme apostolic authority, he defines a doctrine concerning faith or morals to be held by the whole Church." Popes have downplayed infallibility, for example, "The Pope is not an oracle; he is infallible in very rare situations, as we know" (Benedict XVI) and "I am only infallible if I speak infallibly but I shall never do that, so I am not infallible" (John XXIII).
As for the role of Vicar of Christ, what is the difference between God having a "representative" on Earth leading the Church, and God working through the writers of the Bible? Jesus didn't sit down and write down all those words himself. How can you say that the Bible was written (and selected) by God through his...representatives maybe?... on Earth, and that it could never happen again for other issues of the church?
Jesus made Peter the leader of the Church on Earth and gave his successors the role as well. It is all laid out pretty well in the Bible.
Primacy of Peter
Mt 16:18 - upon this rock (Peter) I will build my church
Mt 16;19 - give you keys of the kingdom; power to bind & loose
Lk 22:32 - Peter's faith will strengthen his brethren
Jn 21:17 - given Christ's flock as chief shepherd
Mk 16:7 - angel sent to announce Resurrection to Peter
Lk 24:34 - risen Jesus first appeared to Peter
Acts 1:13-26 - headed meeting which elected Matthias
Acts 2:14 - led Apostles in preaching on Pentecost
Acts 2:41 - received first converts
Acts 3:6-7 - performed first miracle after Pentecost
Acts 5:1-11 - inflicted first punishment: Ananias & Saphira
Acts 8:21 - excommunicated first heretic, Simon Magnus
Acts 10:44-46 - received revelation to admit Gentiles into Church
Acts 15:7 - led first council in Jerusalem
Acts 15:19 - pronounces first dogmatic decision
Gal 1:18 - after conversion, Paul visits chief Apostle
*Gal 2:11-14 - I opposed Cephas to his face for his hypocrisy
Peter's name always heads list of Apostles: Mt 10;14; Mk 3:16-19; Lk 6:14-16; Acts 1:13
"Peter and his companions" Lk 9:32; Mk 16:7
Spoke for Apostles - Mt 18:21; Mk 8:29; Lk 8:45; 12:41; Jn 6:69
Peter's name occurs 195 times, more than all the rest put together
Apostolic Succession
2 Chr 19:11 - high priest is over you in everything of Lord's
Mal 2:7 - seek instruction from priest, he is God's messenger
Eph 2:20 - Church built upon foundation of apostles & prophets
Eph 4:11 - God gave some as apostles, others as prophets...
1 Cor 12:28-29 - God designated in church: apostles, ...
Acts 1:20 - let another take his office
Acts 1:25-26 - Matthias takes Judas' apostolic ministry
1 Tim 3:1, 8; 5:17 - qualifications for: bishops, priests, & deacons
1Tim 4:14 - gift conferred with the laying on of hands
1Tim 5:22 - do not lay hands too readily on anyone
Acts 14:23 - they appointed presbyters in each church
2Tim 2:2 - what you heard from me entrust to faithful teachers
Titus 1:5 - appoint presbyters in every town, as I directed
If you want to have these differences of opinions, then allow us Catholics to have our opinions and be in a church led by the pope. The papacy has done a lot more to help humanity than you or I or any of us has, and wishing for the institution to be abolished is a silly thing to say. No one is forcing you to listen to him.