It never fails to amuse me to hear how little laity think a clergyperson does. I know my spouse's grandfather really did think that all pastors do is preach one hour a week, until we started sending him the church newsletter. And many pastors have the same number of committees and meetings no matter what size the church or churches. At our particular church we have about 800 members and my husband is the only pastor, although there are other full-time staff. We would never get a vacation if he didn't have some help covering the hospitals, and there have been a few times when we had to come home from vacation because someone in the congregation died. He's on call 24/7.
Pastors work very hard and have to deal with extremely difficult and complex situations. Care of souls is arduous work. For example, we had a person accused falsely but he was sent to jail anyway. The situation was complicated by the fact that the accuser, accuser's family, the accused, and the accused's family all attended the church. Needless to say, there were many hours spent in sorting everything out.
Sermons can take many hours to prepare. How would you like to prepare and memorize a 20-minute speech weekly and have about 200-300 people evaluating your every word over dinner?
Then, at least in the United Methodist Church, pastors have responsibilities beyond the local church, in the district and conference and perhaps in the jurisdiction and at the general church level. If pastors aren't careful, all they might end up doing is going to meetings.
Most pastors I know, love people and are there to serve. I know my husband wouldn't be anything else. He is called by God and gifted to serve as a pastor. But sometimes its a heavy gift. My job as a spouse is to help make the load as light as possible and smooth things out when I can. But as a Christian, I also have a calling. All Christians do.
So when people kid me about how little their pastor works, I just smile and say how wonderful and how pleased God must be that their pastor has laity like them to help share in the multitude of responsibilities. That usually ends the conversation.