The issue is the barbaric way the fox is hunted and killed, not the social standing of the hunters. I would feel the same about working class people who badger bait or fight with dogs.
The more people talk about 'toffs' etc the more the main issue blurs as it is taken as jealousy.