Look at the audience in this picture. Golf by nature is going to (at least it used to) attract a better-dressed crowd. Also, golf is a man’s game. I see one woman in this picture. Men should participate in activities that are male-centric.
Notice the one “fat” man in the whole audience. He’s pretty tame by today’s standards. All the men are dressed well and even the ones who are casual still have a clean presentation. In a way, we’re really lucky to live in a time where it’s so easy to stand out from all the slobs and fatties who refuse to present themselves decently.