Yeah I wear one because I hate my arms being bulky with too much clothing, and don't like loose sleeves around my wrists when gripping the club. In the coldest months I wear a baselayer (tight) which covers my arms, then a polo shirt and a gilet and I'm as warm as I need to be. If I wear a jumper I end up faffing about rolling the sleeves up all day so there's no point. What the guy in the OP said was right, if your core is warm you don't really notice your arms being cold. I don't even wear trousers unless it's below 9°c as my legs don't really feel the cold as much either.
Edit: that means I wear shorts, not just walk round in my pants.