I've got a few base-layers from my snowboarding days, and in the winter I would sometimes wear a thermal layer underneath my golf shirt. Only problem is, if it gets too warm you've got to take both tops off to get the base layer off, before putting the short-sleeve shirt on again, and that can scare some of the locals. Far easier to just take your jumper off.
Plus I think it's a very American look, short-sleeves over the top of long-sleeves. Shirt and jumper is far more smart.