Clothes that keep you dry and cut out the wind go a long way to keeping you warm.
By and large evaporative heat losses due to being wet can be far greater than straightforward conductive heat losses . Conductive heat losses are far far greater with a wind to increase the rate of conduction away from your body. Insulation is important and gortex is not a great insulating material but combined with a couple of insulating layers it will keep you far warmer than any amount of insulation and poor protection against wind and wet.
I think a lot of this depends on where you play your golf. On a links the windproof element is key, inland not so much.