I buy large in most shirts and recently bought a JL shirt in large too. As above they are supposed to be athletic fit and despite my not so athletic body, it fits me fine. The only point I noticed is the sleeves are shorter than some shirts, but certainly my fav shirt at the moment.
Check the description of the polo shirt, I order my normal size if it's "regular fit", and go up a size if it's "slim fit". As said earlier, the sleeves are shorter than most other brands
Im XL in a polo shirt unless its the tour version then its a large, the trousers are spot on for size if your 36 then thats what it is, some makes it could be 34 or 38.