JoUK makes some great points. Perception is a big part of it in some cases and that may be where AW is falling foul. I'm guessing AW that you spend more time looking for other peoples balls in the rough than you do your own? Suppose you go to look for someones ball which takes a couple of minutes, then you go to your ball by which time the group behind arrive on the tee. All they see is you taking what they consider to be too long and guess who gets labelled with slow play? The fella who spends all his time in the rough searching for balls isn't slow because as JoUK said, most of the moaners are hackers who do exactly the same thing and consider it to be part of the game.
I always like to get to my ball quickly and will walk ahead of I'm not in the firing line to measure my yardage, asses the shot and choose my club. When it's my turn to play I play, usually as soon as the last persons ball has come to rest.