If theres a load of 4 balls in front I agree you have to accept it but if theres only one or 2 groups just let the 2 balls through, theyre soon out of the wayYou cannot let a group through if there is another group in front. Let them through once there is a clear hole or almost so that they have a gap to fit into.
Letting a 2 ball through a field of 4 balls just because they are faster just slows donw the whole field, and the 2 ball is not VIP with rights over everyone else. It happens. But its just bad luck and they have to suck it up. They can make no complaint about slow play either.