This will happen if the remote party does not send a 'mix-minus' from their sound board (audio mixer), or if their headphones are turned up too loud.

All parties should always use headphones rather than speakers.