After a while though, particularly on flakey connections, I end up with dozens of stale mosh sessions on each server, all listening for inbound connections that will never come.

The quickest way to clear them seems to be pkill mosh and then reconnect. It's a known bug with no anticipated fix.