I'm guessing it's because rsync collects a list of files first and then deletes them rather than interleaving the two operations. This would mean latency not getting in the way as much as the read head switches between the filesystem index and the inodes themselves.

But that's just a guess.