From the papers I've read, the stem separation models all seem to train off what seems like a fairly small dataset that doesn't have great instrument representation.

I wonder if you could assemble a big corpus of individual solo instruments, then permute a cacophonous mix of them. IIRC the main training dataset is comprised of a limited number of real songs. But I think a model trained on real songs might struggle with more "out there" harmonies and mixes.