I popped in here to post this. The aggregate audio device setup in the Midi setup assistant works really well for me. There is some loss in quality still, but definitely not as bad as before.

It also solved a problem I was having where the input volume would progressively get quieter as I was speaking making it so no one could hear me on calls.