I never had any problems with vbox usb passthru from lin.host to win.guest, so you might have a really specific case here, but I know there are loads of people running qemu on lin.host with all kinds of advanced hardware pass through configurations.

Thinking about it, if you have a very old motherboard the IOMMU layout might have hindered proper hardware separation.

But I doubt OP really need pass-through at all.