If I ask to deduce the gender of my voice, can it do that?

This iteration is not trained to do so. But the general model structure should work, i.e. if you finetune with instruction data to do so.

Training a projection layer makes sense, but ultimately you'd want to output audio conditioned on the input rather than text. Is there a way to train a reverse projection with some kind of skip connections to take audio input into account? Or an end to end audio model?

Training an end-to-end model is the eventual goal, yeah; discrete tokens is the most pragmatic approach then. I've successfully trained an end-to-end model at small scale; am in process of getting more data (and money/compute) to scale that up.