Not to mention you just can't trust a model's judgement if the highest bidder chooses what it thinks
But you already can't trust a model's judgement, and there's an entire industry around "GEO" or "AEO", which is basically poisoning training data so that AI mentions your products. The post above is the owner of the model taking a cut of that.
But ultimately this is just an engineering problem, no?
Yes, if you just hack steering into a model it's going to hurt performance, because doing so takes the model out of the regime it was trained for and validated in. But if that steering were to be accounted for (e.g. by rearchitecting the training process) there's no reason why it couldn't work. Diffusion-based image generation models, for example, 'by default' just generated random images out of the noise; steering (i.e. the user prompt) was added on as a secondary input, which models had to be re-trained in order to use.
Comments
Steering can degrade and bias output
Even with basic experiments I've done, it frequently introduces much more hallucination etc and allowing arbitrary steering...
Not to mention you just can't trust a model's judgement if the highest bidder chooses what it thinks
But you already can't trust a model's judgement, and there's an entire industry around "GEO" or "AEO", which is basically poisoning training data so that AI mentions your products. The post above is the owner of the model taking a cut of that.
But ultimately this is just an engineering problem, no?
Yes, if you just hack steering into a model it's going to hurt performance, because doing so takes the model out of the regime it was trained for and validated in. But if that steering were to be accounted for (e.g. by rearchitecting the training process) there's no reason why it couldn't work. Diffusion-based image generation models, for example, 'by default' just generated random images out of the noise; steering (i.e. the user prompt) was added on as a secondary input, which models had to be re-trained in order to use.
Are the models you are using trained with steering being applied.