When DeepMind made the protein folding AI they were given specific instructions by natsec people to prevent it from outputting genetic code for new or existing prions and afaik the prevention was at the training stage so "jailbreaking" shouldn't be possible

Nice! So the only thing anybody training their own protein-folding AI is to not put that restriction, and they could get code for all the prion diceases they want!

Very comforting!