We can absolutely require that AI's not give advice that encourages self-harm or the people involved will go to jail.

Restricting harmless activity is an acceptable outcome of trying our best to prevent vulnerable people in society from hurting themselves and others.