it will only self-replicate if instructed to do so. It has no desires of its own beyond what its context window contains. The tool calls it executes and initial prompt are all the work of humans. Someone could give a launch_nukes tool to a LLM and it might choose to end the world but it's pretty clear who messed up here.

If labor loses its entire share of the pie and that causes things to go to hell though, yeah sure I could see bad things happening.