I was particularly fascinated about the self-replication section [0]; I wasn't aware of the RepliBench paper [1], and now that I skimmed through it, I'm quite concerned. It was clear to me before that there's nothing particular stopping an AI from replicating, but seeing the already high success percentage at current levels of technology (e.g. 46% success of replicating itself onto compute) is very scary.
Comments
This is a really good report!
I was particularly fascinated about the self-replication section [0]; I wasn't aware of the RepliBench paper [1], and now that I skimmed through it, I'm quite concerned. It was clear to me before that there's nothing particular stopping an AI from replicating, but seeing the already high success percentage at current levels of technology (e.g. 46% success of replicating itself onto compute) is very scary.
[0] https://www.aisi.gov.uk/frontier-ai-trends-report#self-repli...
[1] https://arxiv.org/abs/2504.18565