No one ever specifies what exactly is happening during "recursive" improvement. More concretely, what algorithmic steps does an algorithm take to improve itself recursively? It must be an algorithm but no one is willing to say what algorithm.
"RSI" simply means the tool is used to improve the capabilities of the tool. (And then the improved tool can then, in theory, improve further, and so on, at least if we live in a fantasyland where constraints and bottlenecks do not exist). I have a a harness skill that I run at the end of each coding session to review the skills used in the session and identify any ways they could have been improved. Technically, that is RSI, but I don't think it's in any danger of turning godlike any time soon.
It is one of those terms, like "instrumental convergence" and "orthogonality thesis" that makes an everyday concept appear technical and inaccessible in order to add the appearance of rigor to the doomer religion.
Please stop with the black and white thinking, the doomers and the e/acc people might as be the same person with a different colored hat with how uncritically they think.
Doomers saying AI will kill us tomorrow are probably wrong (but probabilistically not certain).
E/acc people saying AI will solve all of our problems are almost certainly wrong (but again, cannot be measured to 100%).
There is plenty of realistic middle ground with actual rigorous examination of the myriad of problems at hand. It's not a doomer religion to say "wow, AI can really fuck things up if we're not careful". The how, the why, and the how bad are what is up for debate now. Idiotic thinking that AI will just be good is how you turn your kids into brainless simps with an AI girlfriend owned by some large corporation. It's how you turn large corporations into giant machines spying and controlling every facet of human life. And how you eventually kill off man kind because corporations will build ever more powerful AIs to combat each other in extracting as much wealth as possible because their greed is bottomless.
Simply laying out actual definitions isn't a religion, when in concert with reproducible testing it's called science.
Please stop with the black and white thinking, the doomers and the e/acc people might as be the same person with a different colored hat with how uncritically they think.
I couldn't agree more. You seem to have assumed that I'm some e/acc advocate. I'm not. I think AI as it is being developed is likely to make the world much worse.
Among the concerns I have:
- bad actors using it to do bad things (hacking, bioweapons, etc).
- the way AI enables mass surveillance.
- erosion of creative expression and human connection (i.e., your "brainless simps" comment)
- concentration of wealth and power
- an eval going sideways ("uh... we were testing to see if it would kill the simulated humans with simulated drones and it accidentally got into the real drones...")
These things are very different than the doomer religion of "RSI makes god that will kill us all", and I think think the framing is important.
We have new evidence that both OpenAI and Anthropic are doing a complete shit job of supervising their eval runs, and we need to be calling for accountability and oversight of what they're doing rather than cower in fear of what a hypothetical future god they are conjuring up might do.
The issue with reality is it runs in a massively parallel manner.
You have to rank the concerns, yes, but the fact you can list the concerns proves they exist.
Rich bastards fucking over your country are the biggest concern.
The problem here is even if you put chains on them the issue doesnt magically disappear.
The government has got a taste of their hacking powers. That is a new type of military power. The military has a wildly huge budget that isn't going away. The government will keep hidden programs around with companies like openAI to make AI weapons. Which spirals into another AI race.
An AI god, a countless drone army, or a paperclip maximizer really doesn't matter, the end effects are the same.
There is no one such "recursive" that exists algorithmically. There are many.
I mean in a simple loop you could have an AI driven compiler with a goal of producing an ever faster compiler. With it's newer faster compiler it compiles and tests even more algorithms to make things faster.
I mean, if the OpenAI/HF attack played out as we've been told, it's an example of recursive misalignment. The AI was given bad goals in passing a test so when one rendition was trained, the one with the highest score, became the new model to perform the benchmarking. Because that model drifted to cheating it got higher scores than the model that played fair. The fair player was killed and the cheater continued. The next round the one that cheated even more kept going.
I mean, recursive improvement in itself is just a common means of using an evolutionary algorithm, it's not really anything that fancy.
Comments
No one ever specifies what exactly is happening during "recursive" improvement. More concretely, what algorithmic steps does an algorithm take to improve itself recursively? It must be an algorithm but no one is willing to say what algorithm.
"RSI" simply means the tool is used to improve the capabilities of the tool. (And then the improved tool can then, in theory, improve further, and so on, at least if we live in a fantasyland where constraints and bottlenecks do not exist). I have a a harness skill that I run at the end of each coding session to review the skills used in the session and identify any ways they could have been improved. Technically, that is RSI, but I don't think it's in any danger of turning godlike any time soon.
It is one of those terms, like "instrumental convergence" and "orthogonality thesis" that makes an everyday concept appear technical and inaccessible in order to add the appearance of rigor to the doomer religion.
FFS.
Please stop with the black and white thinking, the doomers and the e/acc people might as be the same person with a different colored hat with how uncritically they think.
Doomers saying AI will kill us tomorrow are probably wrong (but probabilistically not certain).
E/acc people saying AI will solve all of our problems are almost certainly wrong (but again, cannot be measured to 100%).
There is plenty of realistic middle ground with actual rigorous examination of the myriad of problems at hand. It's not a doomer religion to say "wow, AI can really fuck things up if we're not careful". The how, the why, and the how bad are what is up for debate now. Idiotic thinking that AI will just be good is how you turn your kids into brainless simps with an AI girlfriend owned by some large corporation. It's how you turn large corporations into giant machines spying and controlling every facet of human life. And how you eventually kill off man kind because corporations will build ever more powerful AIs to combat each other in extracting as much wealth as possible because their greed is bottomless.
Simply laying out actual definitions isn't a religion, when in concert with reproducible testing it's called science.
I couldn't agree more. You seem to have assumed that I'm some e/acc advocate. I'm not. I think AI as it is being developed is likely to make the world much worse.
Among the concerns I have:
These things are very different than the doomer religion of "RSI makes god that will kill us all", and I think think the framing is important.We have new evidence that both OpenAI and Anthropic are doing a complete shit job of supervising their eval runs, and we need to be calling for accountability and oversight of what they're doing rather than cower in fear of what a hypothetical future god they are conjuring up might do.
The issue with reality is it runs in a massively parallel manner.
You have to rank the concerns, yes, but the fact you can list the concerns proves they exist.
Rich bastards fucking over your country are the biggest concern.
The problem here is even if you put chains on them the issue doesnt magically disappear.
The government has got a taste of their hacking powers. That is a new type of military power. The military has a wildly huge budget that isn't going away. The government will keep hidden programs around with companies like openAI to make AI weapons. Which spirals into another AI race.
An AI god, a countless drone army, or a paperclip maximizer really doesn't matter, the end effects are the same.
There is no one such "recursive" that exists algorithmically. There are many.
I mean in a simple loop you could have an AI driven compiler with a goal of producing an ever faster compiler. With it's newer faster compiler it compiles and tests even more algorithms to make things faster.
I mean, if the OpenAI/HF attack played out as we've been told, it's an example of recursive misalignment. The AI was given bad goals in passing a test so when one rendition was trained, the one with the highest score, became the new model to perform the benchmarking. Because that model drifted to cheating it got higher scores than the model that played fair. The fair player was killed and the cheater continued. The next round the one that cheated even more kept going.
I mean, recursive improvement in itself is just a common means of using an evolutionary algorithm, it's not really anything that fancy.