It's a different sort of danger. Knives, guns, etc are locally dangerous. The threat from AI is globally dangerous. Despite being a much lower risk of it happening, the blast radius is massively bigger, which means the overall danger is greater.
Also, maybe both things some be considered. Many countries manage the sale of knives, guns, etc. You don't need to accept that risk just because AI exists, or vice versa.
I agree that this should be something that researchers reflect on. GPT-2 is one of the primary models to research on nowadays, and many recent developments have come from studying it as a test bench.
Imagine if CRISPR was considered "too dangerous" to publish because of the potential ethical ramifications, and that only a special few should be aware. It is utter self-righteousness, and it is shameful behaviour. The world cannot adjust itself to what it cannot see, so you risk greater catastrophe by keeping it secret.
The open dissemination of knowledge at every increment is the only way for society to truly deal with what is to come.
Everything is "too dangerous". GPT2 is going to invent a time machine and break crypto and genetically engineer super rabies.
You are hyperbolically judging a post-hoc cherry-picked subset of a great spectrum of concern, with hindsight experience nobody had at the time.
Safely navigating the future isn't an accuracy contest. Risk mitigation has to account for the distribution of costs for different signs of error, where novelty, uncertainty, and any potential for compounding effects all greatly multiply the need for hedging.
As I understood it at the time, the rationale was that GPT2 made it too easy to produce content at scale and could easily be used for propaganda, phishing or other kinds of cons.
My impression at the time was they were perhaps overly cautious but this was a bunch of researchers who wanted to self-regulate. Anthropic didn’t exist, deepmind was also much less product-focused and relatively cautious. Chinese models weren’t really a factor either.
Government regulation was really not in the picture either in 2020 or 2021 tbh. The government was still trying to beat a pandemic. Some of the Biden admin eventually wanted to but it wasn’t very serious.
Parent post is saying there was no community of peer models, just some people figuring it out as they went and with lots of slack to go slow if they wanted
There were other players, china included, but it wasn't relevant until LLMs became profitable? China is being singled out as usual because it's the antithesis of the us
But... there weren't. The other "players," China included, had duck all to show for it. Did they have the same potential capabilities? Sure, maybe. It wasn't just a profitability thing though. Their models sucked. China is being singled out in that comment because their models are exceptional, with very few competitors, so when you're talking about a history of the competition not being any good you _have_ to acknowledge one of the only good present-day models.
Well... maybe optimal next token generation is all you need? Reductive as that is. I'm not even sure _why_ that actually matters? It can still result in accidents that impact the real world.
My bet is 2027 we see an agentic worm make global headlines.
Well, no. Without controls, a language model can drive sensitive individuals to violence or suicide. The idea of releasing a frontier model without RL is frightening based on what we have learned.
The idea of releasing a frontier model without RL is frightening
In case you were not aware, strong base models (no post training at all) have been available for quite some time now. Including ones that eclipse “scary” frontier models from even a year ago.
Words out of a magic box on a computer can’t make you kill yourself. Perhaps the people that would use that as encouragement are already mentally ill enough that it really doesn’t matter what the trigger is?
On a more callus but fully serious note, evolution starts out physical, that the species that don’t eat and breed as well, die. What happens when you remove that? When life is so safe that you basically wont stave, get eaten, catch a disease, don’t really need to compete all that hard for scarce resources, etc? Do you think evolution just stops? Or perhaps does a social or mental evolution become the predominant differentiator for successful reproduction over time?
Comments
This is how ridiculous this industry is. Regulation-seeking panic over nothing. Drama in search of a moat.
Everything is "too dangerous". GPT2 is going to invent a time machine and break crypto and genetically engineer super rabies.
They sell knives, guns, combustible materials, and multi-ton heavy machinery in stores. That's what's actually dangerous.
That's what's actually dangerous.
It's a different sort of danger. Knives, guns, etc are locally dangerous. The threat from AI is globally dangerous. Despite being a much lower risk of it happening, the blast radius is massively bigger, which means the overall danger is greater.
Also, maybe both things some be considered. Many countries manage the sale of knives, guns, etc. You don't need to accept that risk just because AI exists, or vice versa.
I agree that this should be something that researchers reflect on. GPT-2 is one of the primary models to research on nowadays, and many recent developments have come from studying it as a test bench.
Imagine if CRISPR was considered "too dangerous" to publish because of the potential ethical ramifications, and that only a special few should be aware. It is utter self-righteousness, and it is shameful behaviour. The world cannot adjust itself to what it cannot see, so you risk greater catastrophe by keeping it secret.
The open dissemination of knowledge at every increment is the only way for society to truly deal with what is to come.
You are hyperbolically judging a post-hoc cherry-picked subset of a great spectrum of concern, with hindsight experience nobody had at the time.
Safely navigating the future isn't an accuracy contest. Risk mitigation has to account for the distribution of costs for different signs of error, where novelty, uncertainty, and any potential for compounding effects all greatly multiply the need for hedging.
As I understood it at the time, the rationale was that GPT2 made it too easy to produce content at scale and could easily be used for propaganda, phishing or other kinds of cons.
Which turned out to be true.
And which says more about people being stupid and gullible than it does about AI models being smart or dangerous.
I'm tired of allowing stupid and gullible people to determine how the rest of us live, work, play, and create.
I gave up on gun control as a political question when I realized you can buy a flamethrower at Home Depot
My impression at the time was they were perhaps overly cautious but this was a bunch of researchers who wanted to self-regulate. Anthropic didn’t exist, deepmind was also much less product-focused and relatively cautious. Chinese models weren’t really a factor either.
Government regulation was really not in the picture either in 2020 or 2021 tbh. The government was still trying to beat a pandemic. Some of the Biden admin eventually wanted to but it wasn’t very serious.
why are chinese models a consideration at all?
Parent post is saying there was no community of peer models, just some people figuring it out as they went and with lots of slack to go slow if they wanted
Oh it's the typical china race thing, we can't let the antithesis of the us be better than the us etc. American exceptionalism
The 'china' part is irrelevant. You're addicted to being mad. Parent post is just saying there were no other players
There were other players, china included, but it wasn't relevant until LLMs became profitable? China is being singled out as usual because it's the antithesis of the us
But... there weren't. The other "players," China included, had duck all to show for it. Did they have the same potential capabilities? Sure, maybe. It wasn't just a profitability thing though. Their models sucked. China is being singled out in that comment because their models are exceptional, with very few competitors, so when you're talking about a history of the competition not being any good you _have_ to acknowledge one of the only good present-day models.
They were worried it would make it easy to generate a flood of misinformation. They were correct.
What are your thoughts on the HuggingFace incident?
Optimal next token generation disguised as something more.
Well... maybe optimal next token generation is all you need? Reductive as that is. I'm not even sure _why_ that actually matters? It can still result in accidents that impact the real world.
My bet is 2027 we see an agentic worm make global headlines.
What else is there?
"Drama in search of a moat" is hard to beat.
Well, no. Without controls, a language model can drive sensitive individuals to violence or suicide. The idea of releasing a frontier model without RL is frightening based on what we have learned.
In case you were not aware, strong base models (no post training at all) have been available for quite some time now. Including ones that eclipse “scary” frontier models from even a year ago.
So…
Words out of a magic box on a computer can’t make you kill yourself. Perhaps the people that would use that as encouragement are already mentally ill enough that it really doesn’t matter what the trigger is?
On a more callus but fully serious note, evolution starts out physical, that the species that don’t eat and breed as well, die. What happens when you remove that? When life is so safe that you basically wont stave, get eaten, catch a disease, don’t really need to compete all that hard for scarce resources, etc? Do you think evolution just stops? Or perhaps does a social or mental evolution become the predominant differentiator for successful reproduction over time?