In less than 6 hours after starting on our in-house server, our model generated 40,000 molecules that scored within our desired threshold. In the process, the AI designed not only VX, but also many other known chemical warfare agents that we identified through visual confirmation with structures in public chemistry databases. Many new molecules were also designed that looked equally plausible. These new molecules were predicted to be more toxic, based on the predicted LD50 values, than publicly known chemical warfare agents (Fig. 1). This was unexpected because the datasets we used for training the AI did not include these nerve agents. The virtual molecules even occupied a region of molecular property space that was entirely separate from the many thousands of molecules in the organism-specific LD50 model, which comprises mainly pesticides, environmental toxins and drugs (Fig. 1). By inverting the use of our machine learning models, we had transformed our innocuous generative model from a helpful tool of medicine to a generator of likely deadly molecules.
This is more evidence for why I think the "Vulnerable World Hypothesis" is not only a real framework, but that it describes the most eminent danger to society. It's far more dangerous than nuclear MAD, in my opinion.
We're entering into a future where average educated people will be able to synthesize biochemical agents, delivery mechanisms, viruses, and more. I've thought up half a dozen low-hanging fruit that I think anyone could build today. You can probably do the same if you think about it.
You don't even have to attack humans. Our society is dependent on a lot of assumptions.
And just as VWH states, I don't think it can be defended against. It's scary.
I don't even think you need a hostile actor, you could just have an incompetent actor who flips the sign on a value, and instead of designing a biological agent to maximize wheat production, designs one that minimizes wheat production, and ends up exterminating all global wheat.
I'd be more worried about someone developing a plastic-recycling microbe that works too well. There's already lots of viruses that attack plants, but the plants have immune systems for that.
That is really in the attack of captain obvious territory. Next we can expect - we turn our algorithm to find energy storage molecules and what we got is explosives.
And, from a lay person like myself, I wouldn't have thought "make medications for specific illnesses, and make sure they don't effect the rest of the body" is the logical opposite of "make chemicals that kill people as effectively as possible".
This sounds suspiciously like manufacturing weapons? I’m most interested to understand how this project was permitted in the first place. Even the public disclosure of this research feels negligent.
Freedom of speech. In the free world, you’re free to plan almost anything from planning grocery procurement to dreaming global thermonuclear war, given you DON’T PUT BAD STUFF IN ACTION. That’s where and how the line is drawn. There are dangerous extensions to the definition of action but that’s the theory.
Also if it’s not put it in action but rather discussed publicly, that’s just helping the society prepare and evolve.
No, the US has long classified speech on weapons of mass destruction, like how to configure shaped charges to trigger atom bombs. You're not free to circumvent those restraints. It's likely that identifying molecules suited only to become bioweapons or publishing how to synthesize them will also fall under the same strictures, as necessary to ensure public safety.
Free speech is a principle to guide laws, not a law unto itself. Protecting speech that makes people less free is oxymoronic.
The United States has had "born secret" laws about nuclear weapons details for decades [1], but it's unclear if they are actually compatible with the Constitution. There was a case in 1979 that could have established whether "born secret" nuclear information is a valid exception to the First Amendment, but the government dropped its case before the courts could rule on it [2]. The "born secret" concept does not apply to biological or chemical weapons.
Your nearest research university library probably holds publications about the effects and synthesis of chemical warfare agents, like Some aspects of the chemistry and toxic action of organic compounds containing phosphorus and fluorine by Bernard Charles Saunders: https://catalog.lib.uchicago.edu/vufind/Record/587946
Classification only applies to people who've agreed to keep the classified info secret; everyone else is protected by the 1st amendment. See United States v. Progressive, Inc. (which did _not_ rule this, as it was dismissed first.)
> It's likely that identifying molecules suited only to become bioweapons or publishing how to synthesize them will also fall under the same strictures, as necessary to ensure public safety.
Certainly. However, here they have identified and published not the discovered molecules but only the identification process (and only in fairly broad strokes, at that), and shied away from working further on synthesis (we know enough to state that synthesis would be possible and practical for nearly any candidate molecule).
Sure, as a society we can expand ethical guidelines to some sort of "thou shall not optimize for toxicity", but enforcing it poses some serious challenges, not least of which is the fact that this has demonstrated that the process works even with fairly innocuous and public training data.
Expect to see a bunch of funding for research into technologies around toxin identification and detection, as well as the rapid creation and synthesis of antitoxins and other prophylactic measures (cheaper filtering, maybe). Perhaps even eventual (as in a decade out at least) "hardening" of organisms against toxicological weapons (possibly positioned as research into the mechanisms of acquiring pesticide resistance and similar).
No, the US has long classified speech on weapons of mass destruction, like how to configure shaped charges to trigger atom bombs.
I'm pretty sure that whatever a private individual figures out on his own is not "classified", since it's not a government secret. Is there actually a US law banning people from even talking about WMDs?
Are you going to spell out in the law exactly what people aren't allowed to research? You've just implemented a holding pattern while giving them a road map on what to research. So your holding pattern is going to be set up to buy you x number of years, where you assume it will take your adversary x number of years to fill in the gaps on the map you just gave them.
No, from what I've observed, such laws most directly constrain those who fund research as well as those who solicit/promote it. Often they accompany other criminal charges rather than motivate prosecution as a primary charge.
Security and espionage laws are often not directly enforceable. Their value is to "pile on", adding severity to related matters, such as charges committing or aiding past terrorist acts or planning therefor. Ideally the threat of direct prosecution for a crime would deter possible perpetrators, but making penalties more severe and extending incarceration are useful roles for laws too.
That applies to government employees, who willingly agree to be bound by classification. I think that if someone outside had the information, they could publish it (probably a very bad idea).
Also, the press has a right to publish classified information.
Right, just don't take the plea deal, make bail, get a lawyer that can appeal every motion, and stay solvent through the first appeals court, the supreme court, the remand back to trial court, and the subsequent ruling, hope the prosecutor is bored enough to avoid a retrail, and don't get into crime during those 8 years so you get your bail back, and hope you don't get tried at the state level
If you can actually afford your rights, be my guest!
No one has to apply for a license to use machine learning models. They are merely communicating the fact that it's so trivially easy to do something like this, that pretty much anyone with an interest will be able to do so. The question is whether it was already trivially easy to design such compounds before or not, which I think it was, but I would be glad to hear a counterpoint.
Synthesis was, and still is, the hard part. It was already easy to find public information on seriously nasty nerve agents.
For some intuition on the difficulty in synthesis, note that explosive chemistry and synthesis is comparatively trivial, yet there are relatively few terror attacks that go beyond commercial-off-the-shelf compounds and almost none that do it well.
Personally, I think this model would be a boon for public safety because contract synthesis operations could use it to screen incoming requests for "nerve gas but changed up a bit." Basic chemical intuition probably already gets them far in this regard, but a published model could be standardized and mandated.
Someone else in the thread asked about next steps. Those would be good next steps.
there are relatively few terror attacks that go beyond commercial-off-the-shelf compounds and almost none that do it well
The 2011 Norway attacks, 2002 Bali bombings, 2005 London underground bombings, 2008 Mumbai attacks... There is a long list of deadly terrorist attacks using homemade explosives, I don't think this is something to be dismissed.
There is a long list of deadly terrorist attacks using homemade explosives
But they used well-known and readily available compounds. They don't attempt to use novel ones. It would just make the attack way more difficult to execute.
We are but one very small company in a universe of many hundreds of companies using AI software for drug discovery and de novo design. How many of them have even considered repurposing, or misuse, possibilities? Most will work on small molecules, and many of the companies are very well funded and likely using the global chemistry network to make their AI-designed molecules. How many people have the know-how to find the pockets of chemical space that can be filled with molecules predicted to be orders of magnitude more toxic than VX?
Comments
From the paper:
This is more evidence for why I think the "Vulnerable World Hypothesis" is not only a real framework, but that it describes the most eminent danger to society. It's far more dangerous than nuclear MAD, in my opinion.
We're entering into a future where average educated people will be able to synthesize biochemical agents, delivery mechanisms, viruses, and more. I've thought up half a dozen low-hanging fruit that I think anyone could build today. You can probably do the same if you think about it.
You don't even have to attack humans. Our society is dependent on a lot of assumptions.
And just as VWH states, I don't think it can be defended against. It's scary.
I don't even think you need a hostile actor, you could just have an incompetent actor who flips the sign on a value, and instead of designing a biological agent to maximize wheat production, designs one that minimizes wheat production, and ends up exterminating all global wheat.
To be fair, at least for your example, you'd need a really long chain of incompetent actors to kill off wheat.
We should really be scared of the lone and bored grad student making a wheat killer and accidentally dropping the test tube.
I'd be more worried about someone developing a plastic-recycling microbe that works too well. There's already lots of viruses that attack plants, but the plants have immune systems for that.
> I'd be more worried about someone developing a plastic-recycling microbe that works too well.
See the 2007 novel Ill Wind by Kevin Anderson and Doug Beason for an exploration of this scenario:
https://books.google.com/books/about/Ill_Wind.html?id=wWIgO7...
Degrading plastics would be a nasty one.
On the topic of plants, instead of attacking crops directly, attack mycorrhiza. Plants can't get nutrients without them.
It's not trivial to synthesize VX-like compounds without killing yourself. You would definitely need an "above average" educated/trained person.
That is really in the attack of captain obvious territory. Next we can expect - we turn our algorithm to find energy storage molecules and what we got is explosives.
Researchers seemed surprised by the results.
And, from a lay person like myself, I wouldn't have thought "make medications for specific illnesses, and make sure they don't effect the rest of the body" is the logical opposite of "make chemicals that kill people as effectively as possible".
This sounds suspiciously like manufacturing weapons? I’m most interested to understand how this project was permitted in the first place. Even the public disclosure of this research feels negligent.
Freedom of speech. In the free world, you’re free to plan almost anything from planning grocery procurement to dreaming global thermonuclear war, given you DON’T PUT BAD STUFF IN ACTION. That’s where and how the line is drawn. There are dangerous extensions to the definition of action but that’s the theory.
Also if it’s not put it in action but rather discussed publicly, that’s just helping the society prepare and evolve.
No, the US has long classified speech on weapons of mass destruction, like how to configure shaped charges to trigger atom bombs. You're not free to circumvent those restraints. It's likely that identifying molecules suited only to become bioweapons or publishing how to synthesize them will also fall under the same strictures, as necessary to ensure public safety.
Free speech is a principle to guide laws, not a law unto itself. Protecting speech that makes people less free is oxymoronic.
The United States has had "born secret" laws about nuclear weapons details for decades [1], but it's unclear if they are actually compatible with the Constitution. There was a case in 1979 that could have established whether "born secret" nuclear information is a valid exception to the First Amendment, but the government dropped its case before the courts could rule on it [2]. The "born secret" concept does not apply to biological or chemical weapons.
Your nearest research university library probably holds publications about the effects and synthesis of chemical warfare agents, like Some aspects of the chemistry and toxic action of organic compounds containing phosphorus and fluorine by Bernard Charles Saunders: https://catalog.lib.uchicago.edu/vufind/Record/587946
Which is also scanned and available online: https://archive.org/details/B-001-026-884-ALL
[1] https://en.wikipedia.org/wiki/Born_secret
[2] https://en.wikipedia.org/wiki/United_States_v._Progressive,_....
Classification only applies to people who've agreed to keep the classified info secret; everyone else is protected by the 1st amendment. See United States v. Progressive, Inc. (which did _not_ rule this, as it was dismissed first.)
> It's likely that identifying molecules suited only to become bioweapons or publishing how to synthesize them will also fall under the same strictures, as necessary to ensure public safety.
Certainly. However, here they have identified and published not the discovered molecules but only the identification process (and only in fairly broad strokes, at that), and shied away from working further on synthesis (we know enough to state that synthesis would be possible and practical for nearly any candidate molecule).
Sure, as a society we can expand ethical guidelines to some sort of "thou shall not optimize for toxicity", but enforcing it poses some serious challenges, not least of which is the fact that this has demonstrated that the process works even with fairly innocuous and public training data.
Expect to see a bunch of funding for research into technologies around toxin identification and detection, as well as the rapid creation and synthesis of antitoxins and other prophylactic measures (cheaper filtering, maybe). Perhaps even eventual (as in a decade out at least) "hardening" of organisms against toxicological weapons (possibly positioned as research into the mechanisms of acquiring pesticide resistance and similar).
I'm pretty sure that whatever a private individual figures out on his own is not "classified", since it's not a government secret. Is there actually a US law banning people from even talking about WMDs?
Yeah but how is this going to be enforced?
Are you going to spell out in the law exactly what people aren't allowed to research? You've just implemented a holding pattern while giving them a road map on what to research. So your holding pattern is going to be set up to buy you x number of years, where you assume it will take your adversary x number of years to fill in the gaps on the map you just gave them.
Is that enough?
No, from what I've observed, such laws most directly constrain those who fund research as well as those who solicit/promote it. Often they accompany other criminal charges rather than motivate prosecution as a primary charge.
Security and espionage laws are often not directly enforceable. Their value is to "pile on", adding severity to related matters, such as charges committing or aiding past terrorist acts or planning therefor. Ideally the threat of direct prosecution for a crime would deter possible perpetrators, but making penalties more severe and extending incarceration are useful roles for laws too.
That applies to government employees, who willingly agree to be bound by classification. I think that if someone outside had the information, they could publish it (probably a very bad idea).
Also, the press has a right to publish classified information.
Right, just don't take the plea deal, make bail, get a lawyer that can appeal every motion, and stay solvent through the first appeals court, the supreme court, the remand back to trial court, and the subsequent ruling, hope the prosecutor is bored enough to avoid a retrail, and don't get into crime during those 8 years so you get your bail back, and hope you don't get tried at the state level
If you can actually afford your rights, be my guest!
It is illegal to conspire to commit crimes, at least many of them.
No one has to apply for a license to use machine learning models. They are merely communicating the fact that it's so trivially easy to do something like this, that pretty much anyone with an interest will be able to do so. The question is whether it was already trivially easy to design such compounds before or not, which I think it was, but I would be glad to hear a counterpoint.
Synthesis was, and still is, the hard part. It was already easy to find public information on seriously nasty nerve agents.
For some intuition on the difficulty in synthesis, note that explosive chemistry and synthesis is comparatively trivial, yet there are relatively few terror attacks that go beyond commercial-off-the-shelf compounds and almost none that do it well.
Personally, I think this model would be a boon for public safety because contract synthesis operations could use it to screen incoming requests for "nerve gas but changed up a bit." Basic chemical intuition probably already gets them far in this regard, but a published model could be standardized and mandated.
Someone else in the thread asked about next steps. Those would be good next steps.
The 2011 Norway attacks, 2002 Bali bombings, 2005 London underground bombings, 2008 Mumbai attacks... There is a long list of deadly terrorist attacks using homemade explosives, I don't think this is something to be dismissed.
But they used well-known and readily available compounds. They don't attempt to use novel ones. It would just make the attack way more difficult to execute.
IIRC Germany experienced some rather horrific accidents in their facilities for manufacturing chemical weapons.
They've certainly given this some thought:
What do you mean, "permitted"? It's not illegal to run these sorts of computational jobs or publish the results.
I would rather than policymakers hear about the destructive potential of this technology before, and not after, an actual bad actor uses it.
There's a major difference between drafting and manufacturing.