This sounds suspiciously like manufacturing weapons? I’m most interested to understand how this project was permitted in the first place. Even the public disclosure of this research feels negligent.
Freedom of speech. In the free world, you’re free to plan almost anything from planning grocery procurement to dreaming global thermonuclear war, given you DON’T PUT BAD STUFF IN ACTION. That’s where and how the line is drawn. There are dangerous extensions to the definition of action but that’s the theory.
Also if it’s not put it in action but rather discussed publicly, that’s just helping the society prepare and evolve.
No, the US has long classified speech on weapons of mass destruction, like how to configure shaped charges to trigger atom bombs. You're not free to circumvent those restraints. It's likely that identifying molecules suited only to become bioweapons or publishing how to synthesize them will also fall under the same strictures, as necessary to ensure public safety.
Free speech is a principle to guide laws, not a law unto itself. Protecting speech that makes people less free is oxymoronic.
The United States has had "born secret" laws about nuclear weapons details for decades [1], but it's unclear if they are actually compatible with the Constitution. There was a case in 1979 that could have established whether "born secret" nuclear information is a valid exception to the First Amendment, but the government dropped its case before the courts could rule on it [2]. The "born secret" concept does not apply to biological or chemical weapons.
Your nearest research university library probably holds publications about the effects and synthesis of chemical warfare agents, like Some aspects of the chemistry and toxic action of organic compounds containing phosphorus and fluorine by Bernard Charles Saunders: https://catalog.lib.uchicago.edu/vufind/Record/587946
Classification only applies to people who've agreed to keep the classified info secret; everyone else is protected by the 1st amendment. See United States v. Progressive, Inc. (which did _not_ rule this, as it was dismissed first.)
> It's likely that identifying molecules suited only to become bioweapons or publishing how to synthesize them will also fall under the same strictures, as necessary to ensure public safety.
Certainly. However, here they have identified and published not the discovered molecules but only the identification process (and only in fairly broad strokes, at that), and shied away from working further on synthesis (we know enough to state that synthesis would be possible and practical for nearly any candidate molecule).
Sure, as a society we can expand ethical guidelines to some sort of "thou shall not optimize for toxicity", but enforcing it poses some serious challenges, not least of which is the fact that this has demonstrated that the process works even with fairly innocuous and public training data.
Expect to see a bunch of funding for research into technologies around toxin identification and detection, as well as the rapid creation and synthesis of antitoxins and other prophylactic measures (cheaper filtering, maybe). Perhaps even eventual (as in a decade out at least) "hardening" of organisms against toxicological weapons (possibly positioned as research into the mechanisms of acquiring pesticide resistance and similar).
No, the US has long classified speech on weapons of mass destruction, like how to configure shaped charges to trigger atom bombs.
I'm pretty sure that whatever a private individual figures out on his own is not "classified", since it's not a government secret. Is there actually a US law banning people from even talking about WMDs?
Are you going to spell out in the law exactly what people aren't allowed to research? You've just implemented a holding pattern while giving them a road map on what to research. So your holding pattern is going to be set up to buy you x number of years, where you assume it will take your adversary x number of years to fill in the gaps on the map you just gave them.
No, from what I've observed, such laws most directly constrain those who fund research as well as those who solicit/promote it. Often they accompany other criminal charges rather than motivate prosecution as a primary charge.
Security and espionage laws are often not directly enforceable. Their value is to "pile on", adding severity to related matters, such as charges committing or aiding past terrorist acts or planning therefor. Ideally the threat of direct prosecution for a crime would deter possible perpetrators, but making penalties more severe and extending incarceration are useful roles for laws too.
That applies to government employees, who willingly agree to be bound by classification. I think that if someone outside had the information, they could publish it (probably a very bad idea).
Also, the press has a right to publish classified information.
Right, just don't take the plea deal, make bail, get a lawyer that can appeal every motion, and stay solvent through the first appeals court, the supreme court, the remand back to trial court, and the subsequent ruling, hope the prosecutor is bored enough to avoid a retrail, and don't get into crime during those 8 years so you get your bail back, and hope you don't get tried at the state level
If you can actually afford your rights, be my guest!
No one has to apply for a license to use machine learning models. They are merely communicating the fact that it's so trivially easy to do something like this, that pretty much anyone with an interest will be able to do so. The question is whether it was already trivially easy to design such compounds before or not, which I think it was, but I would be glad to hear a counterpoint.
Synthesis was, and still is, the hard part. It was already easy to find public information on seriously nasty nerve agents.
For some intuition on the difficulty in synthesis, note that explosive chemistry and synthesis is comparatively trivial, yet there are relatively few terror attacks that go beyond commercial-off-the-shelf compounds and almost none that do it well.
Personally, I think this model would be a boon for public safety because contract synthesis operations could use it to screen incoming requests for "nerve gas but changed up a bit." Basic chemical intuition probably already gets them far in this regard, but a published model could be standardized and mandated.
Someone else in the thread asked about next steps. Those would be good next steps.
there are relatively few terror attacks that go beyond commercial-off-the-shelf compounds and almost none that do it well
The 2011 Norway attacks, 2002 Bali bombings, 2005 London underground bombings, 2008 Mumbai attacks... There is a long list of deadly terrorist attacks using homemade explosives, I don't think this is something to be dismissed.
There is a long list of deadly terrorist attacks using homemade explosives
But they used well-known and readily available compounds. They don't attempt to use novel ones. It would just make the attack way more difficult to execute.
We are but one very small company in a universe of many hundreds of companies using AI software for drug discovery and de novo design. How many of them have even considered repurposing, or misuse, possibilities? Most will work on small molecules, and many of the companies are very well funded and likely using the global chemistry network to make their AI-designed molecules. How many people have the know-how to find the pockets of chemical space that can be filled with molecules predicted to be orders of magnitude more toxic than VX?
Comments
This sounds suspiciously like manufacturing weapons? I’m most interested to understand how this project was permitted in the first place. Even the public disclosure of this research feels negligent.
Freedom of speech. In the free world, you’re free to plan almost anything from planning grocery procurement to dreaming global thermonuclear war, given you DON’T PUT BAD STUFF IN ACTION. That’s where and how the line is drawn. There are dangerous extensions to the definition of action but that’s the theory.
Also if it’s not put it in action but rather discussed publicly, that’s just helping the society prepare and evolve.
No, the US has long classified speech on weapons of mass destruction, like how to configure shaped charges to trigger atom bombs. You're not free to circumvent those restraints. It's likely that identifying molecules suited only to become bioweapons or publishing how to synthesize them will also fall under the same strictures, as necessary to ensure public safety.
Free speech is a principle to guide laws, not a law unto itself. Protecting speech that makes people less free is oxymoronic.
The United States has had "born secret" laws about nuclear weapons details for decades [1], but it's unclear if they are actually compatible with the Constitution. There was a case in 1979 that could have established whether "born secret" nuclear information is a valid exception to the First Amendment, but the government dropped its case before the courts could rule on it [2]. The "born secret" concept does not apply to biological or chemical weapons.
Your nearest research university library probably holds publications about the effects and synthesis of chemical warfare agents, like Some aspects of the chemistry and toxic action of organic compounds containing phosphorus and fluorine by Bernard Charles Saunders: https://catalog.lib.uchicago.edu/vufind/Record/587946
Which is also scanned and available online: https://archive.org/details/B-001-026-884-ALL
[1] https://en.wikipedia.org/wiki/Born_secret
[2] https://en.wikipedia.org/wiki/United_States_v._Progressive,_....
Classification only applies to people who've agreed to keep the classified info secret; everyone else is protected by the 1st amendment. See United States v. Progressive, Inc. (which did _not_ rule this, as it was dismissed first.)
> It's likely that identifying molecules suited only to become bioweapons or publishing how to synthesize them will also fall under the same strictures, as necessary to ensure public safety.
Certainly. However, here they have identified and published not the discovered molecules but only the identification process (and only in fairly broad strokes, at that), and shied away from working further on synthesis (we know enough to state that synthesis would be possible and practical for nearly any candidate molecule).
Sure, as a society we can expand ethical guidelines to some sort of "thou shall not optimize for toxicity", but enforcing it poses some serious challenges, not least of which is the fact that this has demonstrated that the process works even with fairly innocuous and public training data.
Expect to see a bunch of funding for research into technologies around toxin identification and detection, as well as the rapid creation and synthesis of antitoxins and other prophylactic measures (cheaper filtering, maybe). Perhaps even eventual (as in a decade out at least) "hardening" of organisms against toxicological weapons (possibly positioned as research into the mechanisms of acquiring pesticide resistance and similar).
I'm pretty sure that whatever a private individual figures out on his own is not "classified", since it's not a government secret. Is there actually a US law banning people from even talking about WMDs?
Yeah but how is this going to be enforced?
Are you going to spell out in the law exactly what people aren't allowed to research? You've just implemented a holding pattern while giving them a road map on what to research. So your holding pattern is going to be set up to buy you x number of years, where you assume it will take your adversary x number of years to fill in the gaps on the map you just gave them.
Is that enough?
No, from what I've observed, such laws most directly constrain those who fund research as well as those who solicit/promote it. Often they accompany other criminal charges rather than motivate prosecution as a primary charge.
Security and espionage laws are often not directly enforceable. Their value is to "pile on", adding severity to related matters, such as charges committing or aiding past terrorist acts or planning therefor. Ideally the threat of direct prosecution for a crime would deter possible perpetrators, but making penalties more severe and extending incarceration are useful roles for laws too.
That applies to government employees, who willingly agree to be bound by classification. I think that if someone outside had the information, they could publish it (probably a very bad idea).
Also, the press has a right to publish classified information.
Right, just don't take the plea deal, make bail, get a lawyer that can appeal every motion, and stay solvent through the first appeals court, the supreme court, the remand back to trial court, and the subsequent ruling, hope the prosecutor is bored enough to avoid a retrail, and don't get into crime during those 8 years so you get your bail back, and hope you don't get tried at the state level
If you can actually afford your rights, be my guest!
It is illegal to conspire to commit crimes, at least many of them.
No one has to apply for a license to use machine learning models. They are merely communicating the fact that it's so trivially easy to do something like this, that pretty much anyone with an interest will be able to do so. The question is whether it was already trivially easy to design such compounds before or not, which I think it was, but I would be glad to hear a counterpoint.
Synthesis was, and still is, the hard part. It was already easy to find public information on seriously nasty nerve agents.
For some intuition on the difficulty in synthesis, note that explosive chemistry and synthesis is comparatively trivial, yet there are relatively few terror attacks that go beyond commercial-off-the-shelf compounds and almost none that do it well.
Personally, I think this model would be a boon for public safety because contract synthesis operations could use it to screen incoming requests for "nerve gas but changed up a bit." Basic chemical intuition probably already gets them far in this regard, but a published model could be standardized and mandated.
Someone else in the thread asked about next steps. Those would be good next steps.
The 2011 Norway attacks, 2002 Bali bombings, 2005 London underground bombings, 2008 Mumbai attacks... There is a long list of deadly terrorist attacks using homemade explosives, I don't think this is something to be dismissed.
But they used well-known and readily available compounds. They don't attempt to use novel ones. It would just make the attack way more difficult to execute.
IIRC Germany experienced some rather horrific accidents in their facilities for manufacturing chemical weapons.
They've certainly given this some thought:
What do you mean, "permitted"? It's not illegal to run these sorts of computational jobs or publish the results.
I would rather than policymakers hear about the destructive potential of this technology before, and not after, an actual bad actor uses it.
There's a major difference between drafting and manufacturing.