In the bustling halls of Congress last year, a stark warning echoed from Dario Amodei, the chief executive of the prominent artificial intelligence (AI) start-up Anthropic. He cautioned that emerging AI technologies could soon enable even unskilled and malicious actors to orchestrate large-scale biological attacks, unleashing viruses or toxic substances capable of widespread disease and death.

Amodei’s words sent shockwaves through the Senate chambers, igniting alarm across party lines. Meanwhile, AI researchers in industry and academia engaged in heated debates, grappling with the gravity of the threat he had outlined.

Now, over 90 biologists and scientists specializing in AI-driven protein design have united, signing an agreement that seeks to ensure their groundbreaking research remains a force for good, without exposing the world to catastrophic harm.

Among the signatories are luminaries such as Nobel laureate Frances Arnold, representing laboratories from the United States and beyond. These pioneers argue that the latest AI technologies hold far more promise than peril, paving the way for new vaccines, life-saving medicines, and scientific breakthroughs yet unimagined.

“As scientists engaged in this work, we believe the benefits of current AI technologies for protein design far outweigh the potential for harm, and we would like to ensure our research remains beneficial for all going forward,” the agreement reads, a rallying cry for responsible innovation.

The accord does not seek to suppress the development or distribution of AI technologies. Instead, the biologists aim to regulate the use of equipment needed to manufacture new genetic material – the critical link that could transform theoretical designs into tangible bioweapons.

“Protein design is just the first step in making synthetic proteins,” explained David Baker, the director of the Institute for Protein Design at the University of Washington, who played a pivotal role in shepherding the agreement. “You then have to actually synthesize DNA and move the design from the computer into the real world – and that is the appropriate place to regulate.”

This initiative is part of a broader effort to weigh the risks and rewards of AI, as experts sound alarms about the technology’s potential to spread disinformation, displace jobs at an unprecedented rate, and – in the most dire scenarios – imperil the very existence of humanity itself. Tech companies, academic labs, regulators, and lawmakers find themselves at the forefront of a complex challenge: understanding these risks and devising strategies to address them.

Amodei’s congressional testimony struck a chord, as he contended that large language models (LLMs) – the cutting-edge technology powering online chatbots – could soon aid attackers in developing new bioweapons. However, he acknowledged that such a capability does not exist today. In fact, Anthropic’s own detailed study revealed that, for someone attempting to acquire or design biological weapons, LLMs offered only marginally more utility than a standard internet search engine.

While Amodei and others worry that the convergence of improving LLMs and other technologies could give rise to a serious threat within two to three years, OpenAI – the creators of the renowned ChatGPT chatbot – conducted a similar study that found LLMs pose no significantly greater danger than search engines. Aleksander Mądry, a computer science professor at MIT and OpenAI’s head of preparedness, stated that while researchers will undoubtedly continue refining these systems, he has yet to encounter evidence suggesting they could create novel bioweapons.

Current LLMs are trained on vast troves of digital text scraped from the internet, enabling them to regurgitate or recombine existing information, including data on biological attacks. However, in the quest to accelerate the development of new medicines, vaccines, and other beneficial biological materials, researchers are beginning to construct similar AI systems capable of generating original protein designs.

Biologists acknowledge that such technology could aid attackers in designing biological weapons, but they emphasize that actually constructing these weapons would necessitate multi-million-dollar laboratories equipped with DNA manufacturing equipment.

“There is some risk that does not require millions of dollars in infrastructure, but those risks have been around for a while and are not related to AI,” said Andrew White, a co-founder of the nonprofit Future House and one of the biologists who signed the agreement.

The biologists’ call to action includes developing security measures to prevent DNA manufacturing equipment from being misused with harmful materials – though the specifics of these measures remain unclear. They also advocate for safety and security reviews of new AI models before their release.

Notably, the agreement does not argue for bottling up these technologies or restricting their dissemination. As Rama Ranganathan, a professor of biochemistry and molecular biology at the University of Chicago, and a signatory of the agreement, stated, “These technologies should not be held only by a small number of people or organizations. The community of scientists should be able to freely explore them and contribute to them.”