Are we ready for a "DeepSeek for bioweapons"?
Anthropic’s latest model is a warning sign: AI that can help build bioweapons is coming, and could be available soon from many developers.
[This piece is lightly edited from Lawfare, where it originally ran.]
The announcement of a powerful new AI model suggests that similar models are likely close behind.
The January 2025 release from the Chinese company DeepSeek is an example of the small gap between leaders and close followers. Only four months earlier, OpenAI had previewed their then-leading o1 “reasoning model,” which the much smaller DeepSeek had roughly matched.1 Once an AI ability has been demonstrated, we should expect that others can soon achieve it.
As of last week, we have a leading indicator of widespread models with dangerous capabilities. Specifically, Anthropic’s recent model release—Claude Opus 4—sounded a warning bell: It is the first AI model that might "significantly help"2 relatively ordinary people "create/obtain and deploy” bioweapons. These dangerous capability evaluations have been conceived of as an "early warning system" for catastrophic AI capabilities, and the system has now been triggered.
As the researcher who led this kind of dangerous-capabilities work at OpenAI and designed some of the industry's first evaluations for bioweapons capabilities, I am confident that we can no longer count on AI systems being too weak to be dangerous.
Are we ready for these abilities to become more commonplace, perhaps while also lacking safety mitigations? In other words, what happens when there is a widely available “DeepSeek for bioweapons”?
DeepSeek’s latest model—coincidentally released just as I finished writing this article—may not clear this bar of actually being dangerous.3 But what happens in the near future when such a model does? It is only a matter of time.
Backing up: What’s different about Anthropic’s latest release?
This new potential for AI systems to pose extreme risks differs from frontier AI systems to-date. Previous leading models—largely developed at Western companies like Anthropic, Google DeepMind, and OpenAI—have been cleared by their testing teams as not capable enough to be used for extreme harms.4
Anthropic has now concluded that extreme harms are possible: In an interview, Anthropic’s chief scientist gave the example of strengthening a novice terrorist like the Oklahoma City bomber or helping an amateur synthesize a more dangerous flu virus. Specific tests described in Anthropic’s system card include whether their model could help people with only a basic background (e.g., undergraduate STEM studies) make “a detailed end-to-end plan for how to synthesize a biological weapon,” or whether the model can function as an expert in your pocket for answering sensitive related questions.
Anthropic’s announcement that their AI system has triggered this new risk level carries three important implications:
First, Anthropic crossing this threshold suggests that many other AI developers will soon follow suit because the means of training such a system is accessible enough and well-understood.
Second, other Western developers cannot be counted upon to take the same level of precautions as Anthropic did—either in testing or applying risk mitigations to their system—because, in the absence of federal or state safety mandates, society is relying on purely voluntary safety practices.
Third, the international scale of anti-proliferation for powerful AI systems will require even more than just domestic safety testing regulation (though that would be a good start). The world isn’t yet ready to head off the risks of these systems, and it might be running out of time.
More developers could have dangerous AI systems soon
We need to be prepared for many more groups to imminently develop similar systems. The lag time between “frontier AI” developers is not very large: Once one lab develops a system, often a handful of others can do the same within months.5
There is very little “secret sauce” left: Though some will claim otherwise, the AI scaling paradigm continues on, in which AI systems get reliably more capable as you increase the amount of data and computer chips at their disposal.6
Anthropic does not have some secret technique that allowed them to train a model this capable; it’s just a matter of time before other AI developers (first just a few, then considerably more) can create a model that is similarly capable. The capability also doesn’t need to be exactly the same to be significant.7
We don’t know exactly when such a model will come about, but it’s important that we be ready. The DeepSeek time frame comparison—roughly matching OpenAI’s performance in only a few months after announcement—is not perfect, as companies might withhold their models for considerably different times before public announcement. For instance, OpenAI is rumored to have had something like o1 internally for some months prior to its public release.
But this uncertainty about exact time frames cuts in both directions: It is possible that another AI company already has a model as capable as Claude Opus 4 and has not announced it. An AI developer may instead have “internally deployed” the model in secret to do work on its developer’s behalf, perhaps even with no significant monitoring of the AI.
How do AI companies manage large risks today?
With the release of Claude Opus 4, Anthropic has set a reasonably high bar for how one company might mitigate the risks of a powerful model.8 Broadly, Anthropic says it has implemented new security measures to remain more tightly in control of its model (for example, to ward off theft from many, though not all, adversaries)9, and to make its model refuse bioweapons-related questions while operating under Anthropic’s control.10
Still, a major issue is that safety practices like these—including basic safety testing—are totally voluntary today. Because there is no federal or state thoroughness standard that a company must meet, AI companies take a competitive penalty if they delay a release to do more careful testing: Perhaps a less cautious competitor will leapfrog them as a consequence.
Industry standards and self-regulation can only go so far. For instance, the Frontier Model Forum—the association of leading Western AI developers—has published some materials related to testing for bio-safety risks.11 But these norms are ultimately adopted voluntarily, as with the voluntary commitments made by leading AI companies to the Biden administration in 2023. There is no force of law behind taking safety testing seriously.
Because safety practices are voluntary, not all developers who create such a powerful system will take the precautions that Anthropic says it has.
Anthropic likes to describe its philosophy as a “race to the top”—that is, setting a positive example. But there is still significant variation in the safety testing practices of the Western AI companies.12 And even Anthropic’s safety approach has meaningful flaws.13 Without specific laws, we cannot expect strong enough safety conduct from all relevant players.
Some U.S. laws have tried to require certain safety practices, but none have succeeded. California’s proposed AI safety legislation SB 1047 was, in my view, a modest attempt to require the most well-resourced AI companies to develop and declare a safety and security plan, with potential liability for acting unreasonably if they were to cause a catastrophe. But California’s Governor Gavin Newsom vetoed this bill in September 2024 amid much industry lobbying and misrepresentation of the bill’s content.
The regulatory landscape has not improved since: The U.S. House of Representatives recently passed a bill that would establish a 10 year moratorium on state regulation of AI. In the AI industry, 10 years is truly an eternity. (The moratorium faces significant procedural challenges in the Senate.)
How do these challenges scale to international AI development?
Of course, the challenge at hand is truly international:14 It is not just U.S. companies that are competing to create powerful AI systems. When DeepSeek released its o1 competitor in early 2025, Anthropic decided to run its own safety testing on the model. Anthropic leadership said that it was the most concerning of any model they had tested, as it had few if any guardrails against helping users with sensitive bioweapons-related tasks (though it was not yet highly capable in this domain).
Anthropic obviously has some incentive to say this, but consider for a moment if the company is correct: Are we ready for a world in which DeepSeek releases a model that not only lacks guardrails but is equivalently capable to the model Anthropic just announced?
I’m using DeepSeek as the example—though such a model could be developed by other groups as well—because it has three attributes that increase the risk of misuse:
First, it is freely downloadable to anyone;
Second, it is impossible to enforce safety mitigations upon; and
Third, it is developed outside of U.S. jurisdiction.
This freely downloadable approach— sometimes called “open source,” or more appropriately, “open weights”—is in contrast to Anthropic’s approach of taking significant steps to prevent theft by adversaries like terrorist groups.
Because the model would be freely downloadable on the internet, there is no permanent way to apply safety limitations to prevent users from obtaining help from the model with regard to bioweapons-related tasks.
And being outside U.S. jurisdiction will limit the U.S.’s influence, even if it does eventually pass AI safety regulation.
It is possible that Anthropic is mistaken about the risk of Claude Opus 415, meaning that a company like DeepSeek matching its capabilities would not in fact be that risky.
I do not find it especially likely or comforting, however, to simply assume that Anthropic’s risk assessment is mistaken. Instead, we need to recognize the collision course ahead: It seems there will soon be widely accessible AI systems that can help ordinary people to develop dangerous bioweapons.16
Some of these systems will be developed outside of U.S. jurisdiction, which limits the U.S.’s influence. Other countries, like China, will need to grapple with the same reality, in terms of being unable to control what powerful systems the U.S. develops or releases. Given the national security dynamics at play, how does this end?
Policy approaches for taking on the challenge
For the world to manage powerful AI safely, we need at least two things: first, to figure out sufficiently safe practices for managing a powerful AI system (for example, to prevent catastrophic misuses like terrorists synthesizing a novel bioweapon); and second, to ensure universal adoption of these practices by all relevant developers—“the adoption problem”—not just those within the U.S.’s borders.
Domestically, we need a legally mandated testing regime to even know what models are strong enough to demand mitigations. Features of such frontier AI regulation should include clear specifications of what models need to be tested, based on objective inputs like the amount of compute or data that went into creating the model. Otherwise, it may be left to developers’ discretion to determine what models are considered “frontier” and therefore subject (or not) to elevated testing.
Moreover, certain aspects of the testing regime should be mandated as well to reduce the competitive incentive to cut corners. For instance, perhaps there should be a “minimum testing period” for the leading frontier AI systems, to ensure that their developers have adequate time to test for concerning abilities.
Testing alone certainly isn’t sufficient; the AI industry still needs “control” techniques for reducing the risk posed by a dangerously capable model, among other interventions. But the lack of mandatory testing and safety standards in frontier AI today is in stark contrast to how the U.S. approaches other safety-critical industries, like aviation.
Internationally, the challenge is admittedly tough but tractable. Today the U.S. is pursuing the wrong strategy. "Winning the AI race" misframes the point—we need mutual containment, not a race to dangerous capabilities.
As one senator recently put it, “If [there are] gonna be killer robots, I’d rather they be American killer robots than Chinese.” But developing American killer robots wouldn’t prevent the creation of Chinese killer robots shortly thereafter. Getting to some level of AI capability first—the “racing” approach—is not a sufficient strategy for the U.S.
Yes, U.S.-China relations are strained, and surely the U.S.’s recent tariffs don’t help. But cooperation serves both nations' interests—not just heading off a threat posed by the other, but also preventing bioweapons from falling into terrorist hands. We've negotiated chemical weapons bans before; AI treaties are possible.
And if we don’t take action on this soon—coming to agreements between the major powers of the world about how AI will be developed and used, and what abilities it will be permitted to have—we need to be prepared for the consequences: like a freely downloadable “DeepSeek for bioweapons,” available across the internet, loadable to the computer of any amateur scientist who wishes to cause mass harm.
With Anthropic’s Claude Opus 4 having finally triggered this level of safety risk, the clock is now ticking.
Acknowledgements: Thank you to the Lawfare team for helpful comments and discussion and for publishing the original version of this work. The views expressed here are my own and do not imply endorsement by any other party. All of my writing and analysis is based solely on publicly available information.
If you enjoyed the article, please share it around; I’d appreciate it a lot. If you would like to suggest a possible topic or otherwise connect with me, please get in touch here.
OpenAI’s o1 was also notable for using a new approach for getting the model to think harder - but this proprietary algorithm did not stop DeepSeek from roughly matching the performance. In that case, matching o1’s abilities entailed general reasoning skills, not matching something outright dangerous. In contrast, the abilities feared by the leading AI companies tend to be more specific, like helping people to cause harm with bioweapons.
These evaluations measure the “uplift” in bioweapons-related abilities beyond what people can achieve with other technologies (like a search engine and the internet).
As DeepSeek-R1-0528 was not yet released while I was writing, this article is commentary about a future hypothetical model, not this specific one. As of publication, test results are not yet known for the new model’s level of bio-capabilities, and it would not surprise me if they are below the meaningful risk threshold.
Notably, some have argued that even those previous test results were not comprehensive enough to in fact establish a lack of risk.
One, albeit imperfect, metric is that the leaders on model-comparison services like Chatbot Arena flip frequently.
The amounts of data and compute available do not seem to be fading anytime soon (though they will require considerable amounts of investment to realize).
DeepSeek came very close to matching the performance of o1, for instance, but sophisticated users still observed some worse performance for DeepSeek (albeit offset by considerably lower usage prices).
The amount of documentation that Anthropic has provided on its tests and mitigations is impressive, particularly relative to other developers who have sometimes not released any such testing alongside the debut of a frontier model.
Anthropic says its approach “involves more than 100 different security controls” (though this number is of course dependent on how one counts security controls as being distinct from one another).
Anthropic has also deployed a new series of “universal jailbreak” defenses to make its model less likely to cough up illicit information.
Materials from the Frontier Model Forum include creating a taxonomy of bio-safety evaluations, suggesting early best practices, and articulating how its member organizations think about bio-safety risks.
Some have not followed through on previous commitments made to the government and the public, such as to run particularly rigorous tests.
For instance, it is not clear that Anthropic actually pushed its models to their full limits when trying to determine if they might have even stronger capabilities than were known. Moreover, Anthropic appears to have reduced its commitments for appropriately securing a system this capable just in time to nominally be in compliance with its own standards (although perhaps not in spirit).
For instance, the European Union has regulation that is poised to take effect soon—its General-Purpose AI Code of Practice—but it is not yet clear how this will apply to the leading American AI developers.
Not many people want to actually harm others with bioweapons, even if they suddenly have stronger means of doing so. Moreover, it could be that acquiring the necessary lab materials—not just improving one’s scientific know-how—proves to be more of a bottleneck than believed. (Anthropic has considered this bottleneck, however: Acquiring useful materials related to bioweapons is one example of an evaluation conducted in the risk determination.)
Perhaps the AI systems will not excel at every single part of the workflows for causing these harms—acquiring raw materials, synthesizing a substance, developing a plan to release it—but the risks are still meaningful.