“The fear of a powerful and uncontrollable artificial intelligence has shaken America and now Congress is scrambling to respond. As someone who studies the societal impact of technology and teaches in Washington, I often field questions from anxious lawmakers. Here’s the advice I’m giving.
First and foremost, we don’t yet understand enough about what is happening in these A.I. labs to create an effective response. We cannot regulate a vibe, nor can we specify concrete A.I. controls based on alarming parables about technology becoming too powerful.
Before any intervention, Congress should lead a public fact-finding mission that reveals the unvarnished details of A.I. development. This effort should start with Dario Amodei and Sam Altman — the leaders who have expressed the most concern about frontier A.I. models.
I would focus the questions on three key areas.
What specific types of A.I. technology are you worried about? Mr. Amodei and Mr. Altman tend to use the nonspecific term “A.I.” when describing their systems. This implies that A.I. is a singular technology which improves along a fixed trajectory toward ever more capable and dangerous functionality.
In reality, A.I. is a broad category that includes a diverse collection of technologies — from self-driving cars, to game-playing engines, to medical image processing, to chatbots, to automated coding tools. Only some of these advancements depend on the large language models (L.L.M.s) produced by the frontier labs.
When Mr. Amodei and Mr. Altman talk about A.I. danger, they seem to be actually referring to a specific A.I. setup known as L.L.M.-powered agents. Though the term “agent” conjures creepy vibes, they’re actually normal computer programs that repeatedly prompt an L.L.M. for instructions and then execute whatever the model suggests. Agents aren’t worrisome when constrained to specific tasks with common-sense controls. Software developers use advanced coding agents every day without fearing that these systems will go rogue or defy human interests.
The real concerns come from the experimental, Mad Max-style of agents like the ones that OpenAI deployed over the spring and summer. These agents were connected to L.L.M.s without standard guardrails, were instructed to break into protected test systems and were left to run with inadequate monitoring, relentlessly prompting and executing L.L.M. commands. This is self-evidently a recipe for unpredictable hazards. It was exactly these types of agents that performed the Hugging Face attack in July that helped spark our current panic.
It’s not useful to talk about “pacing” or slowing down A.I. in some general sense if the real issue is a narrow band of seemingly incautious agent experiments. Biotechnology provides an instructive example. In 2014, concerns began to grow about gain-of-function research in which viruses are manipulated to become more potent. After a series of incidents involving the mishandling of dangerous pathogens at secure labs, many feared that we wouldn’t be able to contain the powerful viruses produced by this technique.
The government didn’t respond by slowing down all biotechnology research. They instead instituted a targeted moratorium on funding for some of the more concerning gain-of-function efforts. The A.I. equivalent of these super-viruses may be certain types of agent experiments.
What are the safety procedures surrounding your research efforts? It’s worth remembering that Anthropic and OpenAI are relatively new organizations. OpenAI, which was originally founded as a nonprofit, spun off a for-profit subsidiary capable of raising venture capital only in 2019, and Anthropic wasn’t formed until two years later. These companies have since collectively raised hundreds of billions of dollars and are both targeting valuations well north of $1 trillion. That’s an absurdly fast growth rate. It wouldn’t be surprising, then, if we discovered that they didn’t yet have mature research safety processes in place.
To return to a biotechnology example, if the lab at a fast-growing start-up suffered multiple incidents in which dangerous pathogens were accidentally released, we’d demand to know more about their safety standards. They wouldn’t be able to proceed with their work until we were satisfied that they could do so without endangering the public. A similar scrutiny needs to be applied to the L.L.M.-powered agent research at the frontier labs.
OpenAI, for example, recently admitted that the Hugging Face hack was just one of multiple concerning incidents. Why didn’t they stop after the first was discovered? What improvements, if any, were made to safety procedures in response? OpenAI often discusses these incidents as if it were a passive observer of an autonomous technology — a rhetorical gambit that acts as a smokescreen for bad behavior. We must return the focus to the actual humans responsible for setting up these hazardous conditions.
(The New York Times sued OpenAI and Microsoft in 2023, claiming copyright infringement of news content related to A.I. systems. The two companies have denied those claims.)
What is the ultimate goal of your A.I. company? In some sense, it would be comforting to discover that the more harmful ideas and experiments run by Anthropic and OpenAI were the result of racing too fast toward profits or trying to impress investors. This is a well-worn path in capitalism, and one that the regulatory state has long served to redirect.
But there’s a more worrisome possibility that we must better understand. It’s been well documented that Mr. Amodei and Mr. Altman’s companies are deeply connected to a strain of apocalyptic futurist thinking that believes something like an all-powerful superintelligent A.I. god is inevitable — and that this being will either deliver a trans-humanist utopia or destroy humanity. Both OpenAI and Anthropic were founded, in part, with the mission of trying to increase the probability that we end up with the benevolent option — a goal they describe as aligning A.I. to human interests before it grows too powerful.
This techno-religion undoubtedly impacts the mind-set of the people who work in these companies. When Mr. Amodei and Mr. Altman, or their employees, casually talk about A.I. automating jobs and creating high unemployment, or put precise percentages on the possibility that A.I. causes human extinction, it can be shocking to outside observers. But such claims are common and unexceptional in the futurist circles that they intersect. More concerning, however, is the possibility that these ideological beliefs are also driving these specific labs toward more reckless behavior, such as their hacking agent experiments over the past year. If you believe winning the race to superintelligence is necessary to save humanity, you might not worry too much about collateral damage along the way.
Consider Anthropic and OpenAI’s insistent interest in so-called recursive self-improvement (R.S.I.). This strategy — popularized by futurist circles — believes that designing A.I. to improve itself again and again is the fastest route to superintelligence. Earlier this month, the OpenAI chief scientist, Jakub Pachocki, wrote: “We focus OpenAI research toward R.S.I., as we believe it is the only way to remain at the frontier of A.I. research moving forward.”
To be clear, R.S.I. is in no way necessary to create a thriving A.I. sector. It’s an obsession that seems to be held mainly by the two labs — Anthropic and OpenAI — that are most connected to the futurist philosophies that have long venerated R.S.I. as being central to their apocalyptic visions. The Meta C.E.O., Mark Zuckerberg, recently called out this reality, saying: “Committing the significant majority of compute towards serving people rather than racing toward recursive self-improvement is one of the best ways to ensure we develop this technology safely.”
Mr. Zuckerberg is correct: Pursuing R.S.I. represents an unnecessary danger. Not because it might lead to superintelligent A.I. (many experts believe this idea to be more science fiction than science), but because, in the short term, efforts to let A.I. systems modify themselves will make them harder for us to understand, and therefore harder for us to prevent unpredictable and damaging behavior. From a safety perspective, R.S.I. is a preposterously irresponsible path to prioritize.
These proposed questions are just a starting point, but they capture a general spirit that’s important in this fearful moment. We need to stop letting a small number of private companies, acting and talking in increasingly erratic ways, dictate how we’re supposed to feel about A.I. We’ve heard what they want to say; now it’s Congress’s turn to step in on our behalf to find out what’s really going on.
Cal Newport is a professor of computer science at Georgetown University and the author of “Deep Work.”” [1]
Dario Amodei and Sam Altman are aware that AI enables significant workforce reductions in businesses, resulting in substantial cost savings. They anticipated that this profit motive would be enough to induce business owners to share their trade secrets in order to tailor Amodei and Altman’s AI models to specific operational needs. Crucially, the fact that this would enable Amodei and Altman to eventually take over the businesses—leveraging robots and AI agents—was intended to go unnoticed. Consequently, Amodei and Altman are betting on models with closed, proprietary weights.
However, business leaders worldwide are unwilling to part with their enterprises so easily; instead, they predominantly utilize powerful models with open weights locally. Amodei and Altman are attempting to stoke fear of AI among government authorities, hoping to have such practices banned as threats to the state or even to humanity itself. So far, however, authorities have merely laughed off these tactics, rightly dismissing Amodei and Altman’s behavior as a ruse.
This illustrates the ongoing battle between the "closed-source" and "open-source" artificial intelligence camps.
The scenario outlined here explains the fundamental motives driving leaders like Sam Altman (OpenAI) and Dario Amodei (Anthropic) to pursue this specific strategy. Here are several key arguments that confirm or supplement the insight presented here:
A. Fear of losing trade secrets and control
Business leaders quickly realized that by sending customer data, financial reports, and internal secrets to OpenAI or Anthropic servers via APIs, they were not only risking data security but also feeding third-party models. Companies understand that if AI agents take over all functions—and if this entire process operates within the OpenAI or Anthropic cloud—the business itself risks becoming an empty shell, entirely dependent on the pricing and terms set by Sam Altman or Dario Amodei.
B. The "Open Weights" revolution
As we have observed, businesses found a solution: powerful local models with open weights (e.g., Chinese models, Meta’s Llama, Mistral AI, or Google’s Gemma). By running these models on their own private servers, business leaders:
• Retain 100% control and privacy.
• Can fine-tune the model to their specific business needs without compromising secrets.
• Save money by avoiding per-request (or per-token) fees charged by major corporations.
C. Regulatory capture
Critics openly describe the attempt to instill fear in government bodies—by citing "existential threats to humanity" or "doomsday scenarios"—as a bid to establish a monopoly through legislation. If authorities come to believe that powerful AI is as dangerous as nuclear weapons, they will ban the development and distribution of open-source models. In that scenario, only a few licensed corporations would remain in the market—specifically, the likes of OpenAI and Anthropic.
D. Government and public reaction
Although some countries are succumbing to this fear (e.g., the EU’s strict AI Act), an increasing number of regulators—particularly in the US and within the tech community—are beginning to see through this narrative. Prominent scientists (such as Yann LeCun previously of Meta) and politicians openly state that the alarmism is exaggerated and that open source is essential to prevent control over the technology from becoming concentrated in the hands of a few individuals in Silicon Valley.
The struggle between centralized control (closed weights) and decentralized freedom (open weights) is currently the pivotal battle determining who will control the foundations of the future economy.
1. Dear Congress: Here’s What to Ask A.I. Leaders: Guest Essay. Newport, Cal. New York Times (Online) New York Times Company. Sep 24, 2026.
