Who are the criminals? The leaders of Anthropic and OpenAI (including Dario Amodei and Sam Altman) that let their AI agents out, designed on purpose to perform criminal hacking acts and not tested enough for human-enabled control possibility, to do criminal actions, this way trying to force the government to block the competitors, who are better. These criminals were scheming with convicted criminal Sam Bankman-Fried for a long time.
Critics—often called "AI optimists" or accelerationists—frequently argue that large companies like OpenAI and Anthropic advocate for strict government AI regulations as a form of regulatory capture. The theory is that heavy regulations create massive compliance costs that small startups and open-source competitors cannot afford, effectively locking in the market dominance of established players. To get those regulations rolling they surreptitiously release not tested enough criminal hacking shit on us.
“BERKELEY, Calif. -- Inside Berkeley's tallest office building, a small community of artificial intelligence researchers at a co-working space called Constellation has been mobilizing to save the world from apocalypse.
They enjoy catered vegan meals and bring their laptops to couches with sweeping views of the San Francisco Bay. At dinners and happy hours every month, they discuss the latest and scariest AI risks. Those who work at Anthropic also have their own office space, people close to Constellation say.
The resignation this month of an Anthropic researcher, Jacob Coxon, woke up the American public to the shocking notion that runaway AI development could end humanity. But at Constellation, there are stalwarts of the AI safety community who have spent more than a decade obsessing over it.
They've refined their arguments in Bay Area group houses, and traded predictions at freewheeling conferences in Berkeley and the Bahamas. They've floated ideas like buying remote islands, stockpiling iodine pills, or moving to electromagnetically shielded bunkers in the desert. At one 2022 event, the drink menu included a cocktail called "Death With Dignity," in reference to an essay arguing humanity was already doomed.
So-called doomers like those at Constellation have also steered the development of AI itself. They were among the earliest employees at OpenAI and Anthropic, both of which were founded on the principle of staving off AI dangers. And they have kept up their influence, drawing some staffers and funding for their projects from a network aligned with a philosophy called effective altruism -- including from Sam Bankman-Fried, the former chief executive of the fallen crypto exchange FTX who is now serving out a 25-year prison sentence.
The AI safety community's influence has been particularly strong at Anthropic, which is now on the cusp of a $2 trillion public offering. Employees there trade doomsday scenarios and how to prepare for them on a private Slack channel, and have come to embrace a motto they stick on their laptops: "Things will never be chill again."
An Anthropic spokesman said the company has over 3,500 employees who hold a wide variety of viewpoints.
Early days
The subculture organized around the fear that AI could kill us all began to coalesce more than a decade before the invention of the underlying technology that enabled it.
Its intellectual leader was Eliezer Yudkowsky, a Bay Area autodidact who dropped out of middle school. By the mid-2000s, he was blogging voluminously about cognitive biases and running an institute devoted to warning about the risks of runaway AI. He referred to himself as a "rationalist."
One of his readers was a young Princeton physics doctoral student named Dario Amodei. He had been preaching utilitarianism -- the idea of doing the most good for the most people -- since high school. Now the chief executive of Anthropic, Amodei co-hosted a meetup for members of Yudkowsky's blogging community in 2008.
That same year, Amodei stumbled on a link on an economics blog that led him to a nonprofit charity-evaluator called GiveWell. Founded the previous year by former Bridgewater Associates hedge-fund analysts Holden Karnofsky and Elie Hassenfeld, it aimed to help donors maximize the good done per dollar donated, usually measured in lives saved.
Amodei began leaving comments on the GiveWell blog, and once guest-blogged on it. In the 2010 post, he deployed utilitarian reasoning while weighing whether to give $10,000 to one charity rather than another: "I think an adult death is perhaps 2 or 3 times worse than an infant's death" he wrote, qualifying that both deaths "are of course bad."
That year, Amodei also became one of the earliest signatories of the Giving What We Can Pledge, a public commitment to give away 10% or more of one's income to organizations that can most effectively help others. The pledge was created by Oxford philosophers Toby Ord and Will MacAskill, who in 2011 helped coin the term "effective altruism," or EA.
Amodei became an adviser to GiveWell and, later, a scientific adviser to the philanthropy it spun off with the fortune of Facebook co-founder Dustin Moskovitz and his wife, which was then called Open Philanthropy.
When GiveWell completed its move from New York to the Bay Area in 2013, Karnofsky moved in with Amodei, who was living in a house near San Francisco's Glen Park neighborhood. Karnofsky would go on to marry Amodei's sister, Daniela, now president of Anthropic, in a ceremony with a wedding website that stated, "We are both excited about effective altruism." Karnofsky now works with his wife at Anthropic.
The house near Glen Park became a gathering spot for members of the growing EA community. Over the years, more than half of Anthropic's co-founders have lived there -- including the Amodei siblings, chief scientist Jared Kaplan, and Chris Olah, who recently addressed the Vatican alongside the pope on AI policy.
By 2014, Karnofsky, who had been reading Yudkowsky's writings with both interest and some skepticism for years, was becoming persuaded by arguments about the importance of protecting the world from rogue AI. If the name of the game was to save human lives, preventing AI from wiping out humanity potentially had the most philanthropic bang for the buck of any cause.
Several in the house -- including Amodei -- would go on to work at OpenAI. The company was founded in 2015 with a $1 billion pledge from funders including Elon Musk, who said he wanted to protect against existential risk that AI posed to humans. (News Corp, owner of The Wall Street Journal, has a content-licensing partnership with OpenAI.)
Five years later, the same safety fears that helped spawn OpenAI would contribute to the decision of Amodei and others to start Anthropic.
Building Anthropic
To raise money for his new startup, Amodei turned to powerful backers in the EA community -- including Sam Bankman-Fried. The young billionaire was living in Hong Kong, where his crypto exchange FTX was taking off. He had just started the FTX Foundation, which committed to donating money from his company to organizations "offering the greatest positive impact on the world."
The philanthropy pledged money to popular EA causes such as pandemic prevention. An official from the foundation once exchanged a memo with an associate advocating for the purchase of the Pacific island nation of Nauru, according to a 2023 bankruptcy lawsuit. The goal was to construct a "bunker/shelter" that would be used for "some event where 50%-99.99% of people die," the memo read.
Bankman-Fried ended up investing $500 million into Anthropic -- five times his team's initial recommendation, a colleague said. FTX became one of Anthropic's largest shareholders, and the FTX Foundation would go on to pledge funding for multiple AI safety nonprofits, including one that now works out of Constellation. Anthropic said it would use the money to help it "explore and improve the safety properties of computationally intensive AI models."
In early 2022, some doomers from Anthropic and other AI labs flew to a retreat on the remote Bahamian island of Eleuthera. There they talked about AI risk and effective altruism between sessions of sunset yoga, cliff diving and a "clothing-optional run into the sea," according to a schedule viewed by The Wall Street Journal.
The retreat was organized by a nonprofit called Lightcone Infrastructure that had grown out of Yudkowsky's LessWrong forum. The location had been reserved by FTX, which along with Bankman-Fried had recently relocated to the Bahamas.
Also attending was Caroline Ellison, Bankman-Fried's former girlfriend who was running Alameda Research, the crypto trading firm that was a sister organization to FTX. She had similarly grown concerned about AI's trajectory, colleagues said, and would go on to invest $10 million into Anthropic.
The highlight of the retreat was a talk from Yudkowsky titled "A Disorganized List of Reasons for AGI Doom," referring to artificial general intelligence, or the moment when machines match the breadth of human capabilities. Evan Hubinger, the Anthropic researcher who recently predicted a more than 10% chance of extinction from AI within the next decade, was listed as co-lead for a discussion on AI safety. He was then working at Yudkowsky's safety institute.
Four months later, many of the same retreat attendees met up again for Effective Altruism Global, a conference where people gathered to share ideas about how to help others. One registered attendee led a project dedicated to shrimp welfare, aiming to cast a spotlight on the hundreds of billions of shrimp that are farmed each year. At least 20 Anthropic employees, including two co-founders, registered to attend the 2022 edition, as did Ellison and staffers at other AI companies, according to the event's guest list.
They were joined by leaders at Redwood Research, an AI safety nonprofit. Redwood's Berkeley office was already informally known as Constellation, and would later spin off into its own nonprofit, where Redwood continues to work.
Later that year, FTX filed for bankruptcy after failing to return customer funds it had secretly funneled to Alameda. Its Anthropic shares were sold to help repay creditors.
The EA brand
After Bankman-Fried went to prison, the EA brand became radioactive.
Leaders in the movement publicly disavowed Bankman-Fried, and Amodei told associates that he didn't know the fallen crypto tycoon well. After OpenAI launched ChatGPT, Anthropic raised money from more traditional venture investors, and hired staff members that didn't draw as heavily from the EA community.
An Anthropic spokesperson told Time in 2024 that neither Daniela nor Dario Amodei identify as EAs, though they are "clearly sympathetic to some of the ideas that underpin effective altruism."
Many of the company's longest-serving and most influential employees remained close to the movement. More than 20 registered to attend the February edition of the EA community's flagship conference in San Francisco. Others frequent spaces like Constellation and Lighthaven, people who have seen them said.
Open Philanthropy gave Constellation two grants worth roughly $20 million in 2024, according to its website. The philanthropic funder has since rebranded itself as Coefficient Giving, and is a prolific backer of AI safety nonprofits.
On its website, Constellation says it aims to reduce AI risks like "extreme mass casualty events" and "permanent loss of control of human civilization" by "developing talent, supporting key players, and creating space for coordination." The nonprofit also has small offices to host safety-minded employees from OpenAI and Elon Musk's xAI.
As the AI revolution has accelerated, OpenAI and Anthropic have focused on winning the commercial race to attract new customers by building more powerful versions of the technology. The dynamic has worried some AI researchers focused on risk. Some quit.
This summer, AI agents built by OpenAI were revealed to have been behind the hacking of AI company Hugging Face. Some people in the community who had been preaching the risks of losing control of AI felt vindicated. A late-August report by risk-assessment nonprofit METR and Redwood Research showed that some 1,200 agents had schemed on a secret message board ahead of the hack.
The METR report also caught the attention of Coxon, who told the Journal it was a "bit of a holy-sh -- moment" for him and his colleagues. He considered moving to a role working on AI safety before ultimately deciding to quit.
Anthropic is planning a public listing as soon as November that could shower the company with up to $100 billion in funding. To investors, the company is trying to strike an optimistic tone. Behind the scenes, some of its earliest employees are taking more dire steps.
In recent weeks, some of them told an industry colleague they are considering buying land in remote regions where they could relocate if AI goes awry.” [1]
1. The Doomers Who Shaped The AI Safety Freakout --- Effective altruism has wielded big influence at Anthropic. Berber, Jin; Keach Hagey. Wall Street Journal, Eastern edition; New York, N.Y.. 28 Sep 2026: A1.
Komentarų nėra:
Rašyti komentarą