Former Anthropic researcher: AI employees are "genuinely afraid" for humanity's future

 

 People working on the technology were "genuinely frightened" by how quickly it was developing and what it would imply for humanity, according to an artificial intelligence researcher who left Anthropic.

 

"I believe that if we don't slow down at the current rate of progress, there is a strong chance that we could all die in the immediate future," he said.

 

In light of mounting safety worries about the sector, Jacob Coxon was speaking to the BBC after his resignation article on the risks of AI went viral.
Dario Amodei, the head of Anthropic, the 27-year-old's previous employer, recently urged AI development to slow down, however others have questioned the reasons for this.

 

Elon Musk of xAI and Sam Altman of OpenAI, the leaders of two competing AI companies, have both stated that they support Amodei's proposal for independent oversight of AI model development as well as industry-wide slowing and regulation.

 

In an essay published on Saturday, Amodei stated that while the technology's development was unquestionable, the hazards involved were "serious" and that businesses and governments needed time to handle them.

 

The idea of a delay was supported by Coxon, who was employed at OpenAI before joining Anthropic, but he stated that it would need to be coordinated with China in order to prevent "a race at an international scale".

 

"Because they are caught in a race, the employees of these corporations take their requests for regulation very seriously. He said on Sunday with Laura Kuenssberg, "And they're afraid of the results of that race."

 

Anthropic's CEO, Dario Amodei, calls for a slowdown in AI research. 

 

According to Coxon, the most difficult thing to answer was what an AI apocalypse might look like.

One of the dangers mentioned in Amodei's remarks was that the internet might be taken over by a swarm of bots operating like a supercomputer.

 

This scenario might be feasible in six months to a year, according to Coxon.

 

In response to Coxon's departure, an Anthropic spokesperson told BBC News: "We have "We have always been transparent that AI will bring both enormous benefits and unprecedented risks," an Anthropic representative told BBC News in response to Coxon's exit.

 

"To address these risks, we continue to build models with some of the strongest safeguards in the industry."

The representative continued, "The company has been a pioneer in studying how AI models work." In addition to "aggressively" testing its models and disseminating the results to support scrutiny and stop instances of "AI misalignment," it was the first to offer a strategy for reducing the risks associated with their creation.

 

"This work is also why we believe the world would benefit from the industry adopting a "This work is also why we believe the world would benefit from the industry adopting a lawful, verifiable way to work together to pace how we release powerful models," they stated.

According to Coxon, his colleagues were concerned that the threat might materialize within the next two years.

He added people at AI firms were "planning what to do with their lives and thinking about the impacts of their work", while some were "considering buying land somewhere because they're so scared of the instability as a result of rapid AI progress".

 

 

"They all keep this in their head on a daily basis while working on the technology."

 

 

However, Coxon expressed some hope for AI's future to the BBC, stating that researchers "genuinely want to see the upside" of "solving diseases and improving everyone's lives."

 

Over 10% of anthropologists believe AI "could kill all humans." 

 

Since Coxon's social media post, numerous other members of the industry, like anthropic scientist Evan Hubinger, have also expressed their worries.

"We sincerely think AI has the potential to wipe out humanity! In my opinion, it will be more than 10% over the next ten years," Hubinger stated.

The CEO of AI safety company Faculty, Marc Warner, told the BBC that it was "extremely hard to place a probability" that AI will kill all people.

 

"But it's important to recognize that these people are very sincere in what they're saying," he continued, pointing out that the leaders of various AI companies have long voiced concerns about the hazards associated with AI.

 

Despite being generally upbeat, former prime minister Rishi Sunak, a paid advisor to Anthropic, expressed concern in the Sunday Times about the threat AI poses to humans.

 

 

Dramatic insider warnings concerning AI are rejected by some in Silicon Valley.

 

Nonetheless, some business leaders have hinted that remarks regarding the dangers and potential of AI might be intended to create excitement.

 

 "Sorry, but asking Jacob [Coxon] about AI extinction risk is like asking your AC guy about climate change," Clement Delangue, chief executive of the AI platform Hugging Face, wrote on social media. I'm not suggesting it's inherently boring or incorrect, but let's keep things in perspective."

 

However, Delangue offered to contribute to the possible answers the Anthropic boss suggested once Amodei's piece was published.

 

Several participants in the group informed the BBC that Nvidia CEO Jensen Huang also talked about Coxon's remarks at a conference held last week by the investment firm Goldman Sachs. They said he disregarded them as false.

 

The idea that AI was "going to be the end of humanity" was deemed "complete nonsense" by Huang in the past.

 

Even though Nvidia produces the processors needed to power AI systems, so he might have a commercial stake in an AI boom, his remarks reveal a growing resistance in Silicon Valley against the existential concerns from both current and past employees.

Anthropic and OpenAI are in a duopoly, according to some critics, because Anthropic has been attempting to start a regulatory drive to stifle competition.

According to reports, Anthropic is getting ready for a possible record-breaking initial public offering (IPO) on the stock market, where investors will be able to purchase company shares.

It was anticipated that OpenAI, which was most recently valued at $852 billion (£630 billion), would follow suit. However, Altman of OpenAI stated on Friday that this will not occur this year due to safety concerns.