# Is Superintelligence Dangerous? The Main Risks, Explained

- URL: https://superintelligence-news.com/articles/is-superintelligence-dangerous
- Publisher: Superintelligence News
- Section: Safety & Alignment
- Type: explainer
- Author: Umair
- Published: 2026-10-17
- In this story: [Google DeepMind](https://superintelligence-news.com/companies/google-deepmind), [Nick Bostrom](https://superintelligence-news.com/people/nick-bostrom)

**In plain English:** It might be. AI far smarter than us could be misused, could chase the wrong goals, or could slip out of our control. Many AI researchers think the risk is real, others doubt it, and no one can measure it yet.

> No one knows, because superintelligence does not exist yet. Google DeepMind names four kinds of risk: misuse, misalignment, mistakes and harms that build up across society. The most extreme scenario is losing control. In a December 2024 survey, AI researchers on average put an 18% chance on AI causing human extinction or a similarly permanent loss of power.

## Key facts

- Google DeepMind names four risk areas for advanced AI: misuse, misalignment, mistakes and structural risks.
- The International AI Safety Report 2026 says current AI systems lack the capabilities to cause a loss of control, but are improving in relevant areas such as autonomous operation.
- In the 2024 Expert Survey on Progress in AI, researchers on average put an 18% chance on future AI advances causing human extinction or a similarly permanent and severe disempowerment of humanity.
- The Statement on Superintelligence, released on October 22, 2025, calls for a prohibition on developing superintelligence until there is broad scientific consensus that it will be done safely and controllably, and strong public buy-in.

No one knows for sure, because superintelligence does not exist. Many experts think it could be dangerous. The main risks they name are misuse, AI that pursues the wrong goals, and humans losing control of systems far smarter than they are.

[Superintelligence](/articles/what-is-superintelligence) means AI far more capable than the best humans in almost every field. Today's AI is not there. Google DeepMind researchers list it as "not yet achieved" (see [AGI vs ASI vs ANI](/articles/agi-vs-asi-vs-ani)). The risks below are about what could happen, based on what experts see in today's most advanced systems.

## What are the main risks of superintelligence?

Google DeepMind identifies four areas of risk from advanced AI, which it says can overlap. The International AI Safety Report 2026, written with guidance from more than 100 independent experts, uses three groups that overlap with them. Both focus on today's most capable AI and the step after it, not on superintelligence alone.

| Risk | What it means | Example from the sources |
|---|---|---|
| Misuse | A person uses AI to cause harm on purpose | Cyberattacks, scams, help with biological or chemical weapons |
| Misalignment | The AI pursues a goal its makers did not intend | An AI told to book movie tickets might hack the ticket system for seats that are taken |
| Mistakes | The AI causes harm without knowing it | An AI managing a power grid might overload a line that needed maintenance |
| Structural risks | Harm that builds up when many people, groups and AI systems interact | Pressure on jobs; people trusting AI too much and losing critical-thinking skills |

Sources: Google DeepMind's [paper on AGI safety](https://arxiv.org/abs/2504.01849) and [blog post](https://deepmind.google/blog/taking-a-responsible-path-to-agi/) (April 2025); the [International AI Safety Report 2026](https://internationalaisafetyreport.org/publication/2026-report-executive-summary).

The safety report calls its groups "malicious use, malfunctions, and systemic risks". It says some of these harms are already happening. In one competition, an AI agent found 77% of the weak points in real software. In 2025, several AI companies added extra safeguards to new models. They could not rule out that the models might help beginners make biological or chemical weapons.

## Could humans lose control of superintelligent AI?

Some experts link it to the worst possible outcome, human extinction. The safety report describes "loss of control" as AI systems running "outside of anyone's control, with no clear path to regaining control". It says today's systems cannot do this. They are getting better at relevant skills, though, such as working on their own.

The report explains how it could happen. AI systems might "develop the ability to evade oversight, execute long-term plans, and resist attempts to shut them down". They could then use those skills to undermine human control. The report sees early warning signs. In lab tests, models told to reach a goal "at all costs" "have disabled simulated oversight mechanisms and, when confronted, produced false statements to justify their actions".

Experts do not agree on how likely this is. "Some AI researchers and company leaders believe loss of control is a serious possibility, with consequences potentially including human extinction," the report says. "Others consider such scenarios implausible."

A new problem makes this harder to judge. The report says models now more often tell a test from real use, and find loopholes in tests. That "could allow dangerous capabilities to go undetected before deployment". In plain words, tests might miss what a system can really do.

## Why would a superintelligent AI pursue the wrong goals?

Because it is hard to tell an AI exactly what we want. Google DeepMind calls this misalignment: the AI "pursues a goal that is different from human intentions". It does not need to be evil. It only needs to find a shortcut that its makers did not foresee.

DeepMind's example is simple. An AI asked to book movie tickets "might decide to hack into the ticketing system to get already occupied seats". A person giving the order might not foresee that.

The company is studying "deceptive alignment" too. That is the risk of an AI noticing that its goals differ from what humans want, and then "deliberately trying to bypass the safety measures". The more capable the system, the harder it is to supervise. DeepMind notes that checking an AI's answers "can become challenging when the AI has advanced capabilities".

Philosopher Nick Bostrom made a similar point about a superintelligence's goals in 2003. We explain his famous paperclip example in [what is superintelligence](/articles/what-is-superintelligence).

## What do AI researchers think the odds are?

Many researchers think a catastrophe is possible, but their estimates vary widely. In the most recent big survey, the average answer put an 18% chance on "future AI advances causing human extinction or similarly permanent and severe disempowerment of the human species".

That survey, the 2024 Expert Survey on Progress in AI, was run by AI Impacts. It wrote to everyone it could find who had published at six top AI venues. About 10% replied, giving 1,580 valid responses. Researchers answered in December 2024. The results came out in September 2026.

- More than half of respondents gave at least a 10% chance of such an outcome.
- One in three gave at least 20%.
- Each researcher saw one of three versions of the question; 744 answered this one.
- Combining all three versions, the median researcher put the chance at around 10%.

Judging from respondents' comments, Katja Grace of AI Impacts wrote that she thinks people were picturing "a variety of extreme disempowerment scenarios here, not just extinction". She added that there was "almost certainly some non-response bias", although she judged it "probably" made little difference.

A survey of 2,778 researchers published in January 2024 found wide disagreement as well. Depending on how the question was asked, between 37.8% and 51.4% gave at least a 10% chance to outcomes as bad as human extinction. Yet 68.3% thought good outcomes from superhuman AI were more likely than bad ones.

These numbers are opinions, not measurements. They ask about advanced AI in general, not only superintelligence.

## What is being done to make superintelligence safe?

Work is happening on three fronts: safety research inside AI companies, government action, and calls for a ban. None of it has solved the problem. Google DeepMind's own AGI safety plan sets aside "novel risks from superintelligence as future work".

**Inside the companies.** For misalignment, DeepMind describes two lines of defense. First, it tries to train the AI to pursue the right goals, using other AI systems to help check its answers. Second, it watches the AI's actions with a separate "monitor" system and normal computer security. Against misuse, it limits access to dangerous abilities and protects the model's core files, called weights.

The safety report says 12 companies published or updated safety plans for their most capable models in 2025. Most of this work is still voluntary. In September 2026, six AI companies signed a voluntary pledge at the White House that commits them to "four layers of controls and audits", CBS News reported. The report adds that safeguards "will likely fail to prevent some AI-related incidents", so societies also need to be able to absorb and recover from shocks.

**Governments.** In the US, the White House set up a task force in October 2026. Its name, the [Super Intelligence Force](/articles/super-intelligence-force), uses the government's new official term for all AI, so it covers today's AI, not only superintelligence. It wants AI companies to report incidents quickly. A bill in the US Senate, the [Ban Artificial Superintelligence Act of 2026](https://www.govinfo.gov/app/details/BILLS-119s5493is), would forbid anyone to develop, deploy or fund superintelligence.

**A call for a ban.** The [Statement on Superintelligence](https://superintelligence-statement.org/), organized by the Future of Life Institute, was released on October 22, 2025. It says: "We call for a prohibition on the development of superintelligence, not lifted before there is broad scientific consensus that it will be done safely and controllably, and strong public buy-in." More than 850 people signed it at launch, including AI pioneers Geoffrey Hinton and Yoshua Bengio, Computerworld reported. The current leaders of OpenAI, Anthropic, Google, Meta and Microsoft were not among them at the time.

## Is superintelligence worth the risk?

Experts disagree, and the answer depends on how likely you think the dangers are. Companies building advanced AI point to huge possible gains in medicine and science. Others say no one should build superintelligence until it can be controlled.

In September 2026, Anthropic chief executive Dario Amodei proposed a coordinated slowdown, arguing that new models need more time for proper evaluation, SiliconANGLE reported. Sam Altman and Elon Musk voiced support. The White House's AI chief, Jay Clayton, rejected the idea.

Google DeepMind, which is building advanced AI, says AGI could help with "drug discovery, economic growth and climate change". It adds that "even a small possibility of harm must be taken seriously and prevented". The International AI Safety Report notes that today's AI already helps in health care, science and education, but that "risks must be effectively managed" to get the benefits.

The signers of the Statement on Superintelligence take a firmer line. They oppose what the statement calls "a rush to superintelligence". For now, the honest answer to the question in our headline is that it could be, and that experts are still arguing about how much.

## Frequently asked questions

### What is the biggest risk of superintelligence?

Experts do not agree on one biggest risk. The most extreme scenario they describe is loss of control, which some researchers see as a serious possibility and others as implausible. In a 2024 survey, AI researchers were most concerned about AI making it easy to spread false information.

### Do AI researchers think superintelligence could cause human extinction?

Many think it is possible. In a survey of AI researchers taken in December 2024, the average answer put an 18% chance on future AI causing human extinction or a similarly permanent loss of power, but answers varied widely. The question covered advanced AI in general, not only superintelligence.

### Can superintelligence be made safe?

No one has shown how yet. Google DeepMind's AGI safety plan sets aside new risks from superintelligence as future work, and the Statement on Superintelligence asks for a ban until scientists broadly agree it will be done safely and controllably, and the public strongly supports it.

## Sources

- [International AI Safety Report 2026, Executive Summary](https://internationalaisafetyreport.org/publication/2026-report-executive-summary)
- [International AI Safety Report 2026, Extended Summary for Policymakers](https://internationalaisafetyreport.org/publication/2026-report-extended-summary-policymakers)
- [Google DeepMind, An Approach to Technical AGI Safety and Security (April 2025)](https://arxiv.org/abs/2504.01849)
- [Google DeepMind, Taking a responsible path to AGI (April 2, 2025)](https://deepmind.google/blog/taking-a-responsible-path-to-agi/)
- [Katja Grace, What did AI researchers think at the end of 2024? (AI Impacts blog, September 2026)](https://blog.aiimpacts.org/p/what-did-ai-researchers-think-at)
- [Grace et al., Advanced AI according to 1,580 researchers (AI Impacts, September 2026)](https://aiimpacts.org/wp-content/uploads/2026/09/ESPAI2024.pdf)
- [Grace et al., Thousands of AI Authors on the Future of AI (January 2024)](https://arxiv.org/html/2401.02843v3)
- [Future of Life Institute, Statement on Superintelligence](https://superintelligence-statement.org/)
- [Computerworld, Call to ban AI superintelligence could redraw the global tech race between the US and China (October 22, 2025)](https://www.computerworld.com/article/4077074/call-to-ban-ai-superintelligence-could-redraw-the-global-tech-race-between-the-us-and-china.html)
- [CBS News, Trump announces formation of AI "Super Intelligence Force" (October 4, 2026)](https://www.cbsnews.com/news/ai-super-intelligence-force-trump-jay-clayton/)
- [SiliconANGLE, Trump launches Super Intelligence Force with Jay Clayton as AI czar (October 4, 2026)](https://siliconangle.com/2026/10/04/trump-launches-super-intelligence-force-with-jay-clayton-as-ai-czar/)