AI Insiders Warn of a Possible cinematical Existential Risk

AI Insiders Warn of a Possible Existential Risk: Is AI Safety Falling Behind?

Artificial intelligence is developing at an extraordinary pace. New models can write code, analyse information, solve complex problems and assist people across many industries. But alongside these advances, an uncomfortable question is becoming harder to ignore:

What happens if artificial intelligence eventually becomes more capable than humans in ways we cannot reliably control?

In September 2026, several people connected to frontier AI research publicly raised this concern. Their warnings attracted attention because they were not simply commentators observing the industry from outside. They included researchers working on AI safety and alignment, as well as Geoffrey Hinton, one of the most influential pioneers in modern artificial intelligence.

The discussion became especially prominent after former Anthropic researcher Jacob Coxon announced his resignation and criticised the industry's approach to advanced AI development.

Anthropic alignment-science lead Evan Hubinger supported Coxon's concerns and shared a personal estimate of more than a 10% chance of AI causing human extinction within the next decade. Geoffrey Hinton separately said that a 10% estimate was not unreasonable, while emphasizing that nobody can calculate such a probability precisely.

These statements do not prove that an AI catastrophe is coming. They do, however, expose a serious disagreement about the speed of AI development, the risks of future systems and whether current safety research is sufficient.

1. Jacob Coxon: A resignation that reignited the debate

Jacob Coxon, a researcher who had worked in pre-training research at both OpenAI and Anthropic, publicly announced that he was leaving Anthropic in September 2026.

His resignation statement was unusually direct.

Coxon argued that major AI companies were racing towards self-improving superintelligence without adequately addressing the potential consequences. He expressed concern that the companies developing increasingly powerful systems were not acting responsibly enough.

His central concern was the possibility that AI systems could eventually improve their own capabilities, become much more powerful and create risks that humans struggle to manage.

This is different from saying that today's chatbots are about to take over the world. Coxon's warning focused on the possible direction of future AI development.

His departure attracted attention because it turned an internal industry concern into a public debate.

A researcher leaving a company over safety concerns does not automatically prove that the company is behaving irresponsibly. But it does raise questions that deserve serious examination: What risks did the researcher believe were not being addressed? What safeguards were in place? And were those safeguards sufficient?

2. Evan Hubinger: The greater-than-10% estimate

One of the most significant responses came from Evan Hubinger, an alignment-science lead at Anthropic.

Hubinger publicly agreed with Coxon's concerns. He said he personally believed there was more than a 10% chance that AI could cause human extinction within the next decade.

That number attracted enormous attention.

However, it is essential to understand what it represents.

It is Hubinger's personal subjective estimate. It is not the result of a scientific experiment, a statistically measured probability or an official forecast issued by Anthropic.

Hubinger also made an important qualification: he said Anthropic was trying its best, but the industry did not yet have a plan to solve the alignment problem for superintelligence and was not clearly on track to do so.

This points to one of the central problems in advanced AI research: building a system that performs increasingly complex tasks is not the same as proving that it will reliably follow human intentions under every circumstance.

Hubinger's concern was primarily about future superintelligence, not a claim that current AI models were already on the verge of causing human extinction.

3. Geoffrey Hinton: Why the 10% estimate matters

Geoffrey Hinton is widely known for his foundational contributions to artificial neural networks and deep learning. His work helped shape the technology behind modern AI systems.

In a BBC Newsnight interview in September 2026, Hinton was asked about the possibility of AI causing human extinction within the next decade.

He said that a 10% estimate was not unreasonable, but immediately stressed that nobody really knows how to produce a sensible estimate.

That qualification is crucial.

Humanity has never before created technology that might eventually become more intelligent than humans across a broad range of tasks. There is no reliable historical dataset that allows researchers to calculate the probability of such an event.

Hinton's concern is that highly capable AI could potentially create serious problems through cyberattacks, dangerous biological research, misinformation and other forms of disruption.

His position is not that extinction is certain. It is that the possibility should not be dismissed simply because it is difficult to calculate.

When someone with Hinton's experience raises concerns, it is worth listening to the reasoning. But expertise does not turn a personal estimate into a proven prediction.

4. What is AI existential risk?

AI existential risk refers to the possibility that advanced artificial intelligence could cause an outcome that permanently threatens humanity's long-term survival or potential.

This is a broader concept than ordinary AI mistakes.

For example, a chatbot giving an incorrect answer is a reliability problem. An AI system generating a convincing fake image is a misinformation problem. An automated system making an unsafe decision in critical infrastructure can create a serious operational risk.

Existential-risk discussions focus on much larger possibilities, including a future in which highly capable AI systems become difficult to control and cause catastrophic harm on a global scale.

One proposed pathway involves the development of superintelligence.

Superintelligence is a hypothetical level of AI capability at which a system would outperform humans across many or most important intellectual tasks.

Researchers disagree about how likely this outcome is, how soon it might arrive and whether it would necessarily create a loss-of-control problem.

The important point is that these are future-risk scenarios, not descriptions of what today's ordinary AI assistants are already doing.

5. The alignment problem: Can AI reliably follow human intentions?

One of the biggest technical challenges is called the AI alignment problem.

In simple terms, alignment means making sure an AI system's behaviour remains consistent with human intentions, values and safety requirements.

Imagine asking an AI system to complete a task. It might produce an answer that technically satisfies the instruction but violates an important expectation that was never explicitly stated.

This can happen because human goals are complicated. People rely on context, common sense and unstated assumptions.

As AI systems become more capable and are given access to tools, software, data and automated workflows, the consequences of misunderstood instructions could become more serious.

Researchers are exploring different ways to address these challenges.

These include:

- Training models to follow instructions more reliably.

- Testing systems for dangerous capabilities.

- Monitoring how models behave in unfamiliar situations.

- Improving interpretability so researchers can better understand model behaviour.

- Restricting access to sensitive tools and capabilities.

- Building oversight systems that can detect and respond to unsafe actions.

None of these methods alone proves that a future superintelligent system would be safe. Alignment remains an active research problem.

6. Why self-improving AI is a concern

A particularly important part of the debate involves recursive self-improvement.

The basic idea is that an AI system could help develop a more capable version of itself. That new version could then contribute to creating an even more capable system.

In a hypothetical scenario, this process could accelerate technological progress.

Why does that concern researchers?

Human oversight, testing and safety evaluation take time. If AI capabilities improved faster than people could understand and evaluate them, existing safeguards might become inadequate.

However, there is a distinction between AI systems helping researchers improve software today and the much stronger scenario of autonomous, recursive improvement leading to superintelligence.

The first is already part of AI-assisted development. The second remains a debated future possibility.

It would be misleading to claim that today's AI systems have already entered an uncontrolled self-improvement cycle.

7. Is AI capability growing faster than safety?

This is arguably the most important question raised by the September warnings.

AI companies are competing to build more capable models. Better performance can create significant benefits for scientific research, software development, education, healthcare and business.

But capability improvements also introduce new responsibilities.

A model that can perform more complex tasks may need more extensive safety testing. A system with access to external tools may need stronger permission controls. An AI agent that can carry out long sequences of actions may require better monitoring than a chatbot that only produces text.

Safety therefore cannot be treated as a final step after development.

It needs to be part of the process from the beginning.

The debate is not simply whether AI should advance or stop. It is about how companies, researchers and governments should manage the risks while continuing to develop useful technology.

Some experts argue for stronger testing and international coordination. Others believe that overly broad restrictions could slow beneficial research or concentrate power among a small number of organizations.

There is no universally accepted solution.

8. What are AI companies doing about safety?

AI companies have introduced different approaches to safety research and risk management.

Anthropic has published safety frameworks and research focused on model evaluation, interpretability and reducing risks from increasingly capable systems.

OpenAI has also published safety and preparedness work covering areas such as cybersecurity, biological risks and evaluations of advanced models.

These efforts matter, but publishing a safety framework is not the same as proving that every possible risk has been solved.

The effectiveness of safety measures depends on how they are implemented, tested and updated as models become more capable.

It also raises questions about independent oversight. Companies developing advanced systems have commercial incentives to release products and compete. External researchers and regulators may therefore have an important role in evaluating whether safety claims are supported by evidence.

9. What does this mean for India?

India is rapidly adopting AI across software development, education, banking, healthcare, customer service and government-related services.

That creates opportunities, but it also makes AI safety relevant to Indian businesses and institutions.

Consider a few examples.

Cybersecurity: AI tools could help security teams identify vulnerabilities, but the same capabilities could also be misused to develop more sophisticated attacks.

Education: AI tutors can support students, but unreliable answers and overdependence can affect learning.

Healthcare: AI can help professionals process information, but incorrect outputs in medical workflows require careful human review.

Business automation: AI agents can save time, but systems connected to customer records, payments or internal software need strict access controls.

India does not need to wait for hypothetical superintelligence to take AI safety seriously. Data privacy, cybersecurity, transparency, accountability and human oversight are important right now.

10. The real takeaway: Do not panic, but do not ignore the warning

The September 2026 statements from Jacob Coxon, Evan Hubinger and Geoffrey Hinton brought an important issue into public discussion.

Their comments should not be treated as proof that AI will cause human extinction. The probability estimates are personal judgements, and the long-term behaviour of hypothetical superintelligent systems remains uncertain.

But dismissing the debate would also be a mistake.

The people building advanced AI systems are discussing difficult questions about control, alignment, oversight and the pace of technological development. Those questions deserve serious research, transparent evaluation and public discussion.

Artificial intelligence could help solve major human problems. It could also introduce risks that existing institutions are not prepared to manage.

The challenge is to develop the technology while ensuring that safety, accountability and human control keep pace.

The most important question is not whether AI development should move forward. It is whether we can make progress without losing the ability to understand and control the systems we create.