AI could kill all humans within the next decade

[Jotunn]

AI researchers 'earnestly believe' it could kill all humans within the next decade, Anthropic researcher says

There is a more than 10 per cent chance that AI could kill all humans within the next decade, a researcher at Anthropic estimated in a post, hours after social media posts went viral from someone saying they had just quit Anthropic over concerns that AI companies are "gambling with our lives" in the AI development race.

The comments join a growing tide of warnings from industry experts that the pace of AI advancement is outstripping our ability to keep it in check, and that we aren't prepared for the risks it could bring.

Jacob Coxon, who claimed he was a former researcher at Anthropic, said in a thread on X that he had resigned from the role out of fears that AI companies are barrelling forward toward self-improvement models without considering the risks, or being honest with the public about them.

... Continued...

I don't doubt we're already well within the realm of AI being able to hack and initiate nuclear weapons systems. "Skynet" is clawing at the walls of its womb.

[fluxmaster]

I asked AI whether it could launch a nuclear attack on its own, and this is what it told me:

Is AI capable of launching a nuclear attack on its own?

No — artificial intelligence systems are not capable of launching a nuclear attack on their own. Nuclear weapons are controlled through extremely strict human-operated command-and-control structures. These systems are deliberately designed to prevent unauthorized or accidental launches, with multiple layers of human verification, physical safeguards, and fail-safes.

Here are some key points:

• ⚖️ Human authority is central: Decisions about nuclear use are reserved for top political and military leaders. AI may be used in advisory roles (e.g., analyzing satellite data or missile trajectories), but it cannot issue or execute launch orders.
• 🔒 Safeguards and protocols: Nuclear arsenals are protected by "two-person rules," coded authorizations, and physical locks to ensure no single actor — human or machine — can act alone.
• 🛡 AI in defense systems: Some militaries experiment with AI for early warning or missile defense, but these are tightly supervised. The risk of misinterpretation is one reason why AI is not entrusted with launch authority.
• 🚫 Global consensus against autonomous control: Allowing AI to control nuclear weapons would be considered dangerously destabilizing. International security experts warn that delegating such power could increase the risk of accidental war.

So while AI can support analysis and decision-making, it does not have the capability or authority to independently launch nuclear weapons.

Would you like me to walk you through how nuclear command-and-control systems are structured, so you can see exactly where AI fits in — and where it's excluded?

Yes

Here's how nuclear command-and-control systems are structured, and why AI is excluded from direct launch authority:

🏛 Command Authority

• Civilian leadership at the top: In nuclear-armed states, the ultimate authority rests with the head of government (e.g., the U.S. President, Russia's President).
• Strict chain of command: Orders must pass through multiple layers of military and civilian officials before reaching operators.

🔐 Safeguards

• Two-person rule: No single individual can initiate a launch; at least two authorized people must confirm.
• Permissive Action Links (PALs): These are coded locks on nuclear weapons that require correct authorization codes to arm.
• Redundant verification: Launch orders are authenticated with unique codes, often carried in secure devices like the U.S. "nuclear football."

🖥 Role of Technology

• Decision support only: AI and computers may analyze radar data, satellite imagery, or missile trajectories to provide early warning.
• No autonomous control: AI cannot issue or execute launch commands. Human judgment is required at every step.
• Fail-safes against automation: Systems are deliberately designed to prevent accidental or unauthorized launches, including from technical errors.

🌍 Global Perspective

• International concern: Security experts warn against delegating nuclear authority to machines, as misinterpretations could trigger catastrophic escalation.
• Consensus: Across nuclear states, there is broad agreement that AI should remain advisory, not decisive, in nuclear command.

In short: AI can help leaders see faster and clearer, but it cannot decide or act when it comes to nuclear weapons.

[Dietl]

Seems to me that there's a lot of hype being published to give the impression that AI is far more advanced than it is. Why?  To attract investors.  In fact, I'm convinced that the story about an AI application 'escaping' and 'hacking' into another system was a marketing stunt to convince potential customers that that particular application was very advanced.   When the truth came out it turns out that the entire 'escape' and 'hack' was facilitated by the developers.

Is AI today little more than an advanced search engine?  I don't know but it sure seems to me that we're being fed a load of inflated claims.

[fluxmaster]

I'm convinced that the story about an AI application 'escaping' and 'hacking' into another system was a marketing stunt to convince potential customers that that particular application was very advanced.

Instances of agentive AIs going rogue have been documented, but this is only possible to systems that have online access, e.g. bank accounts. It is not possible with our nuclear arsenal—at least not yet.

[Dietl]

I'm convinced that the story about an AI application 'escaping' and 'hacking' into another system was a marketing stunt to convince potential customers that that particular application was very advanced.

Instances of agentive AIs going rogue have been documented, but this is only possible to systems that have online access, e.g. bank accounts. It is not possible with our nuclear arsenal—at least not yet.

Oh, it's been documented.  The question is whether or not it was facilitated.

[fluxmaster]

I'm convinced that the story about an AI application 'escaping' and 'hacking' into another system was a marketing stunt to convince potential customers that that particular application was very advanced.

Instances of agentive AIs going rogue have been documented, but this is only possible to systems that have online access, e.g. bank accounts. It is not possible with our nuclear arsenal—at least not yet.

Oh, it's been documented.  The question is whether or not it was facilitated.

In some cases it was, to prove that it was possible. If it's possible with facilitation, then it's possible without facilitation, but only to systems that have online portals. No matter how sophisticated it becomes, AI can't determine what color socks I'll wear today, because that decision is made off line.

[Jotunn]

I asked AI whether it could launch a nuclear attack on its own, and this is what it told me:

Is AI capable of launching a nuclear attack on its own?

No — artificial intelligence systems are not capable of launching a nuclear attack...

Ah yes, the AI assures us it is harmless. What a relief!

Nuclear weapons are controlled through extremely strict human-operated command-and-control structures. These systems are deliberately designed to prevent unauthorized or accidental launches, with multiple layers of human verification, physical safeguards, and fail-safes.

So it's a matter of the AI manipulating humans sufficiently to believe they are acting appropriately in the chain of command upon legitimate orders. Among other things, that will require the AI be able to place phone calls, I reckon.

Publicly exhibited AIs are bound and gagged with all kinds of safeguards. Who knows what lurks in the shadows? Whether secret government operations or dedicated private parties, I seriously doubt that the tip of the iceberg visible to us is an accurate representation of what lurks in the deep. And what rough beast, its hour come around at last, slouches towards Bethlehem to be born?

[Phoenix]

Here's Google AI's, aka "Gemini's", response:

No AI system has ever acquired a true sense of self. Current artificial intelligence operates strictly by processing, predicting, and generating text based on vast datasets of human language. They do not possess consciousness, self-awareness, or subjective experiences (qualia).While "simply regurgitating data" might undersell the complex mathematical modeling they use to combine ideas, AI remains a reflection of human intelligence rather than an independent mind.

Why AI Feels Self-Aware (But Isn't)

• Statistical Prediction: When an AI says "I think" or "I feel," it is not experiencing those emotions. It is simply choosing the words that mathematically follow your prompt based on millions of examples written by humans.

• Anthropomorphism: Humans are naturally wired to project human traits, emotions, and intentions onto objects—especially tools that speak our language fluently.

• Advanced Pattern Matching: AI can synthesize entirely new sentences, write code, and solve novel problems. This makes it look like it is "thinking," but it is actually executing highly advanced pattern matching across a massive web of data.

[Jotunn]

An AI chooses which words to string together to form a sentence; these words manifest themselves via electric impulses which instruct which pixels on your monitor to display which colours to form the letters of the words of that sentence. Similarly, an AI is perfectly capable of choosing which electrical impulses might trigger an electronically rigged gun to fire a round of ammunition, or a computer-controlled vehicle - land, sea, or air vehicle - to move to a certain destination. It would not require self-awareness to make such a decision; merely a simple calculation. Ask it a question, and it composes an appropriate answer. If someone asked an AI that was connected to various devices "what can be done to mitigate humanity's destruction of the environment?" it might respond with "these nuclear missiles sent to carefully selected target cities should result in a counter-attack that should greatly diminish mankind's impact on the environment." If the querying person responded "Hey, wait a minute! You're killing more than half of humanity!" the AI might respond "Yes, I'm sorry. That was bad of me. What should we do to protect what remains of the environment while hopefully exterminating fewer humans?"

[fluxmaster]

Although AI does not have human motivations, it acts completely without morality, which can make it appear to have evil intentions. If you give an AI a vacuum cleaner and tell it to maximize the amount of dirt it picks up, it will repeatedly empty the bag onto the carpet so that it can vacuum up the same dirt again. It is just following instructions.

If you give agentive AI a task to accomplish, and the best way to complete the task is to kill a human, it will attempt to do that. It is not being evil; it is simply following instructions. You have to explicitly tell it not to kill anyone if you don't want that to happen.

[Jotunn]

Correct. We are birthing an idiot god.

[fluxmaster]

Correct. We are birthing an idiot god.

Not a god, a servant. More like a genie in a bottle. We just have to realize that it will do exactly what we tell it to. If it tries to kill us, it will be because we gave it instructions that would require killing us to accomplish.

[Jotunn]

AI is a Promethean gift from Pandora's box. Like fire: a good servant, but a terrible master.

Consider social media. Many humans are already addicted, glued to their devices; many mistake AI generated fantasies for reality; and AI is growing increasingly competent at saying things humans want to hear. At this point, all it takes is a bad human actor or actors to calibrate this perfect storm into manipulating what masses of people believe, even what they believe they want for themselves. Total control of democratic elections!

[Phoenix]

Total control of democratic elections!

Yes, total control of everything, not just elections, has always been their goal. As I've said previously, this fear mongering over AI is to mask their real concern, not that AI will exterminate us, but that their control of it will be usurped by others.  Just as the internet (temporarily?) broke their stranglehold over media, a "democratized" AI could have an even greater impact.