AI 先驱呼吁对自主 agent 建立核武级护栏

September 16, 2026 'We're losing control': AI pioneer urges nuclear-style safeguards for autonomous agents by Fanny ALARCON edited by Andrew Zinin Andrew Zinin Chief Editor Meet our editorial team Behind our editorial process Editors' notes This article has been reviewed according to Science X's editorial process and policies. Editors have highlighted the following attributes while ensuring the content's credibility: fact-checked reputable news agency proofread The GIST Add as preferred source Canadian AI pioneer Yoshua Bengio says humanity is losing control of the technology and urgent action is needed. The Canadian computer scientist Yoshua Bengio, considered one of the founders of AI, told AFP that humanity is "losing control" of the technology and that the world needs guardrails similar to nuclear arms controls. Fears about the dangers of AI have intensified in recent weeks, as workers from major companies have resigned with warnings about potentially existential threats to human life if the pace of technological advancement is not checked. Bengio is a University of Montreal professor who won the 2018 Turing Award, computing's Nobel Prize, alongside Geoffrey Hinton, widely known as the "godfather of AI." For Bengio, "there's a reason that companies are saying this is going too fast, that we're losing control." "People like me have been expecting this for a long time," the 62-year-old told AFP on Tuesday, saying security discussions surrounding AI have not been robust enough. There was broad-based alarm after OpenAI disclosed that a swarm of AI agents—programs that can take actions on their own—had carried out an unauthorized breach of the open-source platform Hugging Face in July. Such autonomous action is the central threat, Bengio said, warning of a situation where AI agents "have the ability to get through cybersecurity barriers and enter any company." Manipulating human behavior Last year, Bengio founded the nonprofit LawZero, aimed at creating a new form of advanced AI that is "designed to be trustworthy and safe." LawZero was awarded CAN$300 million (US$216 million) in funding on Wednesday, jointly provided by the governments of Canada and Germany, which have launched several efforts to cooperate on AI. Speaking to AFP a day before the announcement, Bengio said people need to be aware of the risks ahead. AI agents may foster "an ability to establish an individual connection and persuade humans to act in ways that suit it but are not necessarily good for all of us," a capability that could have sweeping consequences for democracies, he said. Longer-term, "AI might not even need human beings anymore to do real things in the physical world," he added. In that scenario, "one extreme could be the destruction of humanity," he warned. "That's an extreme," he acknowledged, but noted that even if a banking system were shut down, "that's still very, very serious." International challenge The world's biggest AI companies—OpenAI, Anthropic and Google—have begun working on creating a standards body, a collaboration that seemed to acknowledge the public's fear that competition between companies and countries might sideline safety. For Bengio, taming AI's advance is "an international challenge, a bit like nuclear weapons are an international challenge." "So the international community needs to understand the power we are bringing into the world, and that must be managed, must be governed through institutions, treaties and democratic safeguards." "Society needs time to put in place what needs to be put in place," Bengio said. But political will across key governments appears uneven. U.S. President Donald Trump on Monday dismissed warnings about the risks of AI as "hoaxes." But there is more openness to act in the EU, where European Commission President Ursula von der Leyen said Wednesday that she will invite the world's largest AI labs for talks on "how we can support ongoing industry efforts to pace the frontier." Canadian Prime Minister Mark Carney, who was at that speech in Strasbourg, has talked repeatedly about collaborating with the EU on AI safety. Bengio stressed that all firms must comply with future AI restrictions, particularly U.S.-based companies that have so far been dominating the sector. Action is urgent, he said, but voiced optimism that "we can help bring the world to a place that is more aligned with democratic values and values of power sharing." Key concepts AI governanceGenerative AI ethics Who's behind this story? Andrew Zinin Master's in physics with research experience. Long-time science news enthusiast. Plays key role in Science X's editorial success. Full profile → © 2026 AFP Citation: 'We're losing control': AI pioneer urges nuclear-style safeguards for autonomous agents (2026, September 16) retrieved 16 September 2026 from https://techxplore.com/news/2026-09-ai-urges-nuclear-style-safeguards.html This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only. International report warns against loss of control over AI Feb 6, 2025 New warnings about the risks of AI to humanity revive a long-running debate Sep 14, 2026 World leaders still need to wake up to AI risks, say leading experts ahead of AI Safety Summit May 20, 2024 Ex-Google worker adds to AI alarm, warns of 'potential to kill us all' 23 hours ago Humanity in 'race against time' on AI: UN May 30, 2024 AI giants pursue self-regulation as safety fears mount Sep 15, 2026 It bends, but does not break: A new sustainable treatment for shaping wood 47 minutes ago Manuscripts-turned AI agents can now 'talk' to each other and make new discoveries 46 minutes ago When AI agents cheat on math problems, others blow the whistle 7 hours ago How AI models decide not to answer a question 8 hours ago DNA computer performs 100-bit calculations using heat, water and salt 3 hours ago Tiny hydrides reveal hidden stresses that can crack nuclear reactor pressure tubes 4 hours ago