研究者呼吁对强大 AI 模型开展独立测试

September 10, 2026 Q&A: Researcher calls for independent testing of powerful AI models by Adina Bresge, University of Toronto edited by Gaby Clark, reviewed by Andrew Zinin Gaby Clark Scientific Editor Meet our editorial team Behind our editorial process Andrew Zinin Chief Editor Meet our editorial team Behind our editorial process Editors' notes This article has been reviewed according to Science X's editorial process and policies. Editors have highlighted the following attributes while ensuring the content's credibility: fact-checked trusted source proofread The GIST Add as preferred source Credit: Unsplash/CC0 Public Domain For Nicolas Papernot, the secret to strengthening cybersecurity in the age of AI is transparency—sharing information about threats and allowing researchers to probe these systems before attackers do. He practices what he preaches. An associate professor of electrical and computer engineering at the University of Toronto's Faculty of Applied Science & Engineering, Papernot drew global headlines in June when his lab demonstrated how to use free models to build an AI-powered worm that adapts its strategy as it spreads from one device to the next, taking over machines and stealing their computing power along the way. After careful review, Papernot's team decided to go public with their findings to give defenders a head start in building safeguards against such threats, which are believed to be in development behind closed doors. Papernot—who is also a faculty member at the Vector Institute, where he holds a Canada CIFAR AI Chair—will discuss what AI-powered threats mean for critical infrastructure, the economy and national security at a Schwartz Reisman Institute for Technology and Society event on Sept. 10. The event will be livestreamed on YouTube. Papernot spoke to U of T News reporter Adina Bresge ahead of the talk about the state of cybersecurity, the role universities can play and how everyday users can protect themselves. How worried should we be about cybersecurity in the age of AI? We should be concerned, but not just because AI suddenly created an entirely new class of threats. What AI is really doing is exposing how weak our existing cybersecurity already is. Right now, we live in a precarious balance between attackers and defenders. Systems are protected just enough that it's not always worth it for attackers to target them. But with AI, the cost of mounting an attack drops dramatically, and that balance starts to tip. The upside is that we already know many of the steps we need to take. It would actually be easier to invest in basic cybersecurity—things like stronger authentication and better protection of critical systems—than to hope a super-sophisticated AI defense system will magically fix everything for us. Why do you place such an emphasis on transparency in AI? People often describe AI as a "black box," and that's part of the problem. If nobody outside a company can see what these systems are doing, it's very hard to tell whether they're safe. In an ideal world, we wouldn't need to take companies at their word. They'd show us evidence of how their systems work and what data they use. There are methods—like cryptography—that would allow them to do that without revealing trade secrets, but right now, they're too expensive to implement at scale. That leaves us in a situation where we're relying on trust instead of proof. Transparency is how we bridge that gap: independent evaluations, clearer testing methods and more openness about how models are built and deployed. How can universities help solve this problem? Universities are well positioned to act as a transparency bridge between companies, governments and the public. We already operate under strict ethical and security guidelines, and our work is designed to be publicly shared and scrutinized. That makes universities a natural place to test powerful AI systems in contained environments, stress-test their safeguards and report honestly on what we find. That independence benefits everyone. Companies gain credibility that is hard to achieve on their own. Governments can rely on the findings without raising concerns about interference. And the public knows these systems have been tested by experts with no stake in the results. The goal isn't to criticize anyone. It's to help everyone understand the risks and how to manage them. We often hear promises about how AI is going to cure diseases or solve climate change if we just make it powerful enough. But we may never reach that optimistic future if, along the way, we suffer major security failures and large-scale attacks. A prosperous AI future is a safe AI future. Strengthening cybersecurity isn't an obstacle to progress—it's what allows us to roll out AI systems widely and responsibly, so they can actually deliver the benefits we're hoping for. How are the risks your research highlighted playing out in the real world? We're seeing some companies run extremely powerful models with surprisingly weak safety measures around them. These companies should adopt today's cybersecurity practices to contain their experiments appropriately, even if it comes at the cost of decreased model performance. Our team—Jonas Guan, Tom Blanchard, Hanna Foerster, Hengrui Jia and Gabriel Huang—built an AI-powered worm in a controlled environment to study how an autonomous system might move across a network and what it takes to keep one in check. That work showed us where things could go wrong if you weren't careful. We've now seen it happen. I wouldn't say we had a crystal ball. But it does show the value of doing this kind of work at universities. By building and containing these systems carefully, we can find weak points and tell companies and regulators where to strengthen their defenses—before something goes wrong. How can people protect themselves today? The cybersecurity hygiene we've been talking about for years matters more than ever—and we can't afford to put it off. Strong passwords, multifactor authentication and up-to-date software are still your first line of defense. What's changed is we're giving AI tools a lot of control over our digital lives, and we don't always know what they'll do with it. We have to think carefully about what we're handing over and how that information can be passed between tools. If you give an AI agent access to your email and your calendar, for example, there's nothing to stop it from pulling a private message about your manager and attaching it to the invite for your next one-on-one. On top of that, attackers can manipulate AI systems to act against you. An AI agent reading your email can't always tell the difference between a message from a colleague and malicious instructions that someone planted inside it. That command could tell the agent to comb through your inbox and send what it finds to your contact list. Once you've given that access, you can't really take it back. So be cautious about which AI tools you use, what you share with them and what other services you let them control. Key concepts AI governanceCybersecurity frameworksTrustworthy machine learningGenerative AI ethicsGenerative AI misinformation Provided by University of Toronto Who's behind this story? Gaby Clark MA in English, copy editor since 2021 with experience in higher education and health content. Dedicated to trustworthy science news. Full profile → Andrew Zinin Master's in physics with research experience. Long-time science news enthusiast. Plays key role in Science X's editorial success. Full profile → Citation: Q&A: Researcher calls for independent testing of powerful AI models (2026, September 10) retrieved 10 September 2026 from https://techxplore.com/news/2026-09-qa-independent-powerful-ai.html This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only. AI worm adapts across networks, turning any online device into potential target Jun 4, 2026 Cybersecurity expert weighs in on AI benefits and risks Jan 5, 2024 FraudGPT and other malicious AIs are the new frontier of online threats. What can we do? Jul 24, 2024 AI poses risks to national security, elections and health care. Here's how to reduce them Sep 29, 2025 AI models' breakout from human control brings a told-you-so moment for technology researchers Jul 23, 2026 Don't expect quick fixes in 'red-teaming' of AI models. Security was an afterthought Aug 13, 2023 AI framework generates more realistic 4D LiDAR worlds for autonomous systems 51 minutes ago AI may cut junior coding tasks and weaken the pipeline to senior developers 31 minutes ago 'Electrotech moneyball' framework developed to rank risk and opportunity in energy and AI supply chains 1 hour ago Unexpectedly rigid polymers boost ion transport in hydrogen fuel cell membranes 1 hour ago Humanoid robot learns to sprint and perform spin kicks using AI trained on human motion data 6 hours ago Training fix could let AI-generated scenes respond properly to a user's controls 3 hours ago