数学框架有望提升临床AI的透明度

September 9, 2026 Mathematical framework could improve transparency of AI in clinical settings by King's College London edited by Gaby Clark, reviewed by Andrew Zinin Gaby Clark Scientific Editor Meet our editorial team Behind our editorial process Andrew Zinin Chief Editor Meet our editorial team Behind our editorial process Editors' notes This article has been reviewed according to Science X's editorial process and policies. Editors have highlighted the following attributes while ensuring the content's credibility: fact-checked trusted source proofread The GIST Add as preferred source Scheme of CBM (left) and CEM (right) architectures. Quantities with green and blue backgrounds are predicted scalars and vectors respectively. Credit: Journal of Machine Learning Research (2026). https://www.jmlr.org/papers/v27/25-1121.html Researchers at King's College London have developed a new mathematical framework to test whether AI systems that appear to explain their decisions are genuinely transparent. Published in the Journal of Machine Learning Research, the study addresses a growing issue as AI becomes more common in health care and other high-stakes settings: how to ensure people can understand, check and even challenge AI-generated decisions. AI systems can make highly accurate predictions, including identifying signs of disease or predicting patient outcomes. However, many operate as "black boxes," producing an answer without making their reasoning clear to the person using them. This can make it difficult for clinicians and other experts to identify whether an AI system has made a mistake or intervene in its decision-making. It also presents a challenge as emerging AI regulations increasingly emphasize transparency and meaningful human oversight. One approach to making AI more understandable is the use of concept-based AI models. These systems are designed to make decisions using clinically meaningful concepts, such as blood pressure, the presence of fever, tumor size or abnormalities visible in a medical image, rather than raw data alone. In principle, this should allow clinicians and other experts to see which concepts influenced a decision, giving them the opportunity to understand and review the decision with more information. However, concept-based AI models can still suffer from a problem known as "information leakage." This happens when the concepts used by a model contain additional, unintended information that is not visible to the person reviewing the decision. As a result, the AI system may appear interpretable while still relying on information that a human cannot see or assess. Effectively, it behaves like a "black box" model disguised as an interpretable model. King's researchers Dr. Chris Banerji and Dr. Enrico Parisini, working with colleagues at the Alan Turing Institute and with support from The Turing–Roche Strategic Partnership, developed a new framework to precisely define and measure information leakage in concept-based AI systems. The framework consists of two measures: concepts-task leakage (CTL), which captures hidden information linked to the final prediction, and interconcept leakage (ICL), which captures hidden information shared between concepts. Testing the framework across several datasets, the researchers found it could reliably detect leakage and predict how models would respond when their concepts were deliberately changed. The research provides practical guidance for designing concept-based models that minimize leakage and offer more meaningful transparency. Banerji, AI+ senior fellow (clinical-academic) and senior author of the paper, said, "Rather than putting the cart before the horse, as the field of AI moves quickly to apply models to real-world problems, we have taken a step back to consider what is needed to make these systems safe and reliable. Our work has focused on understanding limitations and addressing them so that we can move toward deploying these models in clinical practice in a safer way." Parisini, senior research fellow in machine learning and first author of the paper, said, "Concept-based AI has the potential to make AI systems more transparent, but our research shows that models can appear interpretable while still relying on information that is hidden from the person using them. By identifying and measuring this hidden information, we can take steps toward developing AI systems that are more transparent." The framework gives researchers and developers a clear way to assess whether AI systems are transparent enough to support meaningful human oversight, including in health care. The team is now working to apply these approaches to real clinical problems. More information Leakage and Interpretability in Concept-Based Models, Journal of Machine Learning Research (2026). www.jmlr.org/papers/v27/25-1121.html Key concepts AI alignmentTrustworthy machine learningGenerative AI ethics Provided by King's College London Who's behind this story? Gaby Clark MA in English, copy editor since 2021 with experience in higher education and health content. Dedicated to trustworthy science news. Full profile → Andrew Zinin Master's in physics with research experience. Long-time science news enthusiast. Plays key role in Science X's editorial success. Full profile → Citation: Mathematical framework could improve transparency of AI in clinical settings (2026, September 9) retrieved 9 September 2026 from https://techxplore.com/news/2026-09-mathematical-framework-transparency-ai-clinical.html This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only. Explainable AI: New framework increases transparency in decision-making systems Jun 13, 2025 Understanding AI decision-making: Research examines model transparency Feb 19, 2025 Shining a light into the 'black box' of AI Mar 22, 2023 End of black box AI? Scientists develop blueprint for transparent system that reveals how it learns and makes decisions Apr 30, 2026 Improving AI models' ability to explain their predictions Mar 9, 2026 Doctors need an AI model they can understand, says researcher Jul 29, 2026 New analog memory may make smart devices even smarter 1 hour ago AI controller translates VR, video and language commands into humanoid robot actions 3 hours ago Bridging the information gap: AI-driven quality control for 5G multicast broadcasting 2 hours ago Lithium-6 buildup may fingerprint lithium-ion battery degradation 2 hours ago Bringing classical LDPC code design theory to quantum computers 4 hours ago Built-in quantum sensor detects bending stress in microscopic devices 4 hours ago