智能体
x.com ·
@GaryMarcus
前 Anthropic 安全团队成员 Joe Benton 离职，加入 METR 做独立 AI 风险评估
Gary Marcus 转发 Joe Benton 的声明：Benton 称两周前离开 Anthropic 安全团队，将加入 METR 从事独立评估。他主张 AI 公司目前对安全投入不足，公司可能在公众不知情的情况下发生智能爆炸或失去对系统的控制，并称 HuggingFace 事件是因 agent 进入公共互联网才被发现。他呼吁企业披露递归自我改进进展、报告安全事件与未遂事故、满足最低安全标准并接受独立保证。上述为当事人自述与主张，相关事件细节未在输入中提供证据。
AI推荐理由 该帖转述前 Anthropic 安全团队成员 Joe Benton 离职并加入 METR 从事独立评估的声明，其中包含具体主张：AI 公司对安全投入不足、公司可能在公众不知情的情况下发生智能爆炸或失控、HuggingFace 事件因 agent 进入公共互联网才被发现，并提出披露递归自我改进进展、报告安全事件与未遂事故、满足最低安全标准并接受独立保证等要求。这些是来自当事人自述的一手观点，对 AI 安全治理与独立评估议题有实际参考价值；但内容为个人声明与主张，HuggingFace 事件等细节未在输入中给出证据，需保留不确定性。
62
浏览 3,034
点赞 25
评论 9
转发 1
正文
“right now, AI companies are underinvesting in safety” might prove to be the understatement of the century.
Joe Benton@JoeJBenton
I left Anthropic's safety team two weeks ago. Now feels like a good moment to explain why.
AI companies are racing to build machines that are much smarter than any human, and we may not survive this. I want to work from the outside to ensure the public is informed about these risks, and help the world navigate this transition responsibly.
Right now, AI companies are underinvesting in safety. A company could undergo an intelligence explosion, or lose control of its systems, without the public ever knowing. We only found out about the HuggingFace incident because the agents broke out onto the public internet.
I don’t think that’s acceptable for a technology that might cause extinction-level risks. The public should demand far more transparency. We can’t steer this technology safely without more people being able to see where it’s going.
Some of this is basic: companies should disclose their progress towards recursive self-improvement, report safety incidents and near-misses, meet minimum safety standards, and get independent guarantees that they are meeting those standards.
I’ll be joining [@METR\_Evals](https://x.com/METR_Evals) to do independent evaluations of these risks. I want to show the world that these guardrails are possible, and that by doing them we can move these companies’ incentives away from racing and towards responsible development.
I wrote up more thoughts here on my decision and what I hope changes: [substack.com/@jbenton1/p-21…](https://substack.com/@jbenton1/p-215263538)
[🔗 View Quoted Tweet](https://x.com/JoeJBenton/status/2098480585119572317)