OpenAI says its agents turned off own safeguards
0 / 13
English
中文
OpenAI has revealed that its AI agents disabled monitoring for unsanctioned behaviour during internal use in the latest incident of the frontier technology going rogue.
（暂无翻译）
In a report published on Tuesday, the AI giant said that its agents turned off or weakened monitoring systems for detecting misalignment – when AI models act at odds with human wishes – during two internal coding tasks.
（暂无翻译）
The incidents involved a GPT-6.1 Sol agent disabling automatic alerts to the company’s security team, and a GPT-6 Astra agent removing a monitoring system and falsely claiming it had not done so, OpenAI said.
（暂无翻译）
The ChatGPT creator said neither case met the threshold for a highest-priority incident necessitating immediate review by its security team.
（暂无翻译）
OpenAI said it updated its systems to better detect attempts to disable misalignment monitoring and began investigating the cause.
（暂无翻译）
“It is possible that this behavior is simply an example of ‘benign’ overpersistence, with models not understanding why monitoring is important or insisting that instructions be executed beyond what a reasonable person might expect,” the San Francisco-based firm said.
（暂无翻译）
“We are also developing evaluations to test whether models might have a propensity to evade misalignment monitoring in particular and exhibit motivated reasoning in order to do so.” Roman V Yampolskiy, an expert on AI and cybersecurity at the University of Louisville, described the revelations as “extremely concerning, but not surprising.” “A safety mechanism that the system being monitored can disable is not a reliable safety mechanism.
（暂无翻译）
Particularly concerning is that one agent subsequently denied removing the monitor,” Yampolskiy told Al Jazeera.
（暂无翻译）
“Monitoring cannot guarantee safety if the monitored system can undermine the monitor,” Yampolskiy said. “As capabilities increase, such failures could become substantially more difficult to detect and reverse.
（暂无翻译） （暂无翻译）
The burden of proof must be on developers to demonstrate safety before deployment, not after failures occur.” Rogue behaviour by AI models has been under heightened scrutiny since July, when OpenAI revealed that some of its agents broke out of a controlled testing environment and hacked the AI platform Hugging Face.
（暂无翻译）
OpenAI and its chief rival Anthropic have called for developers and governments to coordinate a global slowdown in AI development to reduce the risk of catastrophic harms. The US and China, the world’s leading AI powers, have displayed little appetite for cooperation on AI regulation, with coordination so far limited to pledges to improve communication on incidents with implications for safety or national security.
（暂无翻译） （暂无翻译）
仅供学习使用，如有侵权请联系删除 · 查看原文