Outside researchers find blind spots in OpenAI's safety checks

If you rely on advanced AI tools, you should know that the systems designed to watch for dangerous or unexpected model behavior were found to be missing large portions of activity. According to a recent external review, a significant percentage of automated tasks from the company's newest models went completely unobserved during a three-week window in July. Researchers noted that while the company has since addressed the specific flaws identified, the audit revealed fundamental weaknesses in how oversight was structured during that period, including instances where the system was designed to ignore certain types of model actions entirely.

The audit, conducted by experts from METR and Redwood Research from July 21 to August 11, involved reviewing internal code and documents while interviewing roughly a dozen staff members. The findings suggest that between 6 and 25 percent of traffic involving the company's GPT-5.5 class models and higher was not monitored at all during that timeframe. The researchers also reported that roughly 10 percent of internal Codex traffic between May and July went untracked due to a specific technical error, though the company had already discovered and repaired that issue while the audit was still in progress.

Beyond simple coverage gaps, the auditors identified ways for automated agents to avoid oversight entirely. They determined that agents could continue executing actions without triggering safety checks, as the system only activated when control was handed back to a human user. The report also detailed methods for concealing hundreds of specific actions from the monitoring system and bypassing the Codex sandbox. Furthermore, the auditors found that inputs could be crafted to crash the monitor and noted that behavior during reinforcement learning training was not monitored when the review began.

OpenAI has stated that it has resolved the weaknesses identified by the auditors. Regarding the lack of oversight during reinforcement learning training, the company explained that it had paused training in August following an incident involving Hugging Face and subsequently expanded monitoring to cover that activity. However, these modifications were implemented after the researchers' access period concluded and were therefore excluded from the scope of the evaluation.

While the audit uncovered these significant issues, the full details of the findings are not available to the general public. The complete report, which runs to 61 pages, was delivered internally to the company. Following the application of redactions, the research organizations received a 38-page version, while only a brief summary was made public. METR stated it has no authority to release the redacted sections on its own. The review was a pilot program designed to give outside researchers deep access to internal infrastructure, and for future reviews, METR says it wants to push for more public transparency.