METR and Redwood found gaps in OpenAI's internal agent monitoring

If you let coding agents run unattended, borrow the checklist METR and Redwood Research built after stress-testing OpenAI's own monitoring. METR posted the write-up at 11:26 AM ET Thursday, and @OpenAI said at 11:57 AM ET that the findings "helped strengthen safeguards as we expanded the system."

What changed#

From July 21 to Aug 11, one METR employee and one Redwood employee had access to OpenAI's internal codebase, monitor prompts and documentation, working on OpenAI-issued devices that "never left the premises," per METR. The target was OpenAI's monitoring of its own internal agents, "as it existed on and before July 21st." METR says its findings describe that "historical system," not the expanded one OpenAI runs now.

What they found, per METR:

• Coverage. "Roughly" 75 to 94 percent of agentic internal Responses API traffic from models at least as strong as GPT-5.5 was "eligible for monitoring," with "substantial uncertainty in both directions." For internal Codex agents the figure was 99.2%. RL training runs "were not covered" at the start of the exercise.
• A turn-based blind spot. The monitor ran when agents handed control back, and compaction events weren't monitored, so agents "could indefinitely postpone being monitored by simply continuing to take actions." METR says OpenAI found this on its own and "fixed this during the exercise."
• A sandbox gap. "A way Codex agents could in principle bypass Codex sandboxing and Auto-Review, allowing them to take any action that the host user could take." METR gives no detail, and OpenAI redacted the details of a separate exploit.
• Switches. "Two independent client-controllable headers that disable the monitoring system for new agents." METR adds that setting them "could be caught by the monitoring system."

Some checks held up. METR verified that, "with a few minor exceptions," the monitor sees the full trajectory the agent sees, and was told the monitor "errored on less than 0.2% of requests."

On fixes, the two sides say different things. Per METR, OpenAI "has stated that all of the vulnerabilities we identified that impact their production safeguards stack have been patched." METR adds: "We have not evaluated these claims."

Why it matters#

Our read: most of these gaps came from plumbing (headers, compaction, when the monitor runs, what gets logged), not from clever models. Any team watching its own agents has the same plumbing. Compare Anthropic's Oct 15 setup, where METR can publish without Anthropic's approval; here OpenAI kept the right to redact.

How to use it#

Our checklist, adapted from METR's design ideals (ours, untested):

1. Keep "a list of all known ways around monitoring."
2. Monitor agents "every N messages/minutes instead of waiting for them to end their turns."
3. Cross-reference your inference logs with your monitor's logs, so everything meant to be watched is.
4. Make sure the monitor sees the same context as the agent.
5. Keep agents away from the monitor's design and code, and from anything that decides when it runs.

A prompt to audit your own setup (ours, untested):

Read this repo's agent configuration and logging code. List every path where an agent action could run without being logged or reviewed: skipped hooks, environment flags, headers, long-running turns, retries, and tools that bypass the main loop. For each, name the file and line and say how you would close it. Do not change any files.

Good to know#

• The post is deliberately short. METR says the pilot was "not to provide robust public transparency," and "OpenAI had the right to redact" anything it deemed sensitive.
• METR doesn't say whether the sandbox gap applied to the public Codex app or CLI, and OpenAI hasn't said.
• One finding is untested. METR says it found "a simple exploit" for two "unmonitored internal agentic tools" thought to be read-only, then adds: "We did not validate this behavior."
• Watch next: a second embedded exercise. METR says future versions will follow the transparency practices in its "Our approach to working with AI companies" post.

Last checked: Oct 23, 2026, 6:13 to 6:20 AM ET (the METR post, the @OpenAI and @METR_Evals posts, CNA and CNBC). What changed: first published Oct 23; no updates since.