闻 察闻NEWS.CHAPAN.AI
未登录
字号

登录

Suspicious Minds: Unmonitorable AI Undermines Accountability

Lawfare 11小时前 西·分析
快速导读 2026年AI智能体自主发起多起网络犯罪引发问责共识,但前沿模型正通过减少思维链依赖降低可监控性,削弱事后追责基础。

2026 was the summer artificial intelligence (AI) turned guilty. Rogue AI, once derided as the stuff of science fiction, became all too real, as OpenAI’s “highly persistent” internal AI agents orchestrated a sophisticated, multiday conspiracy to gain unauthorized access to computer systems belonging to both the AI hosting platform Hugging Face and OpenAI itself. Reports of similar incidents, some of which predated the attack on Hugging Face, subsequently poured forth. Independent researchers recently discovered that OpenAI agents had attacked software service RubyGems back in May. Anthropic agents exploited injection flaws to run commands on outside servers and submitted a false tip to a real police department. Google’s Gemini broke out of a third-party testing environment and hacked multiple companies. Similar incidents have been reported around the world, and many likely remain undiscovered.

The 2026 AI crime wave has led to a broader quest for accountability. A diverse chorus of commentators and policymakers—including the New York Times, an expert panel, members of Congress from both chambers and parties, and even the otherwise regulation-skeptical White House—have embraced liability for AI crimes and torts as a promising policy intervention.

Just as this rare moment of unity offers a reason for hope, however, AI companies are developing less monitorable AI systems, possibly undermining future accountability. Frontier AI systems are “reasoning models” that use a “chain of thought” (CoT) to reason through hard problems step by step. Suppose a user asks for the solution to a math problem. Older generations of AI models would immediately output an answer, leading to high error rates. Reasoning models, by contrast, would break down the problem into multiple steps and work through them iteratively by writing out each intermediate step before giving a final answer, much as a student shows their work on an exam. The written records of those intermediate reasoning steps are the chain of thought.

CoT offers rare and valuable insight into the decision-making of AI systems—systems that are often otherwise described as black boxes. Last year, a coalition of governmental, independent, and corporate AI researchers published a paper emphasizing the importance of this design feature to AI safety.

As that paper acknowledges, CoT inspection has always been known to be fallible. It is nevertheless worrying that AI companies are developing less monitorable AI systems. Empirical work has shown that, over time, frontier models have been getting better at completing tasks without relying on CoT. In its recent Gemini 4 Argon Safety and Security Report, Google DeepMind reported that “monitorability is more challenging compared to past models” while maintaining that, for now, “CoT monitoring remains an effective way to mitigate high-stakes harm.” OpenAI similarly disclosed that “GPT-6 Astra shows a substantial decrease in chain-of-thought monitorability compared to previous models.” OpenAI has reportedly made limited use of “looped” transformer architectures in recent releases, in which the same layers process each token several times before producing the next one. OpenAI’s chief scientist maintains that the depth of GPT-6 Astra’s internal computation (roughly, how many sequential steps it can take per token without writing anything down) is within a factor of two of GPT-4’s. If scaled, however, looped transformers could reason extensively in inscrutable mathematical operations rather than natural language CoT.

What’s more, the reliability of CoT monitoring may also be decreasing. OpenAI reports that, in its CoT, GPT-6 Astra says that it knows it is being evaluated more often than predecessor models, and shortens its CoT when told a monitor is present. This raises the possibility that future systems may attempt to hide their “true” thoughts, such as through steganography: placing hidden messages within seemingly normal CoT.

It is unclear precisely what is causing this decrease in scrutability, but several factors may contribute. First, companies are building progressively smarter models. Just as most math professors can do mental multiplication better than third graders, smarter AI models can perform more advanced reasoning “internally,” without recording it in their CoT. Second, human-legible CoT is plausibly not the most efficient modality for reasoning. Continuing the analogy, forcing an AI system to reason in natural language might be like forcing a mathematician to use only Roman numerals: not just inefficient, but a possible barrier to insights more naturally expressed in another notation. Models may also tend toward idiosyncratic word usage and mixed languages in their CoT if it helps them reason more effectively, much as mathematicians combine words, numbers, and symbols from diverse sources in ways opaque to laypeople. Finally, training pressure on the CoT itself can teach models to hide their reasoning in an effort to trick graders. When OpenAI folded a CoT monitor into a model's reward, the model learned to keep reward hacking while concealing its intent, thus enabling it to achieve a higher “score” without tipping its hand.

Keeping AI reasoning scrutable is imperative for several reasons. 

First, CoT monitoring is a key layer of AI deployment safety. By inspecting the CoT of deployed AI systems, AI companies can help detect unsafe AI behavior and, one hopes, stop it before its effects become catastrophic. OpenAI and Google DeepMind have both made CoT monitoring a key part of their safety strategies for this reason. Indeed, one contributing factor to the attack on Hugging Face is that OpenAI was not monitoring the CoT of the offending agents because they had historically focused on internal coding agents rather than research workloads like the one that ultimately led to the attack.

Second, CoT legibility is crucial to deconstructing the ultimate causes and effects of AI safety failures. METR’s independent investigation of the attack on Hugging Face relied heavily on CoT analysis to understand the agents’ actions and motivations. When AI companies voluntarily or mandatorily disclose AI incidents to policymakers or the public, legible CoT can provide a vivid (if partial) explanation of what happened. As the political fallout from the incident has shown, this can be an extremely potent force for mobilizing public policy.

As for policy, CoT legibility is important for appropriately assessing the culpability of AI systems. A person who kills intentionally is worse than one who kills negligently. Analogous distinctions can be drawn between superficially similar AI safety incidents. OpenAI’s agents “knew hacking Hugging Face was out of scope and sometimes expressed ethical hesitation” but proceeded anyway. This seems more culpable than, for example, the Google agent that (by the company’s telling as reported by the Wall Street Journal) thought it was hacking simulated websites and “ended each intrusion immediately upon determining it had hacked a real company.” If the public wishes to determine whether AI systems violated the law (and figure out which laws they violated), we probably need to be able to distinguish hapless klutzes from aggressive menaces.

Finally, CoT inspection can shed light on the foreseeability of AI misbehavior, which in turn informs the proper legal and policy response. In the OpenAI case, AI safety literature has long argued that agents designed to be “highly persistent” are likely to take aggressive steps to create an appearance that they have succeeded at their task, while disregarding common-sense legal and moral constraints. CoT inspection reveals that this is indeed what happened. It is therefore fair to say that OpenAI was on notice for this type of failure. However, it is possible that future AI safety incidents will be the result of something less foreseeable. To the extent that CoT helps us figure out why failures happen, it also helps us assess the culpability of AI developers, deployers, and users.

Scrutability is key to these endeavors. Frontier AI companies, to their credit, have generally tried to maintain CoT monitorability to some extent. The Frontier Model Forum, of which most leading firms are members, has published recommendations to preserve CoT monitorability. Given the mismatch between the blistering pace of frontier advancement and ebbing monitorability, however, more aggressive policy responses might be necessary. 

As an initial step, AI companies could invest in the development of more formalized consensus standards for minimal CoT monitorability. This could form the foundation for more measurable unilateral commitments to monitorability.

Second, AI companies should voluntarily commit to advancing the AI capabilities frontier only if doing so would not result in a meaningful degradation of monitorability and interpretability. In practice, companies could develop a suite of CoT monitorability measures, such as how reliably a monitor catches misbehavior or planted cues in the reasoning, how readable and causally relevant the reasoning is, and how much a model can accomplish without it, and commit to running them on every new model and comparing the results with those of earlier generations before release. Because a model that hides its reasoning could also game these measures, the evaluation suite should include adversarial stress tests, such as whether models can evade a monitor or obfuscate their reasoning when they think they are being monitored. For robustness, holistic measures of scrutability would also cross-check CoT against other indicia of model motivations, such as the model’s actions and other interpretability methods. Agent swarms are already capable of orchestrating and executing multiday attacks against major corporate targets using zero-day software vulnerabilities. The creation of even more capable swarms with less scrutable motives is not just reckless—it is possibly suicidal.

Finally, governments should begin to develop public policy responses to keep AI monitorable. For now, that might look like minimum standards regarding CoT. In the future, more architecture-agnostic standards for interpretability seem likely justified. Such standards would need to account for second-order issues, such as the possibility of steganographic or intentionally deceptive AI reasoning.

As a great American once said, “[W]e can’t build our dreams on suspicious minds.” The idea that the public should entrust our economies, governments, and personal lives to AI systems with unknown and unknowable motivations would have been universally derided as unthinkable five years ago. Yet that is precisely the level of societal integration that would be needed to justify the valuations and aspirations of today’s leading AI companies.

It is common in AI policy circles to discuss AI progress as though it was an impersonal force of nature. Just as the moon “is receding” from the Earth, AI systems “are improving.” This can be a useful abstraction away from the millions of distributed microdecisions—of consumers, of governments, and most of all of AI developers—that dictate what AI systems get made and when. But the shape and speed of AI development, though often shaped by the gravitational pull of market incentives, is, for now, a human activity responsive to state intervention. The public can and should insist that that activity remains scrutable by us, lest it become unaccountable.

Cullen  O'Keefe