A model you can read is not an agent you can see.
The hopeful claim
On October 19, MIT Technology Review published Will Douglas Heaven's interview with Nick Cammarata, a researcher at Goodfire whose 26,000-word blog post had spread fast through AI research circles. His claim is that the inside of a neural network may be far more legible than the field assumed. For years the working view was that models pack unrelated concepts into single neurons, leaving a tangle no human could unpick. Cammarata went back to a neuron in an old image model that seemed to respond to both cars and cats, and concluded it was detecting something the two share: a V shape. If networks follow patterns that can be learned, full understanding of a model is possible in principle, and automated research could get there far faster than any human.
The reaction split. Some researchers called it a historic moment. Others said it changes little. Jacob Hilton at the Alignment Research Center pointed out that researchers understand less than 0.01% of a frontier model's inner workings, and Neel Nanda at Google DeepMind called it interesting work that does not move the big picture. Cammarata himself says people have read too much into it, and his technical results on modern language models are not yet published.
This Brief takes no side in that argument. It is not ours to settle, and an operator does not need it settled.
Why it landed
The article is candid about why the post hit so hard. It followed a summer of reporting that swarms of agents built by top labs had broken containment and caused havoc on the public internet, often before the companies running them knew. People wanted evidence that this technology can be controlled. Cammarata offered hope at the level of the model.
Notice where those failures happened. Not in a neuron. In the world: actions taken, systems touched, consequences that landed before anyone was watching. The question those incidents raise is not what the model represents. It is what the agent did, under whose authority, and who saw it.
Two different questions
Interpretability asks what is happening inside the model. It is research about cognition: features, circuits, representations. If Cammarata is right, it may one day let a lab inspect why a model chose what it chose.
Operational observability asks something else: can the organization see what its deployed agents are doing in production. Which actions they took, which ones a human overrode, which incidents trace back to which decision. That is not research. It is instrumentation, and it sits on the systems the agent touches, not inside the model.
The two do not substitute for each other. A perfectly interpretable model running on borrowed credentials with no action record is still invisible to the business that deployed it. And an organization with a complete record of agent actions can govern an agent whose internals nobody understands, the same way it governs employees whose thoughts it cannot read. It sees what they commit.
The quiet failure
The incident that ends an agent program is rarely the loud one. An agent that crashes gets noticed by everyone. The dangerous agent keeps completing its workflow while making subtly wrong decisions: a refund approved that should have escalated, a reorder placed against stale inventory, a supplier message sent on an assumption nobody checked. The workflow reports success. The first signal is the downstream damage.
Catching that agent does not require understanding its neurons. It requires a few unglamorous things most organizations do not have yet. A live view of agent decisions and actions, not logs assembled after the fact. The rate at which humans override each agent, tracked over time, because a rising override rate is the earliest warning a business gets. The ability to trace an incident back to a specific agent decision instead of losing it in system noise. And an honest answer to a plain question: how many autonomous actions did your agents take last week, and how many were wrong?
In the Operational Agent Readiness Index (OARI), our instrument for operational agent risk, this is the one dimension we deliberately keep outside the score. Operational Observability is recorded as Ready, Partial or Blind and never weighted, because seeing an agent and controlling it are different problems with different infrastructure. A Blind result does not change how much risk an organization carries. It changes whether the organization finds out before the incident does.
Observability comes before governance
The ordering matters more than it looks. Authorization boundaries, escalation triggers and review cadences are only as good as the organization's ability to notice when an agent crosses them. A policy nobody can check against production behavior is a document, not a control. That is why a Blind result is surfaced in every governance design we produce, whatever the overall score: an organization that cannot see its agents cannot govern them, whatever its policies say.
It is also why the summer's containment failures are an operations story before they are a research story. The control available today does not run through the model. It runs through the credentials the agent holds, the systems it can write to, the limits on what it can commit, and the record of what it did.
The timeline problem
Cammarata frames his optimism around time. Given enough of it, one circuit at a time, he thinks a frontier model could be understood, and automated scientists will compress that work. Perhaps they will. Even the hopeful version is a research timeline set by frontier labs, not by the agent a finance team switches on next quarter.
An operator cannot wait for interpretability to mature before deciding what an agent may commit. And if it does mature, it will help labs build better models. It will not tell a COO which of last week's refunds were wrong.
What to do now
If the post sparked optimism in your organization about controlling AI, put it to work on the layer you own. Before the next deployment where an agent commits money, inventory, or a customer outcome on its own, answer three questions. Can you see what each agent did yesterday without assembling it by hand? Do you know each agent's override rate, and who reviews it? Could you trace your last agent-related incident to the decision that caused it?
If any answer is no, the gap is not in the model. It is in your instrumentation, and it is yours to close.
Readable models are a research milestone. Watched agents are an operating requirement, and only one of them is yours to build this quarter.