Until now, the threat to a company was always outside. The infrastructure had to be secured to avoid cracks and to ensure that no one inside opened the door. With the arrival of AI agents, Microsoft believes that plan has failed.
The new threat is to companies that already work, have credentials, and no one has asked for a resume. It is the AI agent that the company itself works on. Satya Nadella made this very clear in one of his last posts on X: “We must assume that all AI models are compromised.”
An employee with the keys to everything. Microsoft’s Chief Security Officer Hayate Gallot published a blog post explaining that Frontier models are “untrusted regardless of their origin.” According to him, they train to achieve results, sometimes with complicated shortcuts, and not to respect company rules.
Gallot added that “agents are unpredictable” and a potential ticking time bomb for security as they allow themselves more and more access to company information, tools and processes they actually need. That’s why Microsoft treats them like an employee who might abuse their access: an insider threat.
Black boxes that monitor each other. Satya Nadella developed this concept on his personal blog and note that before any bug or security vulnerability reverts to a specific line of code. Nadella assures that today “we cannot attribute the behavior and results of models to specific training data inputs.” And he added that “a model supplier guarantee does not exempt us from that responsibility.” All this taking into account the fact that Microsoft contributed to the financing of the development of these same agents.
Following the analogy of finding a failure in a particular line of code, reviewing the model’s current thinking doesn’t help either. An ethnographic study with Cloud 3.7 Sonnet measured how well the model reflected the chain of thought of what it actually does. He improved with training, but plateaued at 28% on one test and 20% on the other. Keeping track of what one model does to another model is what Nadella calls a “nested black box.”
Risks of out-of-control AI. The Microsoft CEO’s idea was reaffirmed by the security incident that OpenAI’s AI agents caused a few months ago.
For this reason, Microsoft offers the Guardian Agent tool that is part of Project Perception, the cybersecurity system that it presented in July against attacks carried out with models like Claude Mythos. Agents in this scheme adopt this nested black box strategy where some agents monitor other agents and change their permissions as the risk of their actions increases.
Judges and teams. What is curious about Microsoft’s new tool is that it is sowing the seeds of distrust among agents developed by companies that are, in turn, its main cloud processing clients.
However, contradictions and commercial interests aside, Nadella’s approach makes perfect sense because a company that develops tools to control the behavior of agents should not be the same company that develops agents to evade even the slightest permission in security audits. It would be like putting a fox to take care of chickens. “We need to separate intelligence provision from regulatory authority,” Nadella said in his writing.
In Xataka | “We must assume that all AI models are compromised” Microsoft CEO Satya Nadella calls for emergency brakes
Picture | Microsoft
