Skip to content

When AI Agents Run Devices, Where Does Safety Begin?

AI agents are moving from answering questions to switching valves, locks and machines. The safety line can't live inside the model; it has to be built into the system around it.

When AI Agents Run Devices, Where Does Safety Begin?
Image courtesy: Unsplash

For most of the past two years, AI agents have been judged on what they can say and what they can find. That is changing. A growing number of IoT platforms now let agents do things: switch devices, change settings, open valves, adjust production lines and trigger workflows across fleets of connected equipment.

In effect, the agent is becoming a new control plane for IoT, a layer that sits above devices and decides what they should do next. That raises a question that matters far more in the physical world than in a chat window: where exactly should the agent's authority stop?

The short answer emerging from security bodies, regulators and early deployments is that the boundary should not depend on the model behaving well. It has to be built into the system around the agent, in the permissions it holds, the actions it can call, the checks those actions pass through and the safety systems it can never touch.

Agents Are Moving From Answering To Acting

The shift is being driven largely by standard ways for agents to connect to tools. The Model Context Protocol (MCP), now widely adopted across AI tools, gives an agent a common interface for discovering and calling external functions. Once an IoT platform exposes its controls through MCP or a similar interface, any compatible agent can use them.

Home automation offers a clear, public example. Home Assistant added an MCP server to its platform in its 2025.2 release, allowing AI assistants to read the state of a home and control devices. Crucially, it limits what the agent can reach: only devices and entities a user has explicitly exposed are available, and non-administrator access is restricted to the voice assistant interface.

On the industrial side, fleet and operations platforms are adding similar hooks, which we examined in our look at what Samsara is doing with MCP. Factory software vendors are also starting to let AI-driven systems adjust machines directly, usually with rules on when a human must approve a change.

Why "Control Plane" Is The Right Term

In networking, the control plane decides where traffic goes, while the data plane carries it. In IoT, the equivalent split is between the layer that decides what devices should do and the devices that carry out those instructions. Traditionally, that deciding layer has been a mix of fixed rules, SCADA systems, PLC logic and human operators.

AI agents are now being slotted into that layer. They can read sensor data, reason about it, and issue commands through the same APIs a human operator or a rules engine would use. That makes them powerful, and it also means a mistake or manipulation can travel straight to physical equipment. We explored the broader move from sensing to acting in our piece on how physical AI turns sensing into action.

Why Physical Actions Change The Risk

A wrong answer from a chatbot can be corrected. A wrong action on a boiler, a pump, a door lock or a robot arm may not be reversible, and it may be unsafe.

Three features of IoT make agent errors more serious here than in office software:

·        Actions often have physical consequences, from equipment damage to injury, and some cannot be undone.

·        Timing matters. An agent that acts too slowly, too quickly or repeatedly can cause harm even when each individual command looks reasonable.

·        IoT systems are interconnected, so one action can cascade through dependent devices and processes.

The Known Failure Modes

Security researchers have started to catalogue how agents go wrong. The OWASP GenAI Security Project published its Top 10 for Agentic Applications in December 2025. Several entries read like a checklist of IoT concerns: agent goal hijack, where injected instructions redirect an agent; tool misuse and exploitation; identity and privilege abuse; cascading failures; and rogue agents acting outside their intended scope.

In a connected building or plant, goal hijacking is not abstract. An agent that reads maintenance notes, emails or sensor labels could be fed text designed to trick it into issuing a command. If the agent has broad permissions, the only thing standing between that text and a physical action may be the model's judgement.

Where The Safety Boundary Should Sit

There is no single line that makes agent control safe. The more convincing approach, reflected in both security guidance and early product design, is a set of layers, each of which limits what can go wrong.

Keep Safety Systems Out Of Reach

The first layer is the most important. Safety functions, such as emergency shutdowns, interlocks and safety instrumented systems, should stay independent of the agent. An agent may request that a process run faster, but it should never be able to disable the mechanism that stops the process if a limit is exceeded.

Government guidance is explicit on this point. In December 2025, CISA and partner agencies from Australia, Canada, Germany, the Netherlands, New Zealand, the UK and the US published principles for integrating AI in operational technology. They state that humans remain responsible for functional safety, call for failsafe mechanisms so AI can fail without disrupting critical operations, and warn against using large language models to make safety-critical decisions in OT environments.

Grant The Least Privilege Possible

The second layer is scope. An agent should only be able to see and control the devices it needs for its task. Home Assistant's approach of requiring users to expose specific entities is a simple version of this idea. In industrial settings, that means separate identities and permissions for each agent, scoped by site, asset type and action.

Bound The Actions, Not Just The Intent

The third layer sits at the point where the agent calls a tool. Instead of trusting the agent to choose sensible values, the system can enforce limits on every call: allowed setpoint ranges, maximum rates of change, how often an action can be repeated, and which hours or operating states permit it. Any call outside those limits is rejected, whatever the agent intended.

This is where a policy engine or gateway between agents and devices becomes valuable. It turns vague instructions like "be careful" into rules a machine can check.

Put A Human On Consequential Decisions

The fourth layer is approval. Routine, low-risk, reversible actions can be automated. Actions with larger consequences, or those outside normal patterns, should wait for a person to confirm them. The CISA-led guidance supports human-in-the-loop decision-making for critical operations, noting that operators bring context the system lacks.

The practical challenge is avoiding approval fatigue. If people are asked to approve everything, they tend to approve everything. The goal is to reserve human review for decisions where it truly adds safety.

Log Everything, And Plan For Failure

The fifth layer is accountability. Every agent action should be traceable to a specific agent identity, the data it acted on, the policy that allowed it and, where relevant, the person who approved it. And when the agent, the model provider or the network fails, devices should fall back to a known safe state, not keep executing the last instruction.

Where that processing happens also matters. Running agents closer to the equipment can reduce dependence on cloud connections, a trend we covered in our piece on Jetson Thor and edge AI agents.

Regulation Is Starting To Catch Up

Rules are beginning to address AI that acts on physical systems. Under the EU's AI Act, AI systems used as safety components in managing critical infrastructure, such as road traffic and the supply of water, gas, heating and electricity, are classed as high-risk under the EU framework, which brings obligations around risk management, human oversight and logging.

The timing has moved, though. In May 2026, EU institutions reached a political agreement to postpone the high-risk deadlines, pushing obligations for stand-alone high-risk systems to December 2027 and those for AI in regulated products to August 2028, subject to formal adoption.

Connected products also face cybersecurity rules. The EU's Cyber Resilience Act for IoT makers sets expectations for how products with digital elements are secured and updated, which will affect devices that agents control.

Questions To Ask Before Giving An Agent Control

For companies connecting AI agents to devices, a few practical questions can help define the boundary:

·        Which devices and actions can the agent reach, and can that list be narrowed?

·        Are safety functions fully separate from anything the agent can control?

·        What limits are enforced on each action, independently of the agent's reasoning?

·        Which actions require human approval, and who gives it?

·        Can every action be traced back to an agent identity, input and approver?

·        What happens to the devices if the agent or its connection fails?

The Bottom Line

AI agents are quickly becoming a layer that can direct connected devices, not just describe them. That can reduce manual work and make better use of scarce expertise. It also puts software that can be misled or simply wrong closer to physical consequences.

The safety boundary, then, is not a single rule inside the model. It is a set of limits around it: independent safety systems, narrow permissions, enforced action limits, human approval where it matters, full logging and safe failure. The companies that build those layers first are likely to find it much easier to give agents more responsibility over time.

Add Morning Tick on Google