Benchmarks rarely make headlines outside the engineering teams that follow them. The latest round of MLPerf Inference results, published in mid-September 2026, is worth a closer look for anyone building connected systems. For the first time, the benchmark includes a test of AI agents running entirely at the edge: not in a data centre, but on a single device with local power and memory limits.
The first result attached to that test comes from NVIDIA’s Jetson AGX Thor developer kit, a compact computer designed for robots and autonomous machines. It completed the new workload more than six times faster than a standard open-source baseline, while keeping accuracy close to 88% on the benchmark’s function-calling checks.
Those figures say something real about where edge computing is heading. They also come with caveats that matter to anyone planning to put AI agents on factory floors, in vehicles or inside machines.
What MLPerf Measured
MLPerf is a set of industry benchmarks run by MLCommons, a consortium of technology companies and researchers. Its inference benchmarks measure how quickly and accurately systems run trained AI models across different scenarios, from large data-centre servers to small edge devices.
A New Test For Agents At The Edge
Version 6.1 of MLPerf Inference added two new tests. One is an end-to-end retrieval-augmented generation benchmark, which measures systems that look up information before answering. The other is Edge Agentic Inference, aimed at AI agents running on local hardware.
Miro Hodak, a co-chair of the MLPerf Inference working group, explained the reasoning behind the edge test: “We added the Edge Agentic Inference test because complex inference systems with agentic properties are increasingly hosted on edge computing devices.”
An agent is different from a chatbot. Instead of answering a single prompt, it works through a task over many steps: deciding what to do, calling tools, reading the results and deciding again. That pattern is much harder on hardware, because each step adds to the context the model must process.
Twenty Tasks, 1,007 Turns
According to IoT Tech News’ report on the results, the edge agentic workload replays twenty recorded software-engineering tasks. Together they contain 1,007 turns, each requiring the system to act, manage tools and handle feedback within the limits of a local device. By the later turns, the context history reaches about 23,500 tokens.
The tasks are software-engineering trajectories rather than physical operations. That is a practical choice for a repeatable benchmark, but it matters when interpreting the result, as discussed below.
The Jetson Thor Result
NVIDIA ran the workload on its Jetson AGX Thor developer kit, configured with 128 GB of unified memory and running in its maximum-performance power mode. The model was Qwen3.6-27B, a 27-billion-parameter language model, served with NVIDIA’s TensorRT Edge-LLM software.
Speed And Response Times
The full 1,007-turn suite finished in 24 minutes and 36 seconds. The same workload on the benchmark’s reference setup, using the open-source llama.cpp software, took 2 hours and 37 minutes. That is a reduction in runtime of roughly 6.4 times.
The system produced output at 52.33 tokens per second. The median time to the first token of each response was 247.12 milliseconds, and each following token took a median of 14.68 milliseconds.
For context, a quarter of a second before an answer starts is quick enough for many interactive and operational uses. It is not fast enough for hard real-time control, such as stopping a machine, which is still handled by dedicated control systems rather than language models.
Accuracy
Speed is only useful if the answers are right. The benchmark checks whether the agent chose the correct function calls, the tool actions that drive each step. The Jetson Thor submission scored 87.94% on those checks.
That figure is respectable for a demanding multi-step task, but it also means roughly one in eight function-calling checks did not produce the expected result. For systems that act in the physical world, that gap is a reminder of why human oversight and hard safety limits remain essential.
How NVIDIA Got There
The more interesting part of the result for engineers is not the headline speed but the techniques behind it. Each addresses a specific bottleneck of running large models on a small device.
Smaller Numbers: NVFP4 And FP8
Large language models store billions of numbers. The more precisely each is stored, the more memory and bandwidth the model needs. NVIDIA compressed the model’s weights and activations into NVFP4, a 4-bit floating-point format supported by its Blackwell GPU architecture, and kept the key-value cache, the model’s record of earlier context, in 8-bit FP8.
Reducing precision this way cuts memory use and speeds up computation. The trade-off is potential accuracy loss, which is why the benchmark measures accuracy alongside speed.
Reusing Work: The Warm Cache
An agent’s context grows with every turn, and much of it repeats. Rather than recompute attention over the whole history each time, the system kept earlier results in memory and processed only the new part of each prompt.
According to the report, about 96% of prompt tokens were served from this warm cache: roughly 13.1 million of 13.6 million tokens did not need to be recomputed. On a device where memory bandwidth is a limiting factor, avoiding that repeated work makes a large difference.
Guessing Ahead: Multi-Token Prediction
The third technique is a form of speculative decoding. Instead of generating one token at a time, the system drafts several tokens ahead and then verifies them. NVIDIA used a tree-based approach that drafted eight steps ahead, which the report says lifted decoding throughput by about 40% compared with a simpler three-step method.
None of these techniques is unique to NVIDIA, and each is an active area of research across the industry. What the benchmark shows is how much they add up to when combined on a single edge device.
What The Result Does Not Show
A benchmark result is a measurement under specific conditions. Several limits are worth keeping in mind.
One Submitter
The report does not document any other vendor’s result in the new edge agentic category. The Jetson Thor figure is therefore a first data point rather than a win in a crowded field. Comparisons with other edge platforms will only be possible as more companies submit results in future rounds.
Software Tasks, Not Robots
The workload consists of software-engineering tasks. A robot or autonomous vehicle runs a different mix of work: perception from cameras and sensors, motion planning, safety checks and communication with other systems, all at the same time. An edge computer running an agent alongside those workloads will have less capacity available than one dedicated to the benchmark.
Developer Kit Conditions
The result was achieved on a developer kit with 128 GB of memory in its maximum-performance mode. Production devices may use smaller memory configurations, tighter power budgets or thermal limits inside sealed enclosures. Real-world performance in a specific product will depend on those choices.
Power And Cost Left Out
The published figures focus on speed and accuracy. The report does not give the power the system drew during the run, the cost of the hardware or the performance per watt. For edge deployments, those numbers often matter as much as raw speed. A robot running on batteries, a gateway in a sealed cabinet or a vehicle with a limited electrical budget will be judged on how much useful work it does for each watt and each dollar.
Buyers comparing edge platforms should ask for those figures directly, measured under workloads close to their own. A faster result that needs more power or a larger enclosure may not be the better choice for a specific product.
Frank Han, the other co-chair of the MLPerf Inference working group, framed the purpose of the results in practical terms: “With performance data from the Inference v6.1 benchmark, customers can better understand the cost-benefit tradeoffs and make informed decisions.” That is the right way to read them: as an input to decisions, not a guarantee of results.
Why Agentic Workloads Are Moving To The Edge
Why run an AI agent on a local device at all, when cloud services can run larger models? Several reasons apply to connected systems.
The first is latency. A machine that must decide what to do next cannot always wait for a round trip to a distant data centre, especially over a variable network.
The second is connectivity. Factories, mines, ships and vehicles do not always have reliable links. An agent that runs locally keeps working when the connection drops. We looked at the connectivity choices behind these sites in our explainer on industrial IoT and industrial connectivity.
The third is data. Sending camera feeds, sensor streams and operational data to the cloud costs bandwidth and can raise privacy and data-residency concerns. Local processing keeps sensitive data on site.
The fourth is scale. Research firm Gartner expects individuals, businesses and governments to create more than 10 billion autonomous agents by the end of the decade, IoT Business News reported. If even a fraction of those agents work alongside machines, a great deal of that computing will need to happen close to the equipment.
The Wider Edge AI Picture
Jetson Thor sits at the top end of edge hardware: a powerful computer built for robots and autonomous systems. Much of the edge AI activity in IoT is happening at the other end of the scale.
In the same month, Telit Cinterion detailed an edge AI SDK that runs small machine learning models directly on cellular modules, for tasks such as vibration analysis, sound detection and simple image classification. Those models are tiny compared with a 27-billion-parameter language model, but they run on devices that cost a fraction as much and can operate on very limited power.
The two ends of the market are complementary. Small models on modules and sensors handle narrow, well-defined tasks close to the data. Larger edge computers like Jetson Thor handle more complex reasoning for machines that need it. Cloud systems remain the place for training, fleet-wide analysis and the largest models.
What It Means For IoT Teams
For teams designing robots, autonomous equipment or intelligent gateways, the MLPerf result suggests that running a capable language-model agent on a single edge device is now practical, at least under favourable conditions.
Several questions follow for anyone evaluating that option. What model size does the task actually need? Smaller models are cheaper to run and may be accurate enough for narrow jobs. How will the agent be kept within safe limits when it gets something wrong? Accuracy close to 88% on a benchmark leaves room for errors that physical systems must be able to catch. How much of the device’s capacity will be left once perception, control and communication workloads are running? And how will models be updated, monitored and replaced across a fleet of devices in the field?
Benchmarks can help narrow the choice of hardware. They cannot answer those questions for a specific deployment. Pilots on real equipment, with real workloads and real network conditions, remain the most reliable guide.
A First Benchmark For A New Kind Of Edge Work
The addition of an edge agentic test to MLPerf is itself a signal. Benchmark bodies add categories when the industry starts asking for them, and the decision reflects a view that AI agents running on local hardware are becoming a real deployment pattern rather than a lab experiment.
NVIDIA’s Jetson Thor result sets the first public marker in that category. It shows that a compact edge computer, with careful software optimisation, can run a 27-billion-parameter agent through a long multi-step task in under half an hour. How that translates into robots, vehicles and industrial machines will become clearer as more vendors submit results and as real deployments report what they achieve outside the benchmark.
