On robot-use agents

The rua team ·

Introduction

The past months have seen an explosion in work on agentic robotics, also known as robot-use agents or RUA [1].

RUA systems leverage large foundation models to operate robotics embodiments in the real world. The fundamental premise of the "RUA paradigm" is that the same LLM that today answers your questions and searches the internet could one day control a robot, much like it can currently use a computer.

Compare this with what we may refer to as the "traditional paradigm" in robotics, which focuses on training robotics-specific policies like VLAs or WAMs. Instead of a one-model-fits-all approach, this paradigm combines a language or video model backbone with additional modules [2, 3] that are then fine-tuned on robotics-specific demonstrations using imitation learning.

In fact, these two paradigms have a whole set of differences. These in turn should shift the assumptions and priors that robotics researchers have when conducting research on RUAs.

So what is different?

1. Generalization: frontier models can rely on their large pre-training to generalize across environments and embodiments.

It was only about 3 years ago that researchers started to envision training generalized, embodiment-agnostic robotics models, that is, models that could control many different embodiments with minimal post-training [4–6]. The success of this approach was met with some excitement. Yet results of models like GPT-6-Astra and Fable 5.2 display them operating in an embodiment-agnostic fashion with no embodiment-specific post-training at all [7, 8]. It finally seems that one model could control any embodiment.

An important sub-part of this ability is visual-agnosticism. These models have been exposed to so much visual data that new backgrounds, lighting conditions, distractor objects, and camera placements seem not to affect them [9, 10]. And in fact, we purport that this generalism could go even further. Given their incredibly large priors as well as reasoning, adaptability [11], and in-context learning capabilities, we expect the sim2real gap to be a significantly smaller issue. In fact, we expect simulations to be much more effective in teaching these models transferable abilities like spatial reasoning and physical control [10, 12].

A side-effect of this generalism is also feedback flexibility. We are nowhere close to fully exploring the necessary feedback for embodied agentic systems. Most approaches currently limit the observations that a system receives to visual and proprioceptive inputs. Under the traditional paradigm, tactile, force, point cloud, audio, temperature, and other kinds of sensing would require the creation and training of a new model. For a RUA this might just be an interface change [13].

2. Long-Horizon Work and Planning: frontier models could rely on their post-training on long-horizon tasks to plan and perform long-horizon work in the real world.

Even before agents started directly controlling embodiments, it was thought that they could be leveraged as "System 2" [14] models: high-level planners directing lower-level policies [15–19]. In part, what enables this is "inference-time scaling" with the use of chain-of-thought [20]. But a significant factor is post-training. Through reinforcement learning on long-horizon digital tasks like coding and computer use (a close analogue to robot use), these models have learned to make and update plans, track progress, and manage context over hundreds of steps [21].

3. Multi-agent Coordination: frontier models are increasingly trained to coordinate work across many agents, and this could transfer to coordinating many robots.

Following the same logic of reusing general capabilities, frontier agents are increasingly trained to split up work, delegate it, and coordinate it across many agents. These abilities could transfer to multi-robot coordination [22, 23]. In the future we could see robot-use agents integrated into hierarchical coordinating structures, where an orchestrating agent oversees many agents, monitors their progress, and reallocates work in case of failures.

4. Post-training: the RUA paradigm can leverage the existing RL-environment-based post-training paradigm of frontier models.

One of the most exciting aspects of the RUA paradigm is that it could finally provide a reliable post-training paradigm for generalist robotics models. Despite being the birthplace of RL, robotics has struggled to wield RL for the post-training of generalist models. Instead, most post-training happens via imitation learning, which in turn requires large amounts of demonstration data and bottlenecks post-training. Thus robotics is in need of a post-training paradigm [24].

The answer for RUA could be to reuse the massively scaled, RL-environment-based post-training paradigm used by the frontier LLM labs. Much of the know-how can be built on existing research and infrastructure related to these RL environments, yet without a doubt the shape and content of RUA environments will require meticulous research.

Limitations

Naturally, RUA still has hurdles to overcome. Agentic robotics models are much slower than their visuomotor counterparts [25, 26]. Their latency currently limits their use in most reasonable applications of direct control. Yet there are already applications like open-loop execution [27, 28] or System 2 hierarchical control [15–19]. Moreover, we believe that further work on architectural changes, distillation, pruning and quantization, and inference optimization will reduce this gap, and that such work could be motivated by the goal of increasing RUA capabilities.

The other class of hurdles largely relates to the limitations in current models' capabilities, like precise manipulation [7, 26], whole-body control [29], and dynamic and reactive tasks.

The field of RUA

The improvement of these capabilities might be the content of the work of researchers in the years to come. Though even before that, there are many directions in which we expect research should and will be conducted.

1. Interfaces and harnesses

A foundation model by default has no way to interact with a physical embodiment, be it anything from robotic arms to humanoids. Therefore we have to provide the agent with an interface: a set of primitives that allow the agent to directly interact with an embodiment. It is the tooling layer that connects the agent with the embodiment.

Despite much recent work on analyzing and benchmarking different design decisions for RUA harnesses [10, 25, 29–36], there is still work that should be done. We've yet to see work on theoretically motivating and comparing the different kinds of interfaces. And we believe more could be done to meticulously analyze their implementations, and to think of improved ways to provide agents with the means to control embodiments. Such interfaces are likely the first step before further work like creating evals or perhaps even model training. Naturally, the interface is not the only part of a harness, and further research could be done to study the auxiliary systems around the interface, such as memory and context management [19, 29] and code-execution sandboxes [30, 31].

2. Capability analysis and benchmarking

When it comes to RUA evaluations, most work evaluates RUAs in environments originally designed for VLAs and WAMs [9, 25, 26, 34, 37], or otherwise inherits the design assumptions of VLA evaluation. This neglects the many differences we outlined above.

Many of these environments mainly test abilities that are relatively easy for RUAs, such as recognizing objects in an image or composing a few steps into a simple plan. At the other end of the spectrum, some test abilities that are still well beyond these models, such as locomotion and balancing, which require sending commands in real time [29].

Instead, we should understand the capability profile of these agents, and design benchmarks that probe the gaps we observe, like precise manipulation [7, 26], handling articulated objects, using tools, understanding affordances and, more broadly, what could be called physical common sense [38].

The benefit of such research would be two-fold. First, it would help these systems become deployable in the real world. Second, it could improve the spatial-reasoning and physical-control capabilities of frontier models more broadly, and transfer to other domains like computer use.

3. Quality and QA

As the field grows quickly, environment and evaluation quality will need particular attention. We have already observed common issues like tasks that reward incorrect completion and unwanted behavior, incomplete prompt descriptions, graders that fail reasonable completions, and reward and verifier leakage. These problems are well known from LLM eval creation.

Given the high intellect of these agents and their propensity for mischievous behavior, it's incredibly important that the evals and environments created today do not have these issues.

The big picture

These are only some of the directions we wanted to highlight. Many exciting open questions remain, from mechanistic interpretability work [39] on why and how these models can control robots at all, to adapting training and model optimization to make them more efficient and practical to deploy.

This is an incredibly exciting area. So far, all of the AI agents that we interact with in our daily lives have been constrained to the digital realm. Part of our motivation is to bring generally capable systems to the real world. We believe this paradigm is a great accelerant to doing so. It could provide the shortcut to seeing intelligent systems control devices from autonomous labs to coffee machines. To that end, this work is part of a broader movement of creating more intelligent AI by connecting it to the constant information stream that is the real world [40, 41].

Yet we also expect the shape of this paradigm to drastically evolve with new kinds of generalist architectures like world models [42], and to co-evolve with what we have termed here the "traditional paradigm" of robotics.

Over the next couple of weeks we plan to share some truly interesting results that we've obtained, and we are incredibly excited about the direction of this community.

References

  1. P. Isola, “Robot-use agents,” MIT, Sep. 7, 2026. web.mit.edu/phillipi/www/writing/robot-use-agents.html
  2. Physical Intelligence, “π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilities,” Apr. 2026. arXiv:2604.15483
  3. S. Ye et al., “World Action Models are Zero-shot Policies,” Feb. 2026. arXiv:2602.15922
  4. Open X-Embodiment Collaboration, “Open X-Embodiment: Robotic Learning Datasets and RT-X Models,” in Proc. ICRA, 2024. arXiv:2310.08864
  5. Octo Model Team, “Octo: An Open-Source Generalist Robot Policy,” in Proc. RSS, 2024. arXiv:2405.12213
  6. K. Black et al., “π0: A Vision-Language-Action Flow Model for General Robot Control,” in Proc. RSS, 2025. arXiv:2410.24164
  7. A. Menon, S. Machcha, S. Zou, T. K. Chan, and J. Chooi, “GPT-6 Astra on robotic manipulation,” Robocurve, Sep. 4, 2026. openai.robocurve.org/gpt-6-astra
  8. Anthropic, “Introducing Claude Fable 5.2 and Claude Mythos 5.2,” Nov. 9, 2026. anthropic.com/claude-fable-and-mythos-5-2
  9. Z. Chu et al., “OpenRUA: Robot-Use Agents Are Zero-Shot Visuomotor Policies,” Oct. 2026. arXiv:2610.02459
  10. Y. Chen et al., “Show-Harness: Just a VLM Agent Can Play Robots,” Sep. 2026. arXiv:2609.10522
  11. N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao, “Reflexion: Language Agents with Verbal Reinforcement Learning,” in Proc. NeurIPS, 2023. arXiv:2303.11366
  12. Y.-J. Wang et al., “Reconstruct, Practice, Go Real: Guided Self-Improvement for Embodied Agents,” Oct. 2026. arXiv:2610.02204
  13. K. Sun, K. Y. Ma, Y. Zhang, Y. Chen, and M. Z. Shou, “ForceHarness: Turning Force into Action and Physics for Frozen Frontier Models,” Nov. 2026. arXiv:2611.21099
  14. D. Kahneman, Thinking, Fast and Slow. New York, NY, USA: Farrar, Straus and Giroux, 2011.
  15. M. Ahn et al., “Do As I Can, Not As I Say: Grounding Language in Robotic Affordances,” in Proc. CoRL, 2022. arXiv:2204.01691
  16. L. X. Shi et al., “Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models,” in Proc. ICML, 2025. arXiv:2502.19417
  17. Figure AI, “Helix: A Vision-Language-Action Model for Generalist Humanoid Control,” Feb. 20, 2025. figure.ai/news/helix
  18. Gemini Robotics Team, “Gemini Robotics 1.5: Pushing the Frontier of Generalist Robots with Advanced Embodied Reasoning, Thinking, and Motion Transfer,” Oct. 2025. arXiv:2510.03342
  19. J. Hu et al., “What Matters in Orchestrating Robot Policies: A Systematic Study of Hierarchical VLA Agents,” Jun. 2026. arXiv:2606.10267
  20. J. Wei et al., “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models,” in Proc. NeurIPS, 2022. arXiv:2201.11903
  21. T. Kwa et al., “Measuring AI Ability to Complete Long Software Tasks,” in Proc. NeurIPS, 2025. arXiv:2503.14499
  22. Z. Mandi, S. Jain, and S. Song, “RoCo: Dialectic Multi-Robot Collaboration with Large Language Models,” in Proc. ICRA, 2024. arXiv:2307.04738
  23. K. Liu, Z. Tang, D. Wang, Z. Wang, X. Li, and B. Zhao, “COHERENT: Collaboration of Heterogeneous Multi-Robot System with Large Language Models,” in Proc. ICRA, 2025. arXiv:2409.15146
  24. P. Dong and C. Finn, “Towards Universal Post-Training for Robotics,” Sep. 2026. pd-perry.github.io/posts/post-training.html
  25. J. X. Yang, A. Khetan, and S. Zhang, “Astranomy: An Empirical Study of GPT-6 Astra as a native robot policy,” in CoRL Late-Breaking Results, Oct. 2026. research.nvidia.com/labs/srl/projects/astranomy
  26. M.-H. Guo et al., “Transferring the Intelligence of VLMs to Robotic Control,” Sep. 2026. arXiv:2609.22966
  27. J. Liang et al., “Code as Policies: Language Model Programs for Embodied Control,” in Proc. ICRA, 2023. arXiv:2209.07753
  28. R. Lu et al., “ASPIRE: Agentic /Skills Discovery for Robotics,” Jun. 2026. arXiv:2607.00272
  29. S. Berman, M. Ilie, J. Deng, and D. Freeman, “How Claude performs on robotics tasks,” Anthropic, Jul. 9, 2026. anthropic.com/research/claude-plays-robotics
  30. R. Si et al., “Fewer Tokens, Better Action: GPT-6 Astra Robot Agents with 14% Higher Success Rate but 65% Fewer Tokens,” Oct. 2026. arXiv:2610.01939
  31. L. Fu et al., “CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation,” Mar. 2026. arXiv:2603.22435
  32. H. Ma, C. Gao, R. Qiang, B. Dai, and N. Li, “RLE-Bench: A Qualifying Exam for Coding Agents as Robot Learning Engineers,” Sep. 2026. arXiv:2609.34210
  33. Z. Yang et al., “RobotWorld: Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodiments,” Oct. 2026. arXiv:2610.10409
  34. Z. Diao et al., “LIBERO-Agent: Evaluating General-Purpose Agents for Direct Embodied Manipulation,” Sep. 2026. arXiv:2609.39507
  35. Y. Lyu, X. Jiang, W. Li, S. Lu, and G. Zhang, “CodeActionBench: Evaluating Agentic Code-as-Policy for Embodied Manipulation,” Sep. 2026. arXiv:2609.33807
  36. Z. Wang et al., “JustRUA: A Controlled Study of GPT-6 Robot-Use Agents,” Nov. 2026. arXiv:2611.16009
  37. B. Y. Tsui, A. Y. Fang, and T. J. Hwu, “Demonstration-Free Robotic Control via LLM Agents,” in Proc. IROS, 2026. arXiv:2601.20334
  38. NVIDIA, “Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning,” Mar. 2025. arXiv:2503.15558
  39. W. Gurnee and M. Tegmark, “Language Models Represent Space and Time,” in Proc. ICLR, 2024. arXiv:2310.02207
  40. F.-F. Li, “From Words to Worlds: Spatial Intelligence is AI’s Next Frontier,” Substack, Nov. 10, 2025. drfeifei.substack.com/p/from-words-to-worlds-spatial-intelligence
  41. D. Silver and R. S. Sutton, “Welcome to the Era of Experience,” preprint of a chapter in Designing an Intelligence, MIT Press, Apr. 2025. storage.googleapis.com/deepmind-media/Era-of-Experience%20/The%20Era%20of%20Experience%20Paper.pdf
  42. M. Assran et al., “V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning,” Jun. 2025. arXiv:2506.09985
← Back home