A vision-language-action model takes what a robot can see, a language instruction and the robot’s own sensor state, then outputs numerical actions. The short name is VLA.
A vision-language model can describe a scene or propose a plan. A VLA must connect perception and language to movement: a gripper pose, joint commands or another action representation that the robot can execute.
On this page
The VLA loop
A practical VLA system repeats a loop:
- A camera supplies images of the workspace.
- The operator or another system supplies an instruction.
- The robot reports proprioception, such as joint positions.
- The model predicts the next action or short action sequence.
- The robot executes the command and observes the result.
Google DeepMind’s Gemini Robotics On-Device 2 model card describes the inputs as text, images and numerical robot proprioception. Its output is numerical robot actions. That is a useful boundary: a VLA does not stop at a sentence about what the robot should do.
What vision, language and action contribute
Vision supplies the current physical scene. The model needs to distinguish the target object from nearby objects and track how the scene changes after contact.
Language supplies intent. “Put the blue cup on the tray” identifies a goal without prescribing every joint movement. The same instruction can require different motions when the cup starts in a different place.
Action connects the model to a robot. Action formats vary by system and embodiment. A tabletop arm, a mobile manipulator and a humanoid do not share identical controls, even if they accept similar instructions.
The model learns a policy over observations and actions. In ordinary terms, it estimates what the robot should do next given the instruction and its current view of the world.
Planning and control are related, but not identical
Long tasks often need a planner as well as a controller. A planning model can break “clear the table” into steps, decide which object to move first and revise the plan when the scene changes. The VLA policy handles the physical actions that carry out those steps.
Google DeepMind presents Gemini Robotics 2 as a VLA controller and Gemini Robotics ER 2 as an embodied reasoning model for multi-step planning. That is one implementation, not a universal architecture.
For a related look at robot foundation models, see our NVIDIA GR00T explainer. The humanoid robotics desk collects the broader hardware and control context.
Why on-device execution changes the design
Running a VLA on the robot can reduce dependence on a network connection and shorten the path from camera input to action. It can also keep some sensor data local. Those benefits come with tight limits on memory, compute and power.
DeepMind says its On-Device 2 model is based on on-device Gemma models and is intended for general-purpose manipulation across multiple robot types. The July 30, 2026 model card also says access is limited to selected trusted testers. It should not be read as a generally available robot controller.
A cloud-hosted controller can use more compute, but adds network latency and availability concerns. The better choice depends on the task’s timing, hardware and safety requirements.
What can go wrong
A VLA model can fail in ways that a text model cannot. A wrong answer may become a physical motion.
Common failure points include:
- an object that was rare or absent in training;
- a camera angle, lighting condition or clutter pattern outside the training distribution;
- an instruction that leaves the goal or forbidden actions unclear;
- a new robot body whose controls differ from the training setup;
- a long task in which a small early error changes every later step.
The On-Device 2 model card describes filtering and safety work, but publication of a model card is not proof that every deployment is safe. Developers still need task-specific evaluation, mechanical limits, emergency stops, human supervision where appropriate and a way to reject uncertain commands.
How to evaluate a VLA system
A useful evaluation separates several questions:
- Instruction following: Did the robot pursue the requested goal?
- Perception: Did it identify the relevant objects and their state?
- Execution: Did the motion succeed without collision or damage?
- Recovery: Can it notice a failed grasp or displaced object and try a sensible alternative?
- Generalization: Does performance hold when the scene, object or instruction differs from training?
- Latency: Does the full perception-to-action loop respond fast enough for the task?
A polished demonstration can answer only the first three for a narrow setup. Generalization and recovery require deliberately varied trials, including failures.
