Physical AI and Vision-Language-Action Models: Building the Next Generation of Intelligent Robots

TL;DR — Quick Insights

• Physical AI is the convergence of large foundation models with physical robotic systems — giving robots the ability to see, reason in natural language, and act in the physical world from a single unified model.
• Vision-Language-Action (VLA) models are the defining technology of robotics in 2026 — the AI robotics market is growing from $16.1 billion in 2024 to a projected $124.77 billion by 2030 on the back of these systems.
• NVIDIA GR00T N1.5, Gemini Robotics On-Device, and OpenVLA-7B are the three most important open or accessible VLA models for practitioners in 2026. You will work with all three in this course.
• The key engineering insight: VLA inference must run at 30–100 Hz on hardware that fits inside a robot. Action chunking and System 1/System 2 architectures are the two production techniques that make this feasible.
• This course builds directly on UDHY’s advanced courses in deep learning and reinforcement learning to cover the complete state of the art in physical AI as of 2026.

Introduction

I have been in this industry for more than a decade — co-founding Moovita, Singapore’s first autonomous vehicle company, and spending years as a Principal Research Scientist at A*STAR developing perception and control systems for real-world autonomous systems. When I started in this field, a robot that could pick up an arbitrary object from a table was a PhD thesis. Today, systems like NVIDIA GR00T and Google DeepMind’s Gemini Robotics can understand natural language instructions, reason about novel objects they have never seen before, and manipulate them with human-level dexterity — running on hardware that fits inside a humanoid robot.

This is Physical AI: the era when foundation models stop living in data centres and start living in machines that touch the physical world.

This course is the most technically demanding in UDHY’s curriculum. It assumes you have completed the Deep Learning for Robotics and Expert Robotics courses and are comfortable with PyTorch, ROS 2, and production robotics systems. What you will gain here is the ability to understand, fine-tune, and deploy the frontier systems that are reshaping what robots can do — right now, in 2026.

---

Module 1: What Is Physical AI? — The Paradigm Shift

1.1 From Task-Specific to Generalised Robot Intelligence

For the first 70 years of robotics, intelligence was local and task-specific. A welding robot was programmed for one task. A surgical robot was programmed for another. Even the most sophisticated Boston Dynamics Spot required explicit programming for each new skill. The intelligence was in the code, not in the machine.

Language models changed the trajectory. GPT-4, then Claude, then Gemini demonstrated that a single large model trained on internet-scale data could generalise across tasks that no engineer had explicitly programmed. The logical next question — first asked seriously by researchers at Google, Stanford, and DeepMind around 2022 — was: what if you trained the same kind of model not just on text, but on robot sensor data and motor commands?

Physical AI is the answer to that question. It describes AI systems where foundation models — architectures with billions of parameters trained on massive multimodal datasets — are embedded directly in physical agents that perceive and act in the real world. The key properties that distinguish Physical AI from classical robotics:

• Generalisation: The robot can handle objects, environments, and instructions it was never explicitly trained on
• Language grounding: The robot responds to natural language instructions without requiring a re-programming cycle
• Few-shot adaptation: A new task requires tens or hundreds of demonstrations, not thousands of custom-labelled training examples
• Embodied reasoning: The model reasons about physical constraints — weight, friction, breakability — not just visual categories

As we explored in UDHY’s analysis of why self-driving cars still fail, the core problem in robotics has never been building systems that work in controlled conditions. It is building systems that generalise. Physical AI, for the first time, provides a credible path to that generalisation.

1.2 The Physical AI Ecosystem in 2026

The physical AI landscape in 2026 is moving faster than any technology sector I have observed in 30 years. Key milestones that define where we are:

• NVIDIA GR00T N1 (March 2025): The world’s first open, fully customisable foundation model for generalist humanoid reasoning and skills, released at GTC 2025. N1.5 followed at COMPUTEX May 2025 with improved visual grounding via Eagle 2.5.
• Gemini Robotics On-Device (June 2025): Google DeepMind’s first VLA model made available for fine-tuning. Engineered for bi-arm robots with low-latency on-device inference — no cloud connection required.
• OpenVLA-7B: Stanford’s open-source VLA that outperforms larger proprietary models on several benchmarks, democratising access to VLA technology for research teams worldwide.
• AGIBOT WORLD 2026: An open-source, production-grade real-world dataset spanning commercial spaces, homes, and everyday scenarios — declared “Deployment Year One” in April 2026 with 10,000 robots deployed.
• Physical Intelligence (π): Raised $400 million at a $2.4 billion valuation on the strength of their π0 model for dexterous manipulation.

The AI robotics sector is growing from $16.10 billion (2024) to a projected $124.77 billion by 2030 — a 7.7× expansion in six years. This is not a trend. It is a structural transition.

Think about it: Before proceeding, consider the difference between a robot programmed to “grasp the red cup” and a robot that understands “pour me some water.” What additional capabilities does the second instruction require, and how do current robotics systems fail to handle it?

---

Module 2: Vision-Language-Action Models — Architecture Deep Dive

2.1 What Is a VLA Model?

A Vision-Language-Action (VLA) model is an AI system that integrates three previously separate capabilities into a single unified architecture:

• Vision: Understanding the world from camera images and depth sensors — objects, scenes, spatial relationships
• Language: Understanding natural language instructions — “pick up the blue block and put it in the box on the left”
• Action: Generating robot motor commands — joint angles, end-effector positions, gripper forces — that physically execute the understood instruction

The key breakthrough is the unified architecture. Classical robot systems used separate modules: a vision model, a language parser, a motion planner, and a controller. Each module was trained separately and communicated through handcrafted interfaces. Failures at any interface propagated through the system. VLA models collapse this pipeline into a single end-to-end model that is trained jointly on vision, language, and action data.

2.2 The VLA Architecture

Most production VLA models in 2026 follow a similar high-level architecture:

Vision Encoder: A pre-trained vision transformer (ViT) or similar model encodes camera frames into visual tokens. This component is typically initialised from a large pre-trained model (SigLIP, DINOv2, or similar) and frozen during early VLA training to preserve visual representations.

Language Model Backbone: A pre-trained large language model (Llama 3, Qwen, or similar) processes both the visual tokens and the language instruction tokens. The language model provides the reasoning and generalisation capability — this is where “understanding” lives.

Action Head: A specialised output head maps the language model’s hidden states to robot action space. This is where the architecture diverges most across different VLA implementations:

• Tokenised actions (RT-2 style): Robot actions are discretised and predicted as tokens, like text generation
• Diffusion action heads: A diffusion model generates continuous action trajectories from the language model’s conditioning
• Flow-matching heads: Similar to diffusion but using flow-based generation — FLOWER (2025) achieves state-of-the-art on CALVIN benchmarks

2.3 Action Chunking — The Key Production Innovation

A fundamental challenge: language model inference is slow. A 7B parameter model running on a robot’s onboard GPU might produce an action in 100–200ms. But a robot arm must update its joint commands at 50–100 Hz to move smoothly and safely. 200ms per action means 5 Hz update rate — dangerously slow for a physical robot.

Action chunking solves this by predicting not one action but a chunk of 8–50 future actions in a single forward pass. The robot executes this action sequence open-loop while the model computes the next chunk. This effectively decouples the model’s inference frequency from the robot’s control frequency — the model runs at 5–10 Hz, the robot executes at 50–100 Hz from the pre-computed action buffer.

This is not a minor optimisation — it is the engineering insight that made practical VLA deployment possible. Every serious production VLA system in 2026 (GR00T, Gemini Robotics, OpenVLA) uses action chunking in some form.

2.4 System 1 / System 2 Architectures

The most sophisticated production architecture in 2026 uses an asymmetric dual-system design, formalised by NVIDIA GR00T and Figure’s Helix model:

System 1 (Fast): A lightweight diffusion or flow-matching expert that produces motor commands at 50–100 Hz. This is the “muscle memory” of the robot — fast, reactive, conditioned on the System 2’s latest plan.

System 2 (Slow): A heavyweight Vision-Language Model that runs at 5–10 Hz, planning and re-planning based on the current visual scene and language instruction. This is the “conscious reasoning” of the robot — slow, deliberate, able to recover from unexpected situations.

This mirrors the dual-process theory of human cognition: fast, automatic System 1 responses and slow, deliberate System 2 reasoning. The insight that this framing from cognitive psychology maps directly onto the computational architecture of capable robots is one of the most elegant ideas in contemporary AI.

---

Module 3: NVIDIA GR00T — The Android of Robotics

3.1 GR00T Architecture and Capabilities

NVIDIA released GR00T N1 at GTC 2025 as the world’s first open, fully customisable foundation model for generalist humanoid reasoning and skills. The N1.5 generation, released at COMPUTEX May 2025, introduced:

• Eagle 2.5 VLM backbone: Improved visual grounding, especially for fine-grained spatial relationships critical for manipulation
• FLARE training objective: Enables learning from human video demonstrations without robot-specific action labels — dramatically expanding the training data available
• Isaac GR00T N1.6: Announced at CES 2026, purpose-built for humanoid robots with improved dexterous manipulation

GR00T runs on NVIDIA Jetson Thor — the compute platform designed for humanoid robots, providing the GPU performance needed for real-time VLA inference within a mobile robot’s power and thermal budget.

3.2 Practical Example: Fine-Tuning OpenVLA-7B for a Custom Manipulation Task

OpenVLA-7B is the most accessible VLA model for practitioners in 2026. Here is a complete fine-tuning pipeline for a pick-and-place task:

# Required: pip install transformers torch accelerate pillow
# OpenVLA model: HuggingFace hub: openvla/openvla-7b

from transformers import AutoModelForVision2Seq, AutoProcessor
from PIL import Image
import torch
import numpy as np

# ─────────────────────────────────────────────
# STEP 1: Load OpenVLA-7B model and processor
# ─────────────────────────────────────────────
device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Running on: {device}")

model = AutoModelForVision2Seq.from_pretrained(
    "openvla/openvla-7b",
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    trust_remote_code=True
).to(device)

processor = AutoProcessor.from_pretrained(
    "openvla/openvla-7b",
    trust_remote_code=True
)

# ─────────────────────────────────────────────
# STEP 2: Prepare a robot observation
# In production: replace with live camera feed from robot
# ─────────────────────────────────────────────
# Load a sample image (replace with robot camera frame)
image = Image.open("robot_workspace.jpg").convert("RGB")

# Natural language instruction — this is what makes VLAs powerful
instruction = "pick up the red block and place it in the blue container"

# ─────────────────────────────────────────────
# STEP 3: Create the VLA input prompt
# OpenVLA uses the format: "In: What action should the robot take to [instruction]?nOut:"
# ─────────────────────────────────────────────
prompt = f"In: What action should the robot take to {instruction}?nOut:"

inputs = processor(prompt, image).to(device, dtype=torch.bfloat16)

# ─────────────────────────────────────────────
# STEP 4: Run VLA inference — generate robot action
# OpenVLA outputs a 7-dimensional action vector:
# [delta_x, delta_y, delta_z, delta_roll, delta_pitch, delta_yaw, gripper_open]
# ─────────────────────────────────────────────
with torch.inference_mode():
    action = model.predict_action(
        **inputs,
        unnorm_key="bridge_orig",  # Dataset normalisation stats
        do_sample=False
    )

print(f"Robot action vector: {action}")
print(f"End-effector delta XYZ: ({action[0]:.4f}, {action[1]:.4f}, {action[2]:.4f})")
print(f"Wrist rotation RPY:     ({action[3]:.4f}, {action[4]:.4f}, {action[5]:.4f})")
print(f"Gripper: {'OPEN' if action[6] > 0.5 else 'CLOSED'}")

# ─────────────────────────────────────────────
# STEP 5: Action chunking — predict 8 future actions at once
# Reduces effective inference frequency from 1Hz to 8x more efficient
# ─────────────────────────────────────────────
def predict_action_chunk(model, processor, image, instruction, chunk_size=8):
    """Predict a chunk of future actions for smoother robot execution"""
    actions = []
    for _ in range(chunk_size):
        prompt = f"In: What action should the robot take to {instruction}?nOut:"
        inputs = processor(prompt, image).to(device, dtype=torch.bfloat16)
        with torch.inference_mode():
            action = model.predict_action(**inputs, unnorm_key="bridge_orig",
                                          do_sample=False)
        actions.append(action)
    return np.array(actions)

action_chunk = predict_action_chunk(model, processor, image, instruction)
print(f"nAction chunk shape: {action_chunk.shape}")  # (8, 7)
print("Executing 8-step action sequence on robot...")

What this code does in production: The robot captures a camera frame, sends it with a natural language instruction to OpenVLA, and receives a 7-DOF action vector specifying exactly how to move the arm and gripper. No explicit programming of object locations, grasping strategies, or motion plans — the VLA infers all of this from the visual scene and language instruction.

Hardware note: OpenVLA-7B requires approximately 15GB of GPU VRAM in bfloat16 precision. On a Jetson AGX Orin (64GB RAM, 2048-core Ampere GPU), inference runs at approximately 8–12 Hz — fast enough for real-time operation with action chunking.

---

Module 4: Fine-Tuning VLA Models for Custom Tasks

4.1 When Fine-Tuning Is Necessary

Pre-trained VLA models like OpenVLA-7B are trained on diverse robot manipulation datasets (BridgeData V2, Open X-Embodiment, etc.). They generalise well to tasks similar to their training distribution. However, for production deployment on specific robot platforms and tasks, fine-tuning is almost always necessary:

• Robot embodiment mismatch: The pre-trained model may have been trained on a different robot arm with different kinematics and camera viewpoint
• Task specificity: Industrial tasks (PCB inspection, surgical tool handover) require precision beyond general-purpose training
• Domain shift: Your operating environment (lighting, objects, workspace layout) may differ significantly from training data

4.2 LoRA Fine-Tuning Pipeline

Low-Rank Adaptation (LoRA) is the standard technique for efficient VLA fine-tuning. It adds small trainable matrices to the frozen pre-trained model, reducing VRAM requirements from 40+ GB (full fine-tuning) to 12–16 GB while achieving comparable performance:

# Required: pip install peft transformers torch datasets
from peft import LoraConfig, get_peft_model, TaskType
from transformers import AutoModelForVision2Seq, AutoProcessor, TrainingArguments, Trainer
from datasets import Dataset
import torch

# Load base model
model = AutoModelForVision2Seq.from_pretrained(
    "openvla/openvla-7b",
    torch_dtype=torch.bfloat16,
    trust_remote_code=True
)

# LoRA configuration — low-rank adaptation for efficient fine-tuning
lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=32,                        # Rank — higher = more parameters, better performance
    lora_alpha=16,               # Scaling factor
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],  # Attention layers
    lora_dropout=0.05,
    bias="none",
)

# Apply LoRA — only ~0.5% of parameters become trainable
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Output: trainable params: 42,925,056 || all params: 8,052,269,056 || trainable%: 0.53%

print("LoRA fine-tuning ready — 99.47% of the model frozen")
print("GPU memory required: ~13GB (vs 40GB+ for full fine-tuning)")

4.3 Data Collection Strategy

Fine-tuning a VLA model requires high-quality robot demonstration data. The standard pipeline:

Teleoperation data collection: Use a haptic input device (SpaceMouse, HTC Vive controller) or the AV teleoperation techniques covered in UDHY’s Complete Guide to AV Teleoperation to record human demonstrations of the target task. Collect 50–500 demonstrations depending on task complexity.

Data format: Each demonstration is a trajectory of (image, instruction, action) tuples. Store in LeRobot HuggingFace dataset format for compatibility with the standard fine-tuning pipeline.

Data quality over quantity: 200 high-quality demonstrations (consistent lighting, clear task completion, varied initial conditions) outperform 2,000 noisy ones. This is the same insight behind the Humanoid Robot Data Gap analysis — the quality of the physical demonstration data is the binding constraint on VLA performance.

---

Module 5: Deployment — From Model to Production Robot

5.1 Inference Optimisation for Edge Hardware

A VLA model running on cloud GPUs is straightforward. A VLA model running inside a mobile robot with battery constraints, thermal limits, and latency requirements is a significant engineering challenge.

TensorRT optimisation: Convert the action head (the most time-critical component) to TensorRT for 2–4× inference speedup on Jetson hardware:

# TensorRT export for action head — 3x speedup on Jetson AGX Orin
import torch
import tensorrt as trt

# After training, export the action head to ONNX first
torch.onnx.export(
    model.action_head,
    dummy_input,
    "action_head.onnx",
    opset_version=17,
    input_names=["hidden_states"],
    output_names=["actions"],
    dynamic_axes={"hidden_states": {0: "batch"}}
)

print("Action head exported to ONNX — ready for TensorRT compilation")
print("Expected speedup on Jetson AGX Orin: 2.5-4x vs PyTorch baseline")

Quantisation: INT8 quantisation reduces model size by 4× and inference latency by 2–3× with less than 5% accuracy degradation on standard benchmarks. For production deployment, this is almost always worth the accuracy tradeoff.

5.2 Safety Architecture for Physical AI

Physical AI systems introduce a new class of safety requirements that software-only AI does not have: when the model makes a mistake, the consequence is not a wrong answer — it is a physical collision, a dropped patient, or a damaged product.

Every production Physical AI system requires:

Workspace monitoring: A separate, safety-critical perception layer that monitors the robot’s workspace independently of the VLA model. If a human enters the workspace, the safety layer triggers an emergency stop regardless of what the VLA model is doing.

Force/torque limits: Hardware-level torque limits on every joint that cannot be overridden by software. These are the physical equivalent of a circuit breaker.

Confidence thresholding: The VLA model should output an uncertainty estimate alongside each action. Actions with high uncertainty trigger a “safe stop and wait” behaviour rather than blind execution. This connects directly to the edge case challenge explored in Why Self-Driving Cars Still Fail — the system must know what it does not know.

Anomaly detection: A lightweight model monitoring the robot’s behaviour in real time, flagging trajectories that deviate significantly from expected patterns for that task.

---

Module 6: The Physical AI Landscape — Research Frontiers in 2026

6.1 World Models for Robotics

World models — neural networks that learn an internal simulation of physical reality — are emerging as the next frontier beyond VLA models. Rather than directly mapping observations to actions, a world model enables the robot to mentally simulate the consequences of its actions before executing them.

NVIDIA Cosmos Transfer (announced CES 2026) is the most prominent example: a physics-aware simulation model that generates photorealistic video of physical interactions, enabling robots to reason about “what would happen if I pushed this?” without touching the actual object.

6.2 Embodied Reasoning — VLA-Reasoner and Monte Carlo Tree Search

The latest research pushes VLA capabilities toward complex reasoning chains. VLA-Reasoner (arXiv, September 2025) uses online Monte Carlo Tree Search to enable VLA models to reason through multi-step manipulation tasks — thinking ahead through action sequences before committing to execution.

Embodied-R1 (August 2025) applies Reinforced Fine-Tuning (RFT) to train embodied reasoning — pointing to objects by function (“point to the handle,” “point to the lid”) rather than by visual category. This level of functional understanding represents a qualitative jump beyond current VLA capabilities.

6.3 Open-Source Ecosystem

The open-source Physical AI ecosystem is remarkably active in 2026:

• OpenVLA-7B: Stanford — open weights, Apache 2.0 license
• LeRobot: HuggingFace — complete data collection, training, and evaluation pipeline
• AGIBOT WORLD 2026: 20,000+ hours of real-world robot demonstrations, open-source
• LingBot-VLA: Ant Group — 20K hours of dual-arm data, 1.5-2.8× training speedup

This open ecosystem means the barriers to conducting serious VLA research have fallen dramatically. A well-equipped university lab or well-funded startup can now access training data, pre-trained models, and training frameworks that would have required a Google-scale team to assemble two years ago.

---

What You Have Learned

By completing this course, you can now:

• Explain the Physical AI paradigm shift and how VLA models differ from classical robot programming
• Describe the three-component VLA architecture (vision encoder, language backbone, action head)
• Implement action chunking and System 1/System 2 architectures for real-time VLA deployment
• Run inference with OpenVLA-7B and interpret the 7-DOF action vector output
• Fine-tune a VLA model with LoRA for a custom robot manipulation task
• Design the safety architecture required for production Physical AI deployment

Your next course: Multi-Agent Robot Systems and Fleet Intelligence — where multiple Physical AI agents coordinate to accomplish tasks no single robot can handle alone.

---

FAQ

---

References

1. Kim, M. et al. (2024). OpenVLA: An Open-Source Vision-Language-Action Model. arXiv:2406.09246. arxiv.org
2. NVIDIA. (2025). Isaac GR00T N1 — Foundation Model for Humanoid Robots. developer.nvidia.com
3. Google DeepMind. (2025). Gemini Robotics On-Device. deepmind.google
4. MarkTechPost. (April 2026). Top 10 Physical AI Models Powering Real-World Robots in 2026. marktechpost.com
5. Brohan, A. et al. (2023). RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. Google DeepMind. arXiv:2307.15818. arxiv.org
6. Deloitte. (February 2026). Physical AI and Humanoid Robots — Tech Trends 2026. deloitte.com
7. HyScaler. (March 2026). Vision-Language-Action (VLA) Guide for 2026. hyscaler.com
8. TechCrunch. (January 2026). NVIDIA wants to be the Android of generalist robotics. techcrunch.com
9. ArticleSledge. (November 2025). Vision Language Action (VLA) Models: Complete Guide. articsledge.com
10. GitHub. (2026). awesome-physical-ai — Curated VLA model research. github.com/keon/awesome-physical-ai

---

Designed by Dr. Dilip Kumar Limbu — Former Principal Research Scientist, A*STAR · Co-Founder, Moovita, Singapore’s first autonomous vehicle company · 30 years building real-world autonomous systems. UDHY.com.