Robot learning · Reinforcement learning

RL-VLA

Residual reinforcement learning for vision-language-action policies

A framework for learning action corrections around a frozen OpenPI policy. A compact PyTorch actor uses state and the proposed action chunk to refine the controller, with SAC and PPO sharing the same observations, action interface, and evaluation workflow.

View on GitHub

Architecture diagram: frozen OpenPI actions and learned residual corrections combine before reaching the task controller — The base policy stays frozen while a residual actor learns bounded corrections to its actions.

Keep the base, learn the correction

OpenPI proposes an action chunk from images, state, and language. A small actor conditions on state, optional frozen features, and the base action prefix, then adds bounded corrections before commands reach the task controller.

Compare SAC and PPO

Two residual learners use a common environment contract: SAC with replay and twin critics, and PPO with fresh rollouts and clipped updates. The training loop accounts for action chunks that end early and separates terminal states from time limits.

Connect policy and simulator

A websocket adapter connects to a separately hosted OpenPI policy. Docker environments package the learners and a headless LIBERO simulator, while checkpoints, run manifests, and per-episode evaluation make experiments inspectable.

Validate the learning pipeline

CPU toy runs exercise training, checkpoint reloads, and evaluation. A real LIBERO smoke check covers reset, RGB rendering, and a controller step. End-to-end learning with a real OpenPI checkpoint remains the next validation stage.

The current results validate the software pipeline; they do not establish improved robot performance.