Offline-to-Online Robotic Reinforcement Learning

Robo-ValueRL: Reliable Value Estimation for Offline-to-Online Reinforcement Learning

Wenke Xia1,*, Pei Ren2,*, Wenbo Yu3, Yizhuo Zhang1, Jifan Li1, Yixue Zhang2,4, Yinuo Zhao2, Qingyang Gao2,
Jianlong Fu5, Jian Tang2, Ji-Rong Wen1, Zhengping Che2,†, Di Hu1,†

  1. 1 Gaoling School of Artificial Intelligence, Renmin University of China
  2. 2 Beijing Innovation Center of Humanoid Robotics
  3. 3 Beijing Forestry University
  4. 4 Peking University
  5. 5 Microsoft Research

* Equal contribution † Corresponding author

Robo-ValueRL studies how value-function reliability shapes offline-to-online robotic reinforcement learning. It learns history-conditioned value estimates, propagates them into quality-conditioned policy pretraining, and uses reliable value guidance to stabilize online improvement from real robot rollouts.

240h offline data 3,000+ online rollouts 86% chip insertion 84% block disassembly
Paper PDF Code GitHub Model & Dataset Hugging Face Community QR Code
Task Videos

Two real-robot testbeds for value-guided policy improvement.

Methodology

Reliable values become a data-utilization interface.

The framework links value estimation, quality-conditioned policy learning, and residual online adaptation in one offline-to-online pipeline.

Robo-ValueRL offline-to-online reinforcement learning pipeline
01

History-conditioned value estimator

Recent visual history reduces ambiguity from occlusions, repeated motions, and visually similar task stages. The three clips can later show value progress, error sensitivity, and history ablations.

02

Quality-conditioned consistency policy

Value differences become action-quality indicators for efficient one-step action generation under high-frequency control. This group can show quality labels and task-specific policy behavior.

03

Online residual adaptation

High-quality rollout segments train a lightweight adapter while preserving the pretrained base behavior. Later videos can show the residual gate, self-correction, and failure recovery.

Results

Reliability metrics predict downstream policy quality.

Across 240 hours of offline demonstrations and over 3,000 online rollout trajectories, reliable value estimates support stronger offline scaling and more stable online improvement.

240h offline demonstrations
3,000+ online rollout trajectories
+26% chip offline gain
+34% block offline gain
Value estimation metrics and downstream success rate

From value reliability to policy improvement

Reliable value functions preserve global progress, maintain local fluency, and detect execution errors. These properties align with downstream policy success.

  • Value-guided offline RL scales better than quality-agnostic behavior cloning.
  • Reliable value guidance filters noisy online rollouts.
  • Residual adaptation improves behavior without overwriting the base policy.
Offline pretraining scaling with heterogeneous data
Offline pretraining scaling with heterogeneous data
Success rate and failure-mode distribution
Success rate and failure-mode distribution

Detailed demos

Recovery behavior belongs with the evidence.

Failure recovery, self-correction, and online adaptation are treated as method/result evidence rather than intro tasks.

Condition

Value estimates become conditions for policy improvement.

Robo-ValueRL treats value reliability as an interface between heterogeneous robot experience and downstream policy learning.

A history-conditioned value estimator infers task progress from multi-view observations and visual history. Its value differences are converted into action-quality indicators, allowing the VLA policy to condition on high-quality behavior during offline pretraining.

During online improvement, the same value guidance filters rollout segments and activates a lightweight residual adapter only where targeted correction is useful. This keeps adaptation focused while preserving the pretrained behavior prior.

History-conditioned value Action-quality condition Value-guided residual adaptation
Robo-ValueRL value conditioning overview
Citation

Cite Robo-ValueRL.

Citation will be updated after the arXiv release.