Q-OIL: Optimistic Exploration in Learning from Human Interventions

Pranav Teegavarapu1,* Sriyash Poddar1,* Rico Qi1 Cleah Winston1 Krisha Khandelwal1 Aykut Onol2 Natasha Jaques1,‡ Abhishek Gupta1,‡

1University of Washington   2Toyota Research Institute

*Equal contribution   ‡Equal advising   CoRL 2026

Humans intervene suboptimally: they intervene late and their corrections are suboptimal. Q-OIL uses these corrections to guide exploration, and achieves 10–50% higher success on real robots with almost 30% less human effort.

Try Q-OIL in your browser

The robot learns to reach the pink ring from scratch. Take over to help it: click a cell to steer it there, or click the maze and use the arrow keys.

Algorithm
Show
In control
–
Episodes
0
Success, last 20
0%
Takeovers, last 20
0%
SuccessSteps under takeoverRobot's best route

The problem

Simulated humans don't act like real ones

So simulated benchmarks rank methods differently than real users do.

Existing methods are fast or optimal, not both

Imitation stops at the person's level. Plain RL gets there slowly.

The fix

How do people intervene?

We asked 8 participants to supervise a simulated robot as it learned 3 manipulation tasks, collected about 2,000 episodes (200k steps) of their interventions, and analyzed when and how they stepped in.

Left: progress over a 12-step window separates intervention states from other states, while the instantaneous action value does not. Right: value curves where takeovers follow plateaus or drops.

Takeovers follow plateaus or drops in value, not single bad actions.

DesignSo Q-OIL keeps the bonus from leaking back to the states before a takeover.

Bar chart: intervention actions are about three times more suboptimal than the reference policy.

Takeover actions are about 3× more suboptimal than an expert policy.

DesignSo Q-OIL rewards corrections instead of copying them.

At intervention states, the converged policy's actions have higher value than the human's actions.

The trained robot picks different, better actions than the person did.

DesignSo corrections only guide exploration, and their influence fades as takeovers become rare.

Rank correlation with real-user algorithm rankings: PAM 1.00, action-suboptimality 0.77, random 0.37.

PAM, our progress-aware intervention model, matches real users' ranking exactly. Its coefficients k, δ, α, β are learned from real intervention data.

P(intervene) = α  if  V*(st) − V*(st−k) < δ,  else  β

BenchmarkSo we compare every method with PAM in simulation.

Reward corrections, don't copy them

We use these insights to design Q-OIL (Q-Optimistic Intervention Learning): a simple change to any off-policy RL algorithm that rewards takeovers instead of imitating them.

HIL RL with interventions
  1. for each step:
  2. act: the person if they take over, else the robot
  3. store (s, a, r, s')
  4. update Q toward r + γ Q(s', a')
  5. update the policy to maximize Q

A takeover is just one more transition.

Q-OIL
  1. for each step:
  2. act: the person if they take over, else the robot
  3. store (s, a, r, s'), plus a bonus b if it was a takeover2
  4. update Qtask toward r + γ Qtask(s', a')
  5. update Qopt toward r + b + γ Qtask(s', a')1
  6. update the policy to maximize Qopt, plus BC on takeovers3

Takeovers are rewarded, and the bonus never leaks backward.

addedchanged123data-driven motivations

How does Q-OIL work?

Each takeover earns a small bonus, which raises the value of the path the person showed, so the robot explores there instead of copying the exact actions. A second, task-only critic provides the bootstrap targets, so the bonus cannot leak back to the states where the robot was failing. As the robot needs help less often, bonus steps become rare and Q-OIL settles on the policy that is best for the task.

How does it compare to other methods?

Q-OILThe bonus stays on the corrected path, and the policy solves the maze.
No decouplingWith one critic, the bonus leaks to every state and the policy goes nowhere.
HILCorrections are just more data. The policy does not find the way in time.

Results

Simulation, with PAM

Success rate on Lift with the simulated human. Success rate on Can with the simulated human. Success rate on Square with the simulated human.
Legend: Q-OIL, Q-OIL without BC, HG-DAgger, HIL, BC+RL, SIRIUS, PVP, RLIF.

Real people, simulated robots

Success rate on Lift with real human supervisors. Success rate on Can with real human supervisors.
Legend: Q-OIL, Q-OIL without BC, HG-DAgger, HIL, RLIF, BC+RL.

Real robots: Franka

Success rate on pen in bowl. Success rate on peg insertion. Success rate on unbuttoning a shirt.
Legend: Q-OIL, HG-DAgger, HIL, BC+RL.

Pretrained VLAs, bimanual robot

Base VLA BC+RL Q-OIL

Ablation: why two critics?

Success averaged over training versus optimism bonus size. Q-OIL stays near 0.7 from 0.01 to 0.5; without decoupling, success falls from 0.3 to near 0.
Success vs. bonus size, averaged over the Robomimic tasks.
One critic
Learned value with a single critic: high values hug the start in the top-left corner.
Q-OIL, two critics
Learned value with decoupled critics: values increase toward the goal in the bottom-right corner.

Learned value on a grid world after 50k steps, from the start (top left) to the goal (bottom right).

Q-OIL is stable across a 50× range of bonus values. With one critic, the bonus leaks backward: value piles up near the start instead of the goal, and performance collapses as the bonus grows.

BibTeX

@inproceedings{qoil2026,
  title     = {Q-OIL: Optimistic Exploration in Learning from Human Interventions},
  author    = {Teegavarapu, Pranav and Poddar, Sriyash and Qi, Rico and
               Winston, Cleah and Khandelwal, Krisha and Onol, Aykut and
               Jaques, Natasha and Gupta, Abhishek},
  booktitle = {Conference on Robot Learning (CoRL)},
  year      = {2026},
}