Stand the blocks up in a row: learn the right height

The OpenMANIPULATOR-X takes six tower-game blocks from a stack and stands each one on its small end, side by side in an arc in front of it. It already can pick and place. What it has to learn is how high: its own idea of the height is off by about a centimetre, like that of a real arm. With Q-learning it finds the right correction for every step by trial and error, in about 200 attempts.

1The job

The blocks are 75 × 25 × 15 mm. Standing, a block is 75 mm high on a foot of 15 × 25 mm: let it go from a few millimetres too high and it falls over. Open the scene in the OpenMANIPULATOR-X simulator: the six blocks stack at x = 155 mm in front of the arm (New blocks), the row's spots are on an arc 240 mm from the arm, from 40° to the left to 5° to the right, on a plate of 46 × 34 cm.

2Why learn a height?

In the simulator the arm knows exactly where it is. A real arm doesn't: the servos have a little play, the links bend, the zero of a joint is a degree off, the table is not where you measured it. Our own OpenMANIPULATOR-X gripped 1 cm too high and missed every block, until we found that the shoulder axis sits 17 mm higher than our model said.

You can measure and calibrate. Or the robot can try, see what happens and correct itself. That is what this tutorial does. The simulator gets a hidden calibration error: the arm is 10 mm higher than it thinks near the stack, and it hangs a little lower further out (4 mm per 100 mm). The learner never sees these numbers; it only sees what happens: a missed block, a block that falls over, a block that stands.

3What you need

The robotSimulatorAPI repository, Node.js 22+, Chrome and Python 3.10+, as in tutorial 1. No extra Python packages: the Q-table is a plain dictionary. The script is python/stand_rl.py.

node server.mjs                        # terminal 1: the simulators
python python/stand_rl.py baseline   # the page's own heights: the problem
python python/stand_rl.py train      # Q-learning, 300 attempts (~15 s), saves stand_q.json
python python/stand_rl.py eval       # the learned policy, skill by skill

To watch it, start the server with a visible browser and use watch:

node server.mjs --show
python python/stand_rl.py watch --speed 2

A trained Q-table is included: add --model python/models/stand_q.json to eval or watch.

4The two skills

Two skills, built into the OpenMANIPULATOR-X page. Each takes one number: a height correction dz in mm on the page's own idea of the right height. You can call them from your own code too.

The observation tells what happened: extra.row.last has the result, and for the tutorial also where the fingertips really ended up (tipsMm) or the real gap (gapMm) and how far a standing block is from its spot (offMm). The learner only uses the result and offMm.

5State and Q-learning

The learner sees only which skill is next and how far the job is:

state = ("pick", blocks on the stack)        # gripper empty: 6 … 1
state = ("place", blocks already standing)   # holding a block: 0 … 5
action = dz in [-15, -10, -5, 0, +5, +10, +15] mm

Twelve states, seven actions: a Q-table of 84 numbers. Every state gets its own correction, because a real arm's error is not the same everywhere: higher on the stack, further out at the row. After every skill:

Q[s][a] += α · ( r + γ · max(Q[s′]) − Q[s][a] )
# r = reward of this skill, s′ = the state after it, α = 0.3, γ = 0.95

Choosing is ε-greedy: with probability ε a random correction (exploring), otherwise the one with the highest Q-value. ε starts at 1 and drops to 0.05 over the first 60 % of training. Compared with tutorial 4 (which skill?), the question here is how: the same method, a different kind of decision.

6The reward

r = -1                                     # every skill costs time
pick:   grasped +2,  missed -2,  pushed or table -6
place:  standing +10 - (mm off its spot),  fell over -10,  pressed: 4 less
r += 20 when all six stand

An attempt ends when all six stand, when the stack is empty and the hand too (a block that fell over is lost for this attempt), or after 40 skills. A miss costs little but wastes time; a block that falls over costs a lot, and the +20 for the whole row can then no longer be earned.

7Train it

python python/stand_rl.py baseline
  pick  dz  +0 mm  missed     fingertips  14.0 mm above the block's bottom  stack 6  standing 0
  pick  dz  +0 mm  missed     fingertips  14.0 mm above the block's bottom  stack 6  standing 0
  …
No correction (the page's own heights): NOT finished: 0/6 standing, 40 skills

python python/stand_rl.py train
episode  100  ε 0.45  return   10.3  last 20: mean return   -3.7, job done 0/20, mean standing 3.6  5 s
episode  150  ε 0.17  return   78.9  last 20: mean return   48.5, job done 9/20, mean standing 5.2  6 s
episode  200  ε 0.05  return   71.6  last 20: mean return   69.5, job done 17/20, mean standing 5.8  8 s
episode  300  ε 0.05  return   75.7  last 20: mean return   70.8, job done 17/20, mean standing 5.8  11 s
Saved stand_q.json (12 states) after 11 s of training

Three training runs (--seed 1, 2, 3):

During training it never gets 20/20: with ε = 0.05 one correction in twenty is still random, and a random correction can knock a block over. eval doesn't explore, and does the job every time.

8What it learned

At the end of training the script prints the best correction for every state:

Learned height corrections:
  pick  (blocks on the stack)      6: -5   5: -10   4: -5   3: -5   2: -10   1: -10
  place (blocks already standing)  0: -5   1: -5   2: -5   3: -5   4: -5   5: -5

Compare that with the hidden error, which the learner never saw:

The Q-values in two states (from models/stand_q.json):

At the stack −5 and −10 both work; which one wins depends on the seed. Placing has only one good answer. Without the hidden error (--zerror 0 --zsag 0) it learns 0 for every placement and 0 or +5 for picking: there is nothing to correct, and +5 still grips 9 mm of the block.

9On the real arm

The script can run the same two skills on the real OpenMANIPULATOR-X. It doesn't use the Node API for that (its browser has no USB connection), but the page in your Chrome, through a remote session:

1. Open indexOpenManipulator.php?remote=ROW1 in Chrome or Edge (any code instead of ROW1). The badge bottom right says Remote session ROW1 · connected.
2. In the Real robot panel: Connect OpenCR (USB), Torque on, tick Real arm follows the simulator.
3. Stack the six real blocks as in the simulator: straight in front of the arm, 155 mm from the first servo's axis, lengthwise towards the arm.
4. Run the script. It asks before every skill (Enter: go, q: stop); keep a hand near STOP (Esc).

python python/stand_rl.py real --session ROW1
python python/stand_rl.py real --session ROW1 --pick-dz -3 --place-dz 2   # your own corrections

The simulator makes every move and the real arm follows its joints. Start with no correction (the default): the learned −5 / −10 are for the simulator's hidden error, not for your arm. Watch where the real fingertips end up, and correct with --pick-dz and --place-dz. The results the script prints are the simulator's; whether the real block stands, you see on the table. To let the arm learn its real corrections by itself, it would need to see that: the USB camera on the gripper (Camera on, the real view in the API) is the place to start.

10Experiments

11Troubleshooting