The fastest lap through the corridor

Teach a robot, by trial and error, to drive the 60 cm corridor with its sharp right-hand bend as fast as it can without touching the walls. Everything runs on your own PC: the Botopia simulator in the background, your Python code in front. About an hour, of which a few minutes of training.

1The goal

The robot is the skid-steer robot: four fixed wheels, it steers by turning the left and right side at different speeds. It gets two commands, exactly like the W/A/S/D keys: fwd (−1 … 1, backward / forward) and turn (−1 … 1, right / left). It starts in front of the corridor. The lap is done when it is past the bend.

This is what you will end up with: the policy that PPO learned, driving its fastest lap.

The trained robot is not much faster than a good hand-written controller, and that is fine: the point is that nobody told it how to take the bend. It found out by itself, from a single number per step that says “good” or “bad”.

2What you need

• The API repository: git clone https://github.com/mdequanter/robotSimulatorAPI.git (API server, Python client and the tutorial scripts; see Getting started). The API server fetches the simulator pages from this site itself, so you do not need PHP or a web server.
• Node.js 22 or newer (node --version). No npm install needed.
• Google Chrome (or Chromium / Edge). The server starts it invisibly in the background.
• Python 3.10 or newer with three packages:

pip install stable-baselines3 gymnasium pillow

stable-baselines3 brings PyTorch along (the CPU version is enough; no graphics card needed). pillow is only for saving a lap as an animated GIF.

The two Python files of this tutorial are in python/ of that repository:

3Start the simulator

Open a terminal in the robotSimulatorAPI folder and start the API server:

node server.mjs
Botopia simulator API on http://localhost:8780/api/  (simulators: http://localhost:8780/)
Chrome headless, robots: wheeled, bike, synchro, mecanum, kiwi, swerve, tracked, skid, ...

Leave this terminal open. Every environment you create is one simulator page in that Chrome, running in lock-step: simulated time only moves when you call step. So training is not tied to the clock: one step of 0.1 s simulated time takes about 1 ms.

4Drive it by hand

Before any learning, try the API yourself. In a second terminal, start Python from python:

from botopia import BotopiaAPI
api = BotopiaAPI("http://localhost:8780")

env = api.create("skid", "corridor")        # a skid-steer robot in front of the corridor
eid = env["id"]
print(env["info"]["drive"])                    # ['fwd', 'turn']

for i in range(30):                             # 30 steps of 0.1 s: full throttle
    r = api.step(eid, {"drive": {"fwd": 1.0, "turn": 0.0}}, dt=0.1)
    if r["events"]:
        print(r["obs"]["t"], r["events"])

print(r["obs"]["pose"])
open("view.jpg", "wb").write(api.frame(eid))      # what the camera sees now

Every step returns an observation (obs: position and heading, speeds, IMU, every motor with PWM, encoder and current, progress in the lane) and the events of that step. Driving straight on, you will soon see a collision event: the robot reached the far wall of the bend. Open view.jpg to see it standing there.

Positions are in metres. x runs along the lanes, z sideways (to the robot's right), yaw is the heading in radians, positive to the left. All fields are described in the API documentation.

5The task as an RL problem

Reinforcement learning needs four things: what the robot sees (observation), what it can do (action), what is good (reward) and when an attempt is over (episode). Those choices are most of the work: the learning algorithm itself comes off the shelf.

reward = 10 × (metres gained along the centre line)
       − 0.1                     # time penalty: every step costs something
       + 10   at the goal
       − 5    when it touches a wall (and the attempt ends)

6The environment class

Learning libraries expect the Gymnasium interface: a class with reset() and step(action). CorridorEnv in corridor_rl.py wraps one simulator. The heart of it, shortened:

class CorridorEnv(gym.Env):
    def __init__(self, robot="skid", url="http://localhost:8780", dt=0.1, max_steps=200,
                 end_on_hit=True, time_penalty=0.1):
        self.api = BotopiaAPI(url)
        self.id = self.api.create(robot, "corridor")["id"]   # one simulator page
        self.action_space = spaces.Box(-1.0, 1.0, shape=(2,))          # fwd, turn
        self.observation_space = spaces.Box(-5.0, 5.0, shape=(10,))
        ...

    def reset(self, seed=None, options=None):
        self.obs = self.api.reset(self.id, lane="corridor", seed=seed)  # back to the start
        self.steps, self.s = 0, 0.0
        return self._features(self.obs, False), {}

    def step(self, action):
        a = [float(np.clip(v, -1, 1)) for v in action]
        r = self.api.step(self.id, {"drive": {"fwd": a[0], "turn": a[1]}}, dt=self.dt)
        self.obs, self.steps = r["obs"], self.steps + 1
        p = self.obs["pose"]
        s, _, _ = path_frame(p["x"], p["z"])                       # progress along the centre line
        hit = any(ev["type"] in ("collision", "blocked", "stall") for ev in r["events"])
        goal = p["z"] >= GOAL_Z

        reward = 10 * (s - self.s) - self.time_penalty + (10.0 if goal else 0.0)
        if hit:
            reward -= 5.0 if self.end_on_hit else 1.0
        self.s = s
        terminated = goal or (hit and self.end_on_hit)
        truncated = self.steps >= self.max_steps and not terminated
        return self._features(self.obs, hit), reward, terminated, truncated, info

path_frame(x, z) finds the nearest point on the centre line (two straight pieces) and returns s, e and the direction of the line there. _features turns that, together with the speeds from the observation, into the 10 numbers of the table above.

7The time to beat

Before training, always build a simple non-learning solution. It tells you whether the task is possible at all and what “good” is. baseline_action aims at a point 0.5 m ahead on the centre line and slows down when it has to turn hard:

python python/corridor_rl.py baseline
Hand-written driver: 4.90 s, wall hits 0, total reward 31.0
Full throttle, no steering: did not finish in 20 s, wall hits 2, total reward -6.8

8Train with PPO

PPO (Proximal Policy Optimization) is a solid default for continuous actions such as throttle and steering. Its policy is a small neural network (two layers of 64) that takes the 10 observation numbers and outputs the two actions, plus a bit of randomness so it keeps trying new things. After every 2048 steps it nudges the network towards whatever earned more reward.

python python/corridor_rl.py train
-----------------------------------------
| rollout/                |             |
|    ep_len_mean          | 49.6        |
|    ep_rew_mean          | 23.6        |
| time/                   |             |
|    total_timesteps      | 61440       |
...
Saved corridor_ppo.zip after 162 s of training

Eight simulators gather experience at the same time (--envs 8): every step is one batch call to the API, and the server steps all eight Chrome tabs in parallel, each on its own CPU core. The code that sets it up is short:

venv = VecMonitor(make_vec_env(8, "skid", "http://localhost:8780"))   # 8 simulators, one API call per step
model = PPO("MlpPolicy", venv, n_steps=256, batch_size=256, learning_rate=3e-4, gamma=0.99,
            ent_coef=0.003, verbose=1, seed=1, device="cpu")
model.learn(total_timesteps=150_000)
model.save("corridor_ppo.zip")

The curve and the table below come from an earlier run with 4 simulators stepped one by one (6½ minutes, best lap 4.80 s). With the faster set-up our run took 2.7 minutes and drove 5.00 s: RL results always differ a little from run to run.

Reading the log

Two lines matter: ep_rew_mean, the average reward of the last attempts, and ep_len_mean, how many steps they lasted. Below is the curve from the run that produced the video at the top.

Notice the dip in ep_len_mean early on: the robot first learns to drive, and crashes sooner. Longer attempts later are a good sign (it gets further), and at the very end the attempts get shorter again because it finishes faster.

Faster training: CPU cores, not the GPU

Where does the training time go? Measured on a laptop with 16 CPU cores and an RTX 5070: about 90 % simulating the robot, about 10 % updating the neural network.

So the way to train faster is more simulators in one batch call: --envs, about one per two CPU cores. A GPU only speeds up the network part, and for a network this small (2 × 64 neurons) copying the data to the GPU and back costs more than it saves: stable-baselines3 itself advises to run PPO with an MlpPolicy on the CPU.

A GPU does pay off for big networks, for example when the robot learns from camera images (CnnPolicy). Then install the CUDA build of PyTorch (pick your CUDA version on pytorch.org; RTX 50-series cards need CUDA 12.8 or newer) and train with --device cuda:

pip uninstall torch
pip install torch --index-url https://download.pytorch.org/whl/cu128
python -c "import torch; print(torch.cuda.is_available())"     # True = the GPU is usable
python python/corridor_rl.py train --device cuda

9Test and watch

Drive test laps without the training randomness (deterministic=True):

python python/corridor_rl.py eval
PPO, run 1: 4.80 s, wall hits 0, total reward 31.3
PPO, run 2: 4.80 s, wall hits 0, total reward 31.3
PPO, run 3: 4.80 s, wall hits 0, total reward 31.3

Three identical times: the simulator is deterministic, the same actions always give the same result. For the test laps a wall touch does not end the attempt; it is counted instead.

Two ways to watch:

# 1. save one lap as an animated GIF
python python/corridor_rl.py watch --gif lap.gif

# 2. live: restart the server with a visible Chrome, then drive one lap in real time
node server.mjs --show
python python/corridor_rl.py watch

10Experiments

Now the fun part. Change one thing at a time, train again, and compare with eval.

Next: tutorial 2, where the robot gets no position at all, only its camera picture, and learns to drive a whole track from that.

Going further: action_mode “motors” in botopia.py lets the network drive the four motors' PWM directly, without the robot's own controller. And every other lane (stairs, gaps, ramps…) and robot works with the same API.

11Troubleshooting