Where a 14-DoF Biped Falls

September 9, 2026

A benchmark score is one number. A policy’s failure boundary is a shape, and two policies with the same score can have very different shapes. This post is about a case where the shape was the whole finding.

We trained a walking policy for Microduck, Pollen Robotics’ 14-degree-of-freedom open-source biped, with the upstream mjlab recipe, saving a checkpoint every 250 iterations. Then we froze checkpoints and used Proofload to measure where each one falls: a grid of lateral push magnitudes and floor frictions, 100 paired rollouts per cell, a survival estimate per cell, and a paired McNemar test with Benjamini–Hochberg correction across the grid.

At a glance

• an early checkpoint looked more robust than the converged one. It wasn’t: the push landed on the same control step in every rollout, so the 100 seeds in each cell were not varying anything. The paired report showed it as 0% or 100% in every cell.
• fixed, the converged checkpoint is significantly safer in 8 of 8 cells
• checkpoint 2250 has a cliff and checkpoint 5999 has a slope. Below 0.600 m/s the cliff wins; above 0.625 m/s the slope wins. 8 cells safer, 5 worse, 1 no significant change.
• a retrain on the upstream VelStand task, meant to widen the boundary, made it worse in 14 of 14 cells
• the developmental sweep shows both recipes pass through the same rigid checkpoint and lose it at the next one sampled; only the baseline recovers

How we measured it

Proofload is a statistical instrument, not a simulator or a training loop. It takes two things from us: a sweep spec naming the axes and their values, and an import path to a rollout function of our own. The function receives one condition and one seed and returns an outcome label and a time. Everything about the robot, the physics and the push lives in that function, in our repo. Proofload never sees MuJoCo.

From there the tool does the same things whatever the system under test is. It builds the grid, derives one seed per condition and replicate from a run salt, so a second run with the same salt gets the same seeds cell for cell and the comparison is paired by construction. It runs the rollouts, appends every episode to a store with its config, seed, outcome, time and whatever metadata the rollout attached, and never rewrites a stored episode. Given one run it computes the cumulative incidence of the event per cell. Given two runs it computes the paired difference per cell, a McNemar exact test, Benjamini–Hochberg correction across the grid, and the minimum detectable effect each cell had, then writes a report that prints the worst-powered cell next to the verdict count. That last line is the one we lean on most below.

Nothing in that list is specific to legged robots. The Go1 diff on the Proofload page came from a different rollout, a different simulator and a different grid through the same verbs. A rollout that runs an agent against a task and returns succeeded or failed fits the same contract, which is where the tool is headed.

1. The seeds weren’t varying anything

The first paired run compared checkpoint 1500 against the converged checkpoint 5999 on 8 cells: four push magnitudes from 0.45 to 0.675 m/s, two frictions, 100 rollouts each. Checkpoint 1500 fell in 0 of its 800 rollouts. Checkpoint 5999 fell in 400 of 800. Taken at face value, the earlier checkpoint was the robust one.

The paired report is what made that impossible to take at face value. Every one of the 8 cells came out at exactly 0.0 or exactly 100.0. The 72-cell developmental sweep run the same night did the same thing: 72 cells, each 0 or 100. A hundred seeds per cell and no within-cell variance at all.

The cause was in our rollout, not in the tool. The push was applied at control step 100, two seconds in, on every seed. The seed varied the initial pose by a few millimetres and nothing else that mattered. A biped’s response to a lateral shove depends on where it is in its stride when the shove lands, and with the push time fixed, every rollout in a cell was landing on the same stride phase. Checkpoint 1500’s gait happened to put a favourable phase at step 100. That is not robustness.

The fix was one line: draw the push step uniformly from 100 to 119, a full 20-step stride window (2.00 to 2.38 s at the 50 Hz control rate), from the same seed that Proofload hands the rollout. The stored episodes now carry the push step in their metadata, so the next reader can check.

Rerun on the same 8 cells, the result inverted. Checkpoint 5999 was significantly safer in 8 of 8, with fall rates from 2% to 40% against checkpoint 1500’s 20% to 74%. The largest q was 0.0027. The worst-powered cell could detect an 18 to 20 pp change, so nothing smaller than that is claimed. Runs 20260909T003019Z-7ab3 and 20260909T003114Z-e81c, report.

We are not claiming Proofload diagnosed this. It ran the grid, stored every episode, and printed a table in which the absence of variance was impossible to miss. The diagnosis took a person and about ten minutes.

2. Two shapes and a crossover

With the phase fixed, the developmental sweep showed something the first run had hidden. Checkpoint 2250 fell in 0 of 300 rollouts at 0.45, 0.525 and 0.60 m/s, and in 100 of 100 at 0.675. Checkpoint 5999 fell in 2 to 15% across the same three cells and 35 to 37% at 0.675. Neither number is the other’s summary. So we resolved the region between them: 2250 against 5999, push from 0.550 to 0.700 m/s in 0.025 m/s steps, both frictions, 100 paired rollouts per cell, 2,800 rollouts in all.

Below 0.600 m/s, checkpoint 2250 is nearly perfect: 2 falls in 600 rollouts. Checkpoint 5999 falls 5 to 16% of the time in the same cells, and Proofload calls 5999 significantly worse in five of those six cells. A benchmark run at 0.55 m/s would rank 2250 above 5999 and would be right about that cell.

At 0.625 m/s the ranking flips. Checkpoint 2250 falls 52% and 58% of the time; checkpoint 5999 holds at 16% in both friction rows. By 0.650 m/s checkpoint 2250 is at 96% and 99%, and from 0.675 m/s up it falls every time, while 5999 stays between 25% and 45% out to the edge of the grid.

That is two boundary geometries. One is a cliff: near-zero failure inside, total failure outside, with the transition inside a 0.025 m/s interval. The other is a slope that starts earlier and never reaches the top within the tested range. A single number at a single push cannot say which one you have, and which one you want depends on how sure you are of the disturbances you will meet.

The table is the one Proofload emits, with its own columns. The rows Proofload calls worse and the one it calls no significant change are in it.

Runs 20260909T010313Z-5abe (checkpoint 2250) and 20260909T010448Z-a6f1 (checkpoint 5999). The no-change cell at 0.575 m/s, μ=0.6, is 0% against 5%, and at n=100 that cell can only detect an 8 pp rise, so it is reported as no significant change and nothing more. The worst-powered cell in the grid is the 0.625, μ=1.0 cell at 19 pp.

3. We tried to move the boundary, and moved it the wrong way

The converged policy’s slope looked like the thing to keep and extend. The upstream repo ships a second task, Mjlab-VelStand-Flat-MicroDuck, which adds stumble recovery and getting back upright to the walking objective. We trained it with the same seed and iteration count as the baseline, took its converged checkpoint, and ran a paired diff against the baseline’s converged checkpoint: 14 cells, push from 0.600 to 0.750 m/s, both frictions, 100 paired rollouts per cell.

VelStand was significantly worse in 14 of 14 cells and safer in none. At 0.600 m/s its fall rate was 62% against the baseline’s 12%, in both friction rows. At the far edge of the grid, 0.750 m/s, it was 81% and 88% against 62% and 61%. The largest q across the 14 tests was 0.0054. The worst-powered cell could detect a 20 pp change. An omnibus permutation test over the whole grid rejects “no cell changed” at p = 0.00025.

We do not have a measured mechanism for why. The video shows one rollout each, and a plausible story about reward interference between standing-up and walking is a hypothesis, not a result of this sweep. What the sweep does establish is that the retrain moved the boundary inward by 19 to 57 pp across the whole tested range, and that a score on the VelStand task alone would not have shown it.

Runs 20260909T102454Z-1090 (baseline) and 20260909T102639Z-21a3 (VelStand), report.

4. Where it broke, along the training run

Both recipes save a checkpoint every 250 iterations, so the same grid can be run along the training trajectory. Nine baseline checkpoints and seven VelStand checkpoints, four push magnitudes from 0.45 to 0.675 m/s, two frictions, 100 rollouts per cell. Each grid below is one sweep at μ=1.0, fall rate per cell, checkpoints running upward from iteration 1000. The μ=0.6 twins are in the linked reports and tell the same story.

Both recipes pass through the same rigid checkpoint. Baseline 2250 falls 0% at every push up to 0.60 m/s and 100% at 0.675. VelStand 1500 and 2250 do the same with the cliff one step earlier: 0 or 1% at 0.45 and 0.525, 99 to 100% at 0.60 and above. Both recipes lose the cliff at the next checkpoint their grid sampled. Baseline 2500 falls 49% at 0.60 m/s; VelStand 3000 falls 69%.

Then they part. The baseline works its fall rate at 0.60 m/s down from 66% at iteration 3000 to 15% at 5999, and the slope in section 2 is what that looks like when resolved. VelStand goes to 82% at 4000, comes back to 58% at 5000 and ends at 63% at 5999. At 0.675 m/s it never gets below 65% after the cliff. The intervention did not fail at the end; it failed to recover from losing the rigid solution, and the sweep says when.

Runs 20260909T003230Z-16d4 (baseline, μ=1.0, μ=0.6) and 20260909T103415Z-e2cb (VelStand, μ=1.0, μ=0.6).

What it cost

Every rollout ran on one Apple Silicon laptop through Proofload’s local backend; nothing was sent to a cluster. The manifests record the platform as macOS on arm64 and do not record the chip. Wall time is summed from the per-episode times stored with each run.

Training ran on Google Colab Pro+ with an NVIDIA L4, about 4 h 45 min for the baseline. We did not record the dollar figure for either training run, so this post does not state one.

Envelope. Lateral push 0.45 to 0.75 m/s, floor friction 0.6 and 1.0, a 500-step (10 s) horizon at 50 Hz, 100 rollouts per cell, push step uniform over a 20-step stride window. Nothing outside that was tested. Sagittal pushes, torques, terrain and hardware are not covered by anything here.

Power. The worst-powered cell in any diff can detect a 20 pp change. Any effect smaller than that in a cell reported as no significant change is invisible to these runs, and the boundary table above prints the figure per cell.

What this does and doesn’t show

This is a map of frozen policies inside the simulator they were trained in. The boundary is per-policy and per-simulator. The push is a velocity impulse at one step, which is a convenient disturbance, not a physical one. We changed the rollout once, after the first paired run, and the runs before that change are kept in the store and reported here as what they are.

The tool ran the grid, kept every episode, computed survival per cell and the paired diff with its correction, and printed what it could not detect. Choosing the axes, reading the zero-variance table, and deciding that VelStand was a bad trade were ours.

Proofload is not released yet. We are looking for early design partners who train policies and want to know where they break, and the code is going open source. Write to us at poissonlabs.ai.

Built on Proofload, MuJoCo, mjlab, microduck_rl (Pollen Robotics / Hugging Face Microduck), and rsl_rl. The Microduck model and recipes are used under their respective licenses.