Where a 14-DoF Biped Falls

September 9, 2026

Method

Prove sweeps of frozen checkpoints under a lateral push: 100 rollouts per cell for each checkpoint on the same seeds in both, a survival estimate per cell, McNemar's test per cell with Benjamini–Hochberg correction across the grid, and the minimum detectable effect printed per cell.

Result

Two checkpoints of a Microduck walking policy under a lateral push: a cliff and a slope, with a crossover between 0.600 and 0.625 m/s. A retrain meant to widen the boundary was worse in 14 of 14 cells.

Could not detect

The worst-powered cell in any diff can detect a 20 pp change, so anything smaller in a cell reported as no significant change is invisible to these runs. Push 0.45 to 0.75 m/s and friction 0.6 and 1.0 in the simulator only: no sagittal pushes, torques, terrain or hardware.

Artifacts

report · receipt · retrain report

A benchmark score is one number. The boundary where a policy starts failing is a shape, and two policies with the same score can have very different shapes. We went looking for the shape and found that it was the whole story, along with a mistake of our own and a retrain that went backwards.

Microduck is Pollen Robotics’ 14-degree-of-freedom open-source biped. We trained a walking policy for it with the upstream mjlab recipe, saving a checkpoint every 250 iterations, and then used Prove to measure where each frozen checkpoint falls under a lateral push. The grid is push magnitude against floor friction, 100 rollouts in each cell for each checkpoint on the same seeds in both, a survival estimate per cell, and McNemar’s test per cell with Benjamini–Hochberg correction across the grid.

It all happened in one evening and the following morning, and it went wrong before it went right. The first paired run said an early checkpoint was the robust one. It wasn’t; the push was landing on the same control step every time, so the hundred seeds in each cell weren’t varying anything. With that fixed, the converged checkpoint was safer in 8 of 8 cells. When we resolved the boundary properly, the two checkpoints turned out to have different shapes, a cliff and a slope, with the ranking flipping between 0.600 and 0.625 m/s. And the retrain we did to widen the boundary made it worse in 14 of 14 cells.

How we measured it

Prove is the statistics and nothing else. We give it a sweep spec that names the axes and their values, and an import path to a rollout function of our own; the function gets one condition and one seed and hands back an outcome label and a time. The robot, the physics and the push all live inside that function, in our repo, and the tool never sees MuJoCo.

What it does with the function is the same regardless of what’s on the other end. It builds the grid and derives a seed for every condition and replicate from a run salt, which is what makes two runs comparable: give the second run the same salt and it gets the same seeds, cell for cell, so the comparison is paired without anyone having to arrange it. Every episode goes into a store with its config, seed, outcome, time and whatever metadata the rollout chose to attach, and nothing in the store is ever rewritten. For one run it computes the cumulative incidence of the event per cell. For two runs it computes the paired difference per cell, runs a McNemar exact test, corrects across the grid, works out the minimum detectable effect each cell had, and writes a report with the worst-powered cell printed right next to the verdict count. That last number is the one we kept coming back to.

None of this cares that the thing under test is a legged robot. The Go1 diff on the Prove page came out of a different rollout function, a different simulator and a different grid, swept before Prove existed and then reproduced with it, and an agent that returns succeeded or failed fits the same contract.

1. The seeds weren’t varying anything

The first paired run went into the store a little after eight on the evening of September 8. It compared checkpoint 1500 against the converged checkpoint 5999 on 8 cells, four push magnitudes from 0.45 to 0.675 m/s at two frictions, 100 rollouts each. Checkpoint 1500 fell in 0 of its 800 rollouts. Checkpoint 5999 fell in 400 of 800. Read as two totals, the early checkpoint was the robust one and the converged one had lost something on the way.

The per-cell table is what stopped us. Every one of the 8 cells was exactly 0.0 or exactly 100.0. Not 3%, not 97%. We had a 72-cell developmental sweep running at the same time, and when it finished it was the same: 72 cells, each 0 or 100. A hundred seeds per cell, and not one cell with any variance in it.

That isn’t what a hundred different seeds do, and it took about ten minutes to find where they’d gone. The push was applied at control step 100, two seconds into the episode, on every seed; the seed jittered the initial pose by a few millimetres and touched nothing else that mattered. A biped’s response to a lateral shove depends on where it is in its stride when the shove arrives, and with the push time fixed, every rollout in a cell was catching the robot at the same phase. Checkpoint 1500 happened to be at a good phase at step 100. That is all its 0 of 800 ever meant.

The fix was one line in our rollout, not in the tool: draw the push step uniformly from 100 to 119, a full 20-step stride window (2.00 to 2.38 s at the 50 Hz control rate), from the same seed Prove hands us. The stored episodes now carry the push step in their metadata, so anyone reading the store later can check.

The rerun on the same 8 cells was in the store twenty-four minutes after the first one, and it said the opposite. Checkpoint 5999 was significantly safer in 8 of 8, with fall rates from 2% to 40% against checkpoint 1500’s 20% to 74%. The largest q was 0.0027, and the worst-powered cell could detect an 18 to 20 pp change, so we don’t claim anything finer than that. Runs 20260909T003019Z-7ab3 and 20260909T003114Z-e81c, report.

To be clear about who found what: Prove ran the grid, kept every episode and printed the table, and the table made the problem hard to miss. Noticing it and working out why was a person’s job.

2. Two shapes and a crossover

With the phase fixed, the developmental sweep showed something the first run had hidden. Checkpoint 2250 fell in 0 of 300 rollouts at 0.45, 0.525 and 0.60 m/s, and then in 100 of 100 at 0.675. Checkpoint 5999 fell in 2 to 15% across the same three cells and 35 to 37% at 0.675. Nothing about those two rows looks like the same curve at different offsets. So about an hour after the first run we went back in and resolved the region between them: 2250 against 5999, push from 0.550 to 0.700 m/s in 0.025 m/s steps, both frictions, 100 rollouts per cell for each checkpoint on the same seeds in both, 2,800 rollouts in all.

At or below 0.600 m/s, checkpoint 2250 is nearly perfect, 2 falls in 600 rollouts, and checkpoint 5999 falls 5 to 16% of the time in the same cells. Prove calls 5999 significantly worse in five of those six. A benchmark run at 0.55 m/s would rank 2250 above 5999, and about that cell it would be right.

At 0.625 m/s the ranking flips. Checkpoint 2250 falls 52% and 58% of the time; checkpoint 5999 holds at 16% in both friction rows. By 0.650 m/s checkpoint 2250 is at 96% and 99%, and from 0.675 m/s up it falls every time, while 5999 stays between 25% and 45% out to the edge of the grid.

So there are two shapes here, not two scores. One is a cliff, almost nothing inside it and everything outside it, with the whole transition inside a 0.025 m/s interval. The other is a slope that starts earlier and never reaches the top within the range we tested. A single number at a single push can’t tell you which one you have, and which one you’d rather have depends on how sure you are about the disturbances the robot will actually meet.

The table below is Prove’s own, with its own columns and ordering, including the rows it calls worse and the one it won’t call either way.

Runs 20260909T010313Z-5abe (checkpoint 2250) and 20260909T010448Z-a6f1 (checkpoint 5999). The no-change cell at 0.575 m/s, μ=0.6, is 0% against 5%, and at n=100 that cell can only detect an 8 pp rise, so it’s reported as no significant change and nothing more. The worst-powered cell in the grid is the 0.625, μ=1.0 cell at 19 pp.

3. The retrain

The slope looked like the thing to keep and extend, so the next morning we tried. The upstream repo ships a second task, Mjlab-VelStand-Flat-MicroDuck, which adds stumble recovery and getting back upright to the walking objective. A policy that had learned to recover ought to hold a wider boundary; that was the whole idea. We trained it with the same seed and iteration count as the baseline, took its converged checkpoint, and diffed it against the baseline’s converged checkpoint on 14 cells, push from 0.600 to 0.750 m/s, both frictions, 100 rollouts per cell for each checkpoint on the same seeds in both, tested by McNemar’s test per cell.

It was worse everywhere. Significantly worse in 14 of 14 cells, safer in none. At 0.600 m/s its fall rate was 62% against the baseline’s 12%, in both friction rows. At the far edge of the grid, 0.750 m/s, it was 81% and 88% against 62% and 61%. The largest q across the 14 tests was 0.0054, the worst-powered cell could detect a 20 pp change, and an omnibus permutation test over the whole grid rejects “no cell changed” at p = 0.00025.

Why is the part we can’t tell you. There’s a plausible story about the standing-up reward interfering with walking, and it’s a hypothesis; nothing in this sweep tests it. What the sweep does show is that the retrain moved the boundary inward by 19 to 57 pp across the whole tested range, and that if we’d only looked at its score on the VelStand task we’d never have known.

Runs 20260909T102454Z-1090 (baseline) and 20260909T102639Z-21a3 (VelStand), report.

4. Where it broke, along the training run

Both recipes save a checkpoint every 250 iterations, so we ran the same grid along each training trajectory: nine baseline checkpoints and seven VelStand checkpoints, four push magnitudes from 0.45 to 0.675 m/s, two frictions, 100 rollouts per cell. Each grid below is one sweep at μ=1.0, fall rate per cell, checkpoints running upward from iteration 1000. The μ=0.6 twins are in the linked reports and say the same thing.

The two recipes look the same for a long time. Both pass through a rigid checkpoint. Baseline 2250 falls 0% at every push up to 0.60 m/s and 100% at 0.675. VelStand 1500 and 2250 do the same with the cliff one step earlier, 0 or 1% at 0.45 and 0.525 and 99 to 100% from 0.60 up. Both lose the cliff at the next checkpoint their grid sampled: baseline 2500 falls 49% at 0.60 m/s, VelStand 3000 falls 69%.

After that they part. The baseline works its fall rate at 0.60 m/s down from 66% at iteration 3000 to 15% at 5999, which is the slope from section 2 taking shape. VelStand goes to 82% at 4000, comes back to 58% at 5000, and ends at 63% at 5999; at 0.675 m/s it never gets below 65% again. The retrain didn’t go wrong at the end. It lost the rigid solution at the same point the baseline did and never found its way back, and the sweep can point to the checkpoint where that happened.

Runs 20260909T003230Z-16d4 (baseline, μ=1.0, μ=0.6) and 20260909T103415Z-e2cb (VelStand, μ=1.0, μ=0.6).

What it cost

Every rollout ran on one Apple Silicon laptop through Prove’s local backend. Nothing went to a cluster. The manifests record the platform as macOS on arm64 and don’t record the chip. Wall time is summed from the per-episode times stored with each run.

Training ran on Google Colab Pro+ with an NVIDIA L4, about 4 h 45 min for the baseline. We didn’t record the dollar cost of either training run, so there isn’t one here.

Envelope. Lateral push 0.45 to 0.75 m/s, floor friction 0.6 and 1.0, a 500-step (10 s) horizon at 50 Hz, 100 rollouts per cell, push step uniform over a 20-step stride window. Nothing outside that was tested: no sagittal pushes, no torques, no terrain, no hardware.

Power. The worst-powered cell in any diff can detect a 20 pp change. Anything smaller than that in a cell reported as no significant change is invisible to these runs. The boundary table above prints the figure per cell.

What this does and doesn’t show

This is a map of frozen policies inside the simulator they were trained in. The boundary belongs to this policy and this simulator and nothing else. The push is a velocity impulse at one step, which is convenient and not physical. We changed the rollout once, after the first paired run, and the runs from before that change are still in the store and reported here as what they are.

The tool ran the grid, kept every episode, computed survival per cell and the paired diff with its correction, and printed what it couldn’t detect. Choosing the axes, reading the zero-variance table and deciding that VelStand was a bad trade were ours.

Prove is open source and in private preview. If you train policies and want to know where they break before someone else finds out, talk to us.

Built on Prove, MuJoCo, mjlab, microduck_rl (Pollen Robotics / Hugging Face Microduck), and rsl_rl. The Microduck model and recipes are used under their respective licenses.