A browser sandbox that evolves a small neural controller for a 2-D physics task and lets you watch it learn live. This report maps the territory we charted: a recipe for tasks that train fast, the wall where they stop, sixteen working demos, the foils, and the theory under each one.
The loop is simple, the emergent behaviour isn't.
A setup defines a 2-D scene — a cart on a frictionless rail (position-controlled, so it never falls), optionally carrying a multi-link arm, a ball, or a target. A policy (a small multi-layer perceptron) reads an observation each physics step and outputs one or more actions (cart force, joint torques). An objective scores every step with a dense reward. A black-box optimizer — chiefly CMA-ES — evolves a population of policies against that reward, and the canvas plays the current champion as it improves.
The whole thing is deterministic and headless-runnable, which let us hold every demo to one rule: it must pass an automated training probe (random policy → solved, in <~15 generations) before it ships. That discipline is what produced a clean map instead of a pile of half-working toys.
Six properties. Hit all six and a random policy climbs to mastery in seconds of training. Miss one and you usually have a foil.
Nonzero everywhere, near-monotone improvement from a random policy. The optimizer always has a slope to climb.
The very first random policy already scores something climbable — the first target sits where the body starts.
Reward now reflects action now. No long delays between doing the right thing and being paid for it.
Solvable by reacting to the current observation — no anticipation, planning, or momentum-banking required.
A position-controlled cart never falls. Stable bodies self-recover; the policy refines, it doesn't rescue.
No well-paying local optimum that leads away from the goal. Potential-field shaping can turn even "go around" reactive.
Task complexity is not the wall. Instability is. Stable control of any flavour — multi-joint coordination, moving targets, obstacle routing, selection, sequencing — trains fast. The two things that break it are an unstable pose (you must balance/swing up) and an unshaped sparse reward (you only score on a rare event).
Sixteen demos, each gated on a headless probe. Grouped by the control problem they pose. Most also ship a "watch it learn" curriculum variant.
Shadow a drifting dot
chain_trace — 2-D closeness, collapses to 1 − |Δx|/REF.A chirp dot — find the speed wall
chain_trace · cartAccel 3200Stay inside a drifting band
zone_keep — full inside, ramps outside.Slide under a falling dot
catch_drop — x-only closeness · cartAccel 3200Hop to whichever post is lit
catch_drop reuseTouch the lit posts in a self-paced cycle
tag_sequence — closeness + a +1 tag bonus · cartAccel 3200Chase the closest of three dots
catch_drop against the nearest dotHold the midpoint of two dots
catch_drop on the computed midpoint (sharpened distScale)Reactive evasion — the inverse of chasing
flee_evade — the negation of catch_drop closeness, saturating at a clearance.Match the dot's speed, not its position
vel_match (reads a one-line velTarget kinematics field)Cart + elbow reach a target
chain_trace positive-closeness (2 actions)The arm follows a moving target
chain_traceRoute the hand around an obstacle
reach_avoid — closeness − soft repulsionHold a target while an obstacle drifts through
reach_avoid (zero new code)Thread the needle into a narrow gap
thread_slot — closeness − two capped barriersCarry a ball in a tray to a target — without spilling
plate_spin = ½·ball-centered + ½·cart-on-targetThe sandbox also carries a pre-existing menagerie — single/double/triple pendulums, a spring pendulum, golf & putt, a bullet-dodge scene, and the multi-link chain — which span the harder, sometimes-foil end of the spectrum (swing-up, sparse strikes). The sixteen above are the demos that train fast and immediately.
Four tasks resist reactive evolution. None of them because they're "too complex" — each violates a specific recipe property. The honest part: knowing whether it's a fundamental limit or just a shaping gap.
| Task | Why it fails | Nature | Path to making it work |
|---|---|---|---|
| balance-angle inverted pole tracks a swept lean |
Two problems at once: the pole is unstable, and a moving set-point dilutes the "stay upright" signal that a fixed target gets for free. | fundamental instability |
A curriculum that starts with a static target and a low-gravity bootstrap, then widens — the framework supports it, but it's no longer "immediately working". |
| chain-trace a passive whip traces a moving curve |
Requires anticipation & momentum exploitation — the tip is downstream of swinging links, so you must pre-load the swing. Reactive evolution stalls; the canonical zeroth-order foil. | fundamental anticipatory |
The differentiable simulator's exact gradient traces it cleanly — but that's a different optimizer, not reactive CMA-ES. A capability gap, honestly reported. |
| pong-volley a bat returns a pitched ball |
Sparse: a random gen-0 bat never even contacts the ball, so the reward is ≈0 everywhere with no gradient to climb. | shapeable sparsity |
Feed the predicted intercept point and reward shrinking distance to it every step — turns it into a catch-drop clone. Doable; just not free. |
| chain reacher (up-target) 4-link arm reaches a target overhead |
The target is up while the chain hangs down — reaching it is a multi-link swing-up to an unstable pose. Fails from scratch (fitness −18) and stalls even with a target-ease curriculum (level 3/15). | fundamental instability |
None that's cheap. But note: multi-DOF itself is fine — the shipped arm-reach proves it. The limiter is the unstable pose, not the action count. |
balance_up task trains reliably from a 30° start tilt (the trained policy holds upright ~90–100% of the time, run after run). From a 5° tilt the outcome is a lottery: across repeated runs, the best-of-3 upright fraction ranged from 22% to 74% — some runs stall farming the center/calm bonuses, some stumble into balancing. We first reported this as a clean single-run fact (9.86 vs 3.88 fitness); building an executable test for it (foils-probe.js) exposed the variance. The corrected claim is stronger, not weaker: near the unstable equilibrium the fitness landscape is flat and noisy, so the gradient doesn't just shrink — training degenerates into chance. A displaced start restores a strong, consistent "catch the falling pole" slope that CMA-ES locks onto every time.
No backprop on the reward. These are black-box optimizers searching directly in policy-parameter space. The sandbox offers a whole zoo; this project leaned on the first two.
| Method | What it is & why it fits |
|---|---|
| CMA-ES (workhorse) | Covariance Matrix Adaptation Evolution Strategy. Samples a population from a Gaussian, then adapts the full covariance toward the directions that improved fitness — it learns the local curvature of the reward landscape. Smooth, sample-efficient, and ideal for a smooth low-dimensional MLP. Solved every demo here in tens of generations. |
| sep-CMA-ES | CMA-ES with a diagonal covariance — O(n) state instead of O(n²), so it scales to larger networks at the cost of ignoring cross-parameter correlations. |
| NEAT / NEAT-full | NeuroEvolution of Augmenting Topologies — evolves the network structure (nodes and connections) alongside the weights, growing complexity from a minimal seed. Useful exactly where the policy-class shape matters, e.g. the antisymmetric reflex a swing-up needs. |
| The rest of the zoo | Differential Evolution, PSO, a flock of swarm metaheuristics (cuckoo, whale, fish, ant-colony, firefly, bat), simulated annealing, parallel tempering, CEM, finite-difference & SPSA gradient estimators, NES, random search — all available for head-to-head comparison on identical tasks. |
| Differentiable sim | A separate autodiff (Adam) copy of the world for exact gradient probes. It's what shows chain-trace is solvable in principle — the foil is reactive-evolution-specific, not problem-specific. |
| Policy classes | A small MLP throughout, but CNNs over a rendered phase-space image and NEAT topologies are first-class options. |
The suite trains most tasks blind — black-box search that needs only a reward. But most of these tasks have a known, smooth model, and when you use it the results change completely. This is the axis that really separates the techniques.
CMA-ES, NEAT, DE, PSO… ask for nothing but a scalar reward — no model, no smoothness. Maximally general (works on discrete, contact-rich, unknown-dynamics tasks), but weaker: it stumbles, and on hard unstable tasks it plateaus. The honest slogan: "anything can be optimized, even blind — just not always well."
When you have the analytical model (the equations of motion), you backprop the reward through the simulated trajectory and use the exact gradient (the adjoint method). This is the traditional path — control theory, differentiable simulation. Fast, reliable, and it cracks tasks blind search can't. The cost: you need the formula, and it must be smooth.
double-balance-probe.js). Exact gradients through the differentiable simulator (Adam) reach 100% hold with a robust ±12° basin (double-balance-adam.js), and a single pose-conditioned controller holds all four equilibria on command — both-up, up/down, down/up (a pendulum on a hanging link), both-down — at 0.94–1.00 (double-poses-adam.js). Reach is the same story: the grad-free arm lands ~35px; the differentiable one reaches 4.7px (arm-reach-adam.js). The trick that makes it work: train in the near-linear regime (short horizon) so backprop-through-chaos stays clean.
A task is exact-gradient-trainable iff the loss is a smooth function of the policy parameters through the dynamics — smooth model, no hard events.
| Regime | Setups | Differentiable? | Why |
|---|---|---|---|
| Smooth ODE control | pendulums (single/double/triple/spring/twin), chains, all arm-*, cart tracking (rail-catcher, catch-drop, cruise, accel, safe-zone, midpoint, flee), plate-spin | ✅ fully | mass-spring ODE is smooth; even the obstacle repulsion is a smooth potential field |
| Contact / collision | ball-single, golf, putt, pong, dodge | ⚠ soft-only | impact flips velocity discontinuously → needs a soft contact model (approximate); dodge's discrete random spawns break the tape |
| Discrete / selection | prioritize-nearest (argmin), tag-sequence (events), pick-side (square-wave flip) | ❌ no | reward is piecewise-constant → zero/undefined gradient; grad-free only |
| Technique | Needs | Meaning |
|---|---|---|
| Differentiable physics (diff-sim + Adam) | the equations of motion | backprop through the simulated trajectory (adjoint) |
| LQR / optimal control | the linearized model | solve Riccati analytically |
| Differentiable rendering / optics | the light-transport / projection formula | backprop through image formation |
| Grad-free (CMA-ES, NEAT, DE, PSO…) | nothing but a scalar reward | blind black-box search |
| Gradient-estimate (SPSA, FD-GD, NES, "adam") | reward only, but many probes | finite-difference the gradient — the middle ground |
One-liner: differentiable needs the formula (physics / optics); grad-free needs only a score. So we frame the differentiable versions as the baseline — the traditional, stronger tool — and grad-free as the general fallback that proves you can learn even without the model.
The CNN presets underperform because the hand-crafted Top-K observation already hands the MLP the relevant state. A CNN's edge is a spatial, translation-invariant prior — it only pays off on a raw field observation. The honest demo (planned): pixel-dodge on a raw occupancy grid, where CNN ≫ MLP-on-pixels but both < MLP+TopK. The lesson: CNN is for when you must see raw space and can't hand-craft features.
Rigid-unravel puzzles are contact-dominated + combinatorial (which piece, which order) — a planning/search showcase, not a differentiable-physics one, and they'd reinforce the grad-free side we're trying to balance. Sokoban-style discrete solution-space viz doesn't fit the continuous focus either. Both are parked as stretch goals.
A curriculum ramps one difficulty knob from easy to hard across levels; a fitness gate advances when the champion clears thresholdFrac × ceiling for two generations running. Every demo above ships a "watch it learn" variant.
Each demo exposes a difficulty parameter — rail-catcher's drift amplitude (30→150), the safe-zone band width (120→55), the obstacle radius (3→35), the bowl stiffness (deep→flatter). The level climbs 0→15 as the policy masters each rung, so you literally watch the task get harder while the controller keeps up.
Set the advance threshold just below each demo's achievable cap. Too low and it breezes 2 generations per level (no visible struggle); too high and it grinds. The right setting produces a back-loaded arc — easy levels fly by, the hard ones genuinely take work.
Most "it won't learn" moments weren't the optimizer's fault — they were a reward that quietly paid for the wrong thing. Four worth remembering.
| Demo | The deception | The fix |
|---|---|---|
| tag-sequence | With pure closeness, touching the lit post relocates it — which drops the closeness reward. So the greedy policy learned to hover just outside the touch radius forever (high reward, zero tags). | A one-shot +1 bonus on each tag, so advancing the cycle outweighs the momentary reward dip. 0 tags → 16. |
| arm-slot | Two wall-barriers strong enough to keep the tip out of the walls were also strong enough to push it back out of the slot — the funnel inverted (reaches → 58px, stalls). | Keep the target outside the barrier zone (clean reach) and let the barrier bite only very near a wall. A monotone funnel in. |
| arm-reach | The reach reward's sparse "+1 inside the radius" bonus barely fires when the hand hovers at the radius edge — so fitness stayed ~0.8 and the curriculum gate starved (stalled at level 8). | Switch to a positive-closeness reward that reflects reach quality, so the gate sees real progress. |
| plate-spin | A flat tray turns "keep the ball centered" into balancing an inverted pendulum — the foil, hidden inside a friendly-looking demo. | Ship a concave bowl only: gravity self-centers the ball, making centered a stable minimum. |
Every claim in this report is backed by a script that runs without a browser.
foils-probe.js asserts the boundary claims themselves — pong-volley must stay sparse, the up-target reacher must stay unreachable — so a silent change that invalidates this report breaks the build. Building it also corrected one of our own claims (the balance-lottery datum above).Every number below regenerates from a committed probe (robustness-probe.js, before-after-probe.js, optimizer-bakeoff-probe.js, foils-probe.js). Run them with node <probe>.js — no browser required.
Result: 16/16 demos robust at 5/5. The per-run metric samples show the honest spread (the trainer has unseeded RNG, so each run is an independent sample).
| Demo | Runs | Per-run metric | Demo | Runs | Per-run metric |
|---|---|---|---|---|---|
| rail-catcher | 5/5 | 6, 4, 6, 3, 5 px | prioritize-nearest | 5/5 | 9, 8, 9, 12, 10 px |
| catch-drop | 5/5 | 80–100% caught | arm-reach | 5/5 | 29–35 px |
| pick-side | 5/5 | 65–74% on-post | arm-track | 5/5 | 25–35 px |
| flee-the-dot | 5/5 | 241 px clearance ×5 | arm-obstacle | 5/5 | 27–52 px, 0% inside |
| cruise-control | 5/5 | 22–23 px/s | arm-dodge | 5/5 | 43–68 px, 0% inside |
| accelerating-track | 5/5 | 3–7 px (head) | arm-slot | 5/5 | 19–31 px, 0% contact |
| safe-zone | 5/5 | 100% in-band ×5 | plate-spin | 5/5 | 8–13 px, no spill |
| midpoint-meet | 5/5 | 3–7 px | tag-sequence | 5/5 | 16–17 tags / 8 s |
The reward integral of the untrained gen-0 genome vs. the champion, on identical deterministic rollouts. 16/16 champions win. Small ratios are the demos whose dense reward gives gen-0 a high floor by design (the "gen-0 rigged" recipe property) — the champion still separates cleanly.
| Demo | gen-0 | champion | × | Demo | gen-0 | champion | × |
|---|---|---|---|---|---|---|---|
| rail-catcher | 1.21 | 7.88 | 6.5 | tag-sequence | 2.41 | 18.65 | 7.7 |
| catch-drop | 2.10 | 6.37 | 3.0 | prioritize-nearest | 4.97 | 7.63 | 1.5 |
| pick-side | 1.93 | 6.50 | 3.4 | arm-reach | 0.10 | 5.92 | 56.8 |
| flee-the-dot | 3.39 | 7.56 | 2.2 | arm-track | 0.48 | 6.43 | 13.3 |
| cruise-control | 4.77 | 7.55 | 1.6 | arm-obstacle | 2.43 | 5.78 | 2.4 |
| accelerating-track | 1.59 | 7.02 | 4.4 | arm-dodge | 0.23 | 4.66 | 20.0 |
| safe-zone | 1.40 | 8.00 | 5.7 | arm-slot | 0.50 | 6.11 | 12.2 |
| midpoint-meet | 0.12 | 7.62 | 65.8 | plate-spin | 4.65 | 7.11 | 1.5 |
One easy task (rail-catcher, 30 gens × pop 30) and one hard one (arm-obstacle, 60 gens × pop 36); lower is better. The headline is honest and a little humbling: on a recipe-clean task, even random search clears the pass bar — the recipe does the feasibility work; the optimizer buys precision and consistency. CMA-ES is near-best on both, untuned — that's why it's the workhorse.
| Optimizer | rail-catcher (miss px) | arm-obstacle (tip px, % inside) | Note |
|---|---|---|---|
| CMA-ES | 3 | 35 (0%) | near-best on both, no tuning — the workhorse |
| sep-CMA-ES | 3 | 35 (0%) | matches CMA-ES here; scales to bigger nets |
| NEAT-full | 4 | 52 (0%) | topology growth costs budget on fixed-shape tasks |
| DE | 5 | 50 (0%) | solid on both |
| PSO | 3 | 51 (0%) | excellent easy-task precision |
| NES | 59 | 66 (0%) | the one laggard — needs step-size tuning |
| Annealing | 15 | 26 (0%) | sniped the best hard-task score this seed |
| Random search | 23 | 40 (0%) | clears both pass bars(!) — quantifies the recipe |
Three ideas survived vetting but aren't shipped, each blocked on a known, named thing rather than a mystery.
gen-0 random vs. trained champion, side by side
app.js can't run in Node, so it's the one thing not headless-verifiable. Batched for a live pass.Drive the target with your mouse; the trained brain follows live
Two carts split a falling stream — emergent division of labour
step(world, dt, cmd) drives one w.cart. Two carts means real physics work, not just a new setup.One more boundary worth recording: the adversarial machinery in the engine co-evolves environment knobs (gravity, push strength), not a second neural policy — so AI-vs-AI duels are net-new infrastructure, and were correctly rejected as out of reach.