Teaching frogs to hop without telling them how

1. Results at a glance

Distances are for the frog's torso over one 10-second episode. Scores depend on the scoring rule in force, so scores from different experiments are not comparable; distances and airborne fractions are.

2. The setup

2.1 Body

The frog is seven rigid pieces: a torso (with a head and a stiff, fixed front leg), plus two back legs each made of a thigh, a shin and a foot. Six revolute joints (hip, knee and ankle on each back leg) connect them. Each joint has a position-target motor with a strength limit. Nothing is animated: the motors push, and gravity, friction, momentum and collisions decide what happens.

2.2 Senses

The brain receives 21 inputs, each scaled to roughly −1 to 1:

• the angle and angular speed of each of the six joints (12 inputs);
• body tilt and spin;
• forward and vertical speed;
• height above the ground;
• whether each back foot is touching the ground;
• a clock: the sine and cosine of a 1.5 Hz tick, which gives the brain a built-in rhythm to work with.

2.3 Brain

A fully connected network of 21 inputs, two hidden layers of 16 neurons with tanh activation, and 6 outputs: 726 weights and biases in total. Each output picks a target angle within its joint's range, and that joint's motor drives towards it. Choosing target poses is far easier to evolve than choosing raw forces. Evolution only ever sees the 726 numbers as a flat list; it never knows they form a network.

2.4 Physics

planck.js 1.4.2, a port of the Box2D engine, with a fixed step of 1/60 s, 8 velocity and 3 position iterations, and gravity of −10 m/s².

2.5 Episode

One episode is 600 steps (10 s). It ends early if the torso or head touches the ground; we call that a fall. In E5 it also ends early if the frog stalls (§3.2). One episode takes about 16 ms of computer time.

3. Method

3.1 Genetic algorithm

Each generation, every brain drives a frog on the same three courses and scores the average. Then:

3.2 Scoring rules

The score is the only thing evolution sees, so it is where the real design work happens. We used four scoring rules over the course of the project, and for E5 added a stall rule to rule A:

• Distance only (E1):
  score = distance − 0.002 × motor work (J)A fall ends the episode but costs nothing extra.
• A, scale by survival (E2, E3, E5; the default):
  score = distance × (seconds upright / 10) − 0.002 × motor work
• B, flat fall penalty (E2):
  score = distance − 10 (if fell) − 0.002 × motor work
• Airborne only (E4): as A, but only forward distance covered while no part of the frog touches the ground counts.
• Stall rule (E5, added to A): the episode also ends if the torso has not moved at least 0.5 m forward in the last 2 s. A stalled episode is scaled by survival exactly like a fall, so standing still after a lunge costs the survival multiplier:
  score = distance × (seconds before a fall, a stall or the end / 10) − 0.002 × motor work

3.3 Courses

Courses are generated from a seed, so only the seed needs storing and the browser can rebuild the identical ground. We keep four kinds apart, following the same discipline as our image classifier (select on validation, report on test):

4. E0: probes before evolution

Before evolving anything, we checked the body and physics with simple controllers.

• Open-loop kicking always rotates the body in flight. Staying upright needs feedback from the tilt and spin senses.
• With distance as the only score, one uncontrolled leap beats any cautious gait. We expected reward hacking before training started.
• Joint torque was reduced to 30/30/15 Nm (hip/knee/ankle) for all later experiments.

5. E1: the reward hack

A 30-generation smoke run with the distance-only score (§3.2).

The population fall rate was 99–100% in every generation.

Finding. Evolution optimised the score and ignored the intent. It produced ever-longer dives followed by a crash: the reward hack predicted in E0. Diving is not a failure of the algorithm. It is the correct answer to the question we asked.

6. E2: fixing the score

Two cures, 100 generations each, seed 1, both keeping the energy term: A scales distance by the fraction of the episode survived; B subtracts a flat 10 for falling (§3.2).

Fell: share of all the population's episodes that ended in a fall. Holdout: the champion's score on the 5 holdout courses. Champion: average over that generation's 3 training courses. Scores under A and B are not directly comparable: A scales distance, B subtracts a penalty.

• Both cures stopped the suicidal dive within about 10 generations. Champions then stayed upright for the full 10 s (standing still for most of it, as §7 explains).
• Both plateaued at about 7–8 m per 10 s.
• B was noisier. Its median stayed negative until about generation 40, and its holdout score was unstable (1.37 at generation 85 while the training best was 6.84).

Decision. A (scale by survival) is the default for everything that follows. It is simpler to explain and gave a smoother signal.

7. Gait analysis (corrected)

With falling fixed, we measured how the champions move on the flat showcase course: the share of time with no part of the frog touching the ground (airborne), the number of separate flights, and the share of time the stiff front leg is on the ground.

Original finding (superseded): second reward hack. The champions mostly scoot on the fixed front leg with occasional short hops. Nothing in the score says "hop", and scooting is a stable way to cover distance without falling.

The exhibits in this paper are stills from the recordings used by the site. Each still is centred on the frog with the ground fixed, so height above the ground is to scale; gridlines are 1 m apart. Unless a caption says otherwise, stills are taken every 0.4 s for the first 2.4 s, followed by the final recorded frame. A red border on the final still marks a recording that ended early (a fall or a stall); a green border, one that lasted the full 10 s.

8. E3: the main run

300 generations with rule A, seed 1. The run took about 7 minutes. Its selected champion, generation 277, was the one the previous preview of this site shipped.

8.1 The winner's curse

• From about generation 120, the champion's holdout score often dropped far below its training score: by more than 3 points in 57 of 300 generations.
• Each generation is scored on only 3 courses, so the top scorer is often the individual that was luckiest on those 3, not the most capable.
• The population median kept improving steadily (−0.29 to 6.77), which shows the population as a whole got better even though individual champions were noisy.

8.2 Model selection

The last generation's champion is not necessarily the best one to ship. The exporter scores the champions of the last 50 generations on the 10 validation courses (seeds 2001–2010) and ships the best. The holdout courses (1001–1005) are never used for selection, so the shipped champion's holdout score is still an unbiased estimate. The same procedure is used for E4 and E5.

9. Two behaviours, and lunge and freeze

9.1 Two behaviours, as first measured

Late E3 champions switch between two behaviours depending on small differences in the terrain. Each cell is one 10-second episode: distance, time survived, and share of time airborne.

• The safe behaviour is about 17% airborne, reaches about 9.4 m and does not fall. We first called it a "scoot"; see §9.2.
• The hop is about 60% airborne. It covers about 10 m in under 3 s, then crashes.

Original interpretation (superseded). Evolution has found real hopping but cannot yet keep it stable, and the survival-scaled score rewards the safe scoot about as much as a short, fast hop.

9.2 Correction: lunge and freeze

Distance at each whole second on the flat course (seed 0) and holdout course 1001:

• Under the survival-scaled score this is optimal: distance is banked early, and standing still keeps the full survival multiplier.
• This is the third reward hack, and the same behaviour we had misreported as the second.
• The "hop" is the same lunge carried on too far, which ends in a fall.
• E5 (§11) addresses this directly.

9.3 A note on the showcase

The course was never changed. The flat showcase course (seed 0) was fixed before any of these results were seen. We did not change it to one where a champion looks better. On it, the E3 selected champion hops for 2.8 s and falls, and the previous preview said so on its front page. The E5 champion (§11) is shown on the same course.

10. E4: airborne-only scoring Complete · corrected

E4 was designed while we still believed the champions scooted. It counts only forward distance covered while no part of the frog touches the ground, still scaled by survival: 150 generations, seed 1. The aim was to stop rewarding scooting.

Scores count airborne distance only, so they are not comparable with E2 or E3 scores. Distances and airborne fractions are comparable.

10.1 Selection

Using the same procedure as E3, generation 120 was chosen, with validation 6.00 and holdout 5.88. The latest champion (generation 149) had validation 4.26 and holdout 4.36.

10.2 Findings

• Changing the score did not change the behaviour. Champions stayed around 16–30% airborne, much as in E3. We first reported this as a "skip gait"; see the correction below.
• The fast hop appeared again and still was not stable. It reached 55–80% airborne, then crashed within 2.5–5.1 s. Scoring airtime more directly did not make it stable.
• The selected champion (gen 120) is the most robust one tested. It completed all 10 s on the flat course and on all 5 holdout courses.

Original interpretation (superseded by E5). The limiting factor is not what the score rewards but whether the fast hop can be controlled at all.

E4 was never shipped, and no E4 recordings were exported, so there are no E4 exhibits.

11. E5: the stall rule Shipped

Rule: the E2-A score (distance × fraction of episode survived, minus the energy term) plus a stall rule. The episode ends if the torso has not moved at least 0.5 m forward in the last 2 s. A stalled episode is scaled by survival exactly like a fall, so standing still after a lunge now costs the survival multiplier. 300 generations, seed 1.

Why: E3 and E4 champions lunged and then froze (§9.2). The stall rule makes continuous forward progress the only way to keep the multiplier.

Champion columns are averages over that generation's 3 training courses. Population fell / stalled: share of all the population's episodes that ended that way.

11.1 Selection

Holdout over the last 50 generations: min 11.66, mean 20.04, max 23.97. Selection on the validation courses (seeds 2001–2010, last 50 generations) chose generation 298.

The latest champion scored slightly higher on holdout. Holdout is never used for selection, so that does not change the choice; the selected champion's holdout (23.04) is close to its validation (24.12).

11.2 The selected champion, second by second

Measured with the stall rule switched off, to show what the frog does when nothing stops it:

11.3 Recorded champions on the flat showcase course

With the stall rule on, as shipped on the site's ghost race:

11.4 Findings

• The stall rule produced a continuous hopping gait. The selected champion moves about 2.5 m/s steadily for the full 10 s, is about 35% airborne, and completed all 10 s on the flat course and on all 5 holdout courses. It is still moving at 10 s.
• The winner's curse is still present (holdout min 11.66 over the last 50 generations), so validation-based selection remains necessary.
• The E3 and E4 limit was the score, not control. Once standing still stopped paying, evolution found a stable repeated hop within about 150–200 generations.

Decision. E5 generation 298 replaces E3 generation 277 as the published champion.

12. Limitations and future work

12.1 Threats to validity

• One seed per experiment, one run each. E2 to E5 all used seed 1, and no repeat runs with other seeds are reported. E5 ran once. How much outcomes vary between runs is unmeasured, so differences such as A against B, or E5 against E3, could partly be luck.
• The stall thresholds were not tuned. 2 s and 0.5 m were chosen once and never varied. We do not know how sensitive the result is to them.
• Three courses per generation. Selection during evolution is noisy by design, which is what produces the winner's curse (§8.1). It is still present in E5.
• Small evaluation sets. The holdout estimate rests on 5 courses and the validation choice on 10, and selection only considered the last 50 generations. Validation and holdout disagree noticeably (for E3 gen 299, 4.36 against 6.08), which is a reminder of how noisy a handful of courses is.
• Summary gait metrics misled us. Airborne share, flights and front-leg contact are totals over an episode, and they did not reveal the lunge and freeze. Distance over time did. Other summaries in this paper may hide behaviour in the same way.
• Gait figures are single episodes. Airborne share and distances by second come from one episode per champion per course. The flat showcase course alone decides what the site shows.
• Two dimensions, one body. The frog is a 2D body with a stiff front leg and a fixed 1.5 Hz clock. Results may not carry over to other bodies, to 3D, or to real animals.
• Energy term untested. The 0.002-per-joule energy cost was kept throughout and never varied or removed, so we do not know how much it shapes the gaits.
• Scores are rule-specific. Scores under A, B, airborne-only scoring and the stall rule measure different things. Only distances and airborne fractions compare across experiments.
• E1 was short. The distance-only run was a 30-generation smoke run; we did not test whether divers persist over longer runs.

12.2 Candidate next steps (not yet tested)

• Repeat E5 with other seeds.
• Vary the stall thresholds.
• More courses per generation, to reduce the winner's curse.
• Longer runs.
• An articulated front leg that can absorb landings.
• An evolvable clock frequency.
• A curriculum from short hops to long ones.

13. Reproducibility

• Deterministic. Every random choice in training comes from one seeded random number generator, and the physics is deterministic. The same configuration and seed produce the same frogs.
• Seeded courses. Every course is generated from its seed; the seeds used for validation, holdout and showcase are listed in §3.3.
• Resumable. Training can be stopped at any time. It finishes the current generation and saves a checkpoint holding the population, the random number generator's exact state and the generation number. The checkpoint is written to a temporary file and then renamed, so a crash mid-save cannot corrupt it.
• Tested. The project's test suite checks that a run stopped and resumed produces exactly the same frogs as one that ran straight through.
• Checked in the browser. The live panel on the front page re-runs a champion's brain in the same physics code, under the same rules including the stall rule, and compares every frame with the training recording.
• Figures from data. Every exhibit in this paper is drawn in your browser from the exported run data (configuration, score history, champion genomes and recordings) of E5 and, for the historical exhibits, E3. None are hand-made images.