Agents that learn without rules

One pitch, two players, no rules given

For two agents to learn against each other, I needed a game that is fast to simulate. I wrote the engine myself, in NumPy, vectorized: 128 matches run in parallel, and 500 full matches play out in 0.4 seconds on a laptop. A pitch with FIFA proportions, players with bounded speed and acceleration, a ball slowed by friction, possession, a probabilistic tackle on contact and collisions between players.

The rules of the match are deliberately minimal:

• both players spawn anywhere on the pitch; each one scores in the opposite goal;
• a match lasts 20 seconds, after which it is a draw;
• a ball that leaves the pitch is given back to the opponent;
• the player carrying the ball runs slower, and a ball kicked faster than 12 m/s cannot be controlled;
• every tenth of a second, a player picks one of 16 running directions, and possibly a kick among 24 directions.

One policy for both sides

Each player sees the pitch in its own frame: player 2 sees it rotated by 180°. The same network therefore plays both sides, and every match provides two streams of experience.

The final reward is zero-sum: ±1 for a goal, 0 for a draw. Since scoring is very rare at first, I add a dense exploration reward — go to the ball, move it forward, keep it, avoid kicking it out — which decays linearly to zero at half of training. The second half optimizes nothing but "score or concede".

PPO, 128 matches × 64 steps per iteration, 4 epochs, with an action mask on the ball-related heads. The released agent is trained in two stages: 15 million steps with the dense reward (19 minutes), then 21 million in purely sparse self-play (29 minutes), that is 36 million steps in 48 minutes on a laptop without a GPU. The snapshot I kept is the one that beats the scripted player best, a choice confirmed on a second set of seeds.

73% wins against the scripted player

The agent clearly beats the scripted player and almost never loses to a random one. Against itself it sits at an exact balance (32.4% / 35.2% / 32.4%), which is the sign of healthy self-play.

To check that there is no cycle and no collapse, I had eight successive versions of the training play each other, 200 matches per pair: no older version beats a newer one.

Half of the work

This list stays visible: ruling ideas out with numbers is part of the job.

• Potential-based shaping. It is the form that provably does not change the optimal policy (Ng et al., 1999), but since a match always ends, the shaping sum is the same constant whatever the behaviour. The critic learns exactly that constant and the term vanishes from the advantage: 1% wins against the random player after 2.5 million steps, versus 20% with the annealed dense reward.
• A pool of past opponents, AlphaStar style: at equal budget it clearly loses to mirror self-play (score 0.38) and shakes more.
• Separate actor and critic networks: a draw in head-to-head play (0.505) for 60% more compute.
• CAPS smoothness regularization: no measurable gain in smoothness, and a real loss of strength (score 0.431).
• Top/bottom symmetry augmentation: entropy collapsed and the run was stopped.
• My first evaluations. The earlier version of this project pitted an attacker against a defender, and I measured everything on 12 matches from a fixed kick-off. Replayed over 500 matches with random starts, those evaluations were in fact measuring a single repeated match: the 0 / 100% oscillations I read as instability were mostly a measurement artefact.

Walk first, avoid obstacles later

Second playground: a four-legged creature, simulated in 2D with pymunk, that must reach a flag. Each leg has a motorized hip and knee, 8 motors in total. The agent receives 28 observations (torso orientation, velocities and height, angle and velocity of each joint, distance to the target, 5 distance sensors) and a reward combining progress toward the target, a survival bonus and penalties for tilt, energy and collisions; +200 on arrival, −10 for a fall. Training uses Stable-Baselines3 PPO with parallel environments.

Deterministic evaluation over 20 episodes. Walking is learned and holds up on uneven ground; obstacle avoidance is not solved yet, and it is the next step.