Pokémon TCG: training and evaluating agents

Pokémon TCG is a card game with hidden information and long chains of small decisions: which card to play, which attack to use, which cards to choose for an effect. In these experiments the learner makes about 70 recorded decisions per game. Metamon learns from a large archive of human games, but this Kaggle competition came with no large training dataset. I built a workflow around the supplied game simulator to train policies by imitating scripted players and by reinforcement learning, then to evaluate them against fixed scripted opponents.

Supplied

Competition simulator and SDK

The compiled game engine and its Python interface, used as supplied.
Built for this project

Gymnasium wrapper

Turns each decision into a choice among legal options, with a mask. A multi-card selection becomes a sequence of picks that ends with STOP.
Built for this project

PyTorch policy

Embeds card and attack identities and scores each legal option against the board, hand, open menu and cards already picked.
Built for this project

Behavior cloning

Learns from games played between scripted teachers, with training and holdout sets split by game.

Teachers: four imported rule-based agents (supplied)
Built for this project

Weight transfer

Loads the cloned network into a custom Stable-Baselines3 policy and checks that its outputs are unchanged.

Library: Stable-Baselines3 (supplied)
Built for this project

PPO fine-tuning

Trains against mixtures of opponents. Frozen self-play snapshots are supported.

Library: MaskablePPO from sb3-contrib (supplied)
Built for this project

Seat-swapped evaluation

Plays each deck against each scripted opponent from both seats and reports 95% intervals.
Built for this project

Recorded-game inspection

Records every decision of a game, with the options, the choice and the policy’s probabilities, to step through afterwards.

The workflow as implemented. Dashed outlines mark supplied foundations, and solid outlines mark what I built. Behavior cloning and PPO train the same policy network.

My part, briefly

Clone first. PPO with the stock policy had plateaued at about a 13% win rate, so I cloned the scripted teachers first and fine-tuned the clone to 24.0%. A PPO-only run with my own network, measured alongside, reached 13.9%, so the network alone made little difference.
The first fine-tuning run. It slowly lost what the clone knew: across checkpoints the Iono mirror fell from 11.0% to 2.5%. My diagnosis was the entropy bonus pushing a sharply peaked clone back toward uniform, with a critic that started out predicting almost nothing. The recipe that worked dropped the bonus and lowered the learning rate and target KL, all at once, so it doesn’t isolate the cause.
Two checks. Before fine-tuning, the training script stops unless the transferred policy gives bit-identical logits and value to the clone on the same observation. When the specialist reached 82.6%, I checked how its games ended: every win was a deck-out, in a 100-game audit and again across all 678 wins of an 800-game re-run.

What I built

A Gymnasium environment around the simulator. Each decision becomes a choice among legal options with a mask, the opponent’s turns run inside the environment, and the reward arrives only when the game ends.
Multi-card selections split into a sequence of masked picks closed by a STOP action, so training and evaluation handle them the same way.
A PyTorch policy that embeds card and attack identities and scores every legal option against the board, the hand, the open menu and the cards already picked.
Behavior cloning from games between four imported scripted teachers, with whole games held out for validation. The cloned weights then move into a custom Stable-Baselines3 policy for PPO fine-tuning against mixtures of opponents.
Seat-swapped evaluation of each deck against each scripted opponent, and a recorder that saves every decision of a game for inspection.

Environment and action handling

The environment wraps the battle loop in reset() and step(). It plays the opponent’s turns itself and resolves decisions the learner does not make.
Options are padded to 128 slots. The legal-action mask is exposed both in the observation and through action_masks(), the interface MaskablePPO expects.
A multi-select decision becomes a series of masked single picks. A reserved STOP slot commits the chosen set, and the selection commits automatically once it reaches the maximum count.
The reward is +1 for a win, −1 for a loss and 0 for a draw, given only at the end of the game.

Policy network

The core network is plain PyTorch, independent of Stable-Baselines3. It keeps one learned embedding per card identity (32 dimensions) and per attack identity (16).
Each option becomes a latent vector. The context combines the game state, embeddings for the 12 board slots, pooled hand and discard embeddings, and averages over the menu options and the options already picked, so each pick of a multi-select is conditioned on the earlier ones.
One shared scorer turns every option into a logit, and a separate head estimates the value of the position.

Learning from four scripted teachers

The first question was how much of the scripted players’ behavior the policy could learn. Each model played mirror matches, where both sides use the same deck, against each of the four teachers.

Behavior cloning helped more than the custom policy Pooled win rate, 800 seat-swapped mirror games per entry against four fixed scripted teachers (reported evaluation, July 2026)

Reported pooled mirror win rate against four fixed scripted teachers (Lucario, Abomasnow, Dragapult ex and Iono decks), 800 seat-swapped games per entry. A perfect copy of the teachers would score about 50%.
Model	Win rate	95% interval	Note
Earlier stock-policy RL baseline	about 13%	11.1–15.8%	Approximate, from the 8 July report
Custom policy, PPO only	13.9%	11.6–16.4%	From the 9 July report
Behavior cloning	20.0%	17.4–22.9%	From the 9 July report
Behavior cloning, then tuned PPO	24.0%	21.1–27.1%	From the 9 July report

Whiskers show 95% Wilson intervals. In a mirror game both sides play the same deck. The teachers play Lucario, Abomasnow, Dragapult ex and Iono decks, and a perfect copy of them would score about 50%. The baseline is an earlier policy stack, reported approximately on 8 July, and the other three entries are from the 9 July report.

PPO alone reached 13.9%, close to the earlier baseline of about 13%. Behavior cloning reached 20.0%, and tuning PPO after cloning reached 24.0%.

Every model remained below parity with the teachers. The clone matched the teacher’s move on 79.6% of held-out decisions, leaving room for errors to accumulate over a game.

Cloning data, transfer and PPO settings

The cloning data came from 2,000 games between the teachers, covering all 16 ordered deck pairings with seats swapped: 286,035 recorded decisions. Games were split into training and holdout sets by game, so no game appears in both.
Holdout accuracy was 0.796 (0.803 for single picks, 0.765 for multi-select), against 0.217 for choosing a random legal option.
After the cloned weights load into the Stable-Baselines3 policy, the training script feeds one observation through the cloning code’s input path and through Stable-Baselines3’s own, and stops unless the logits and the value are exactly equal (torch.equal, no tolerance). Every fine-tuning run logged the check as passing.
The first fine-tuning recipe (entropy bonus 0.01, learning rate 5e-5, target KL 0.02) didn’t beat the clone. On checkpoints evaluated every 25,600 steps the pooled rate stayed between 17% and 19%, and the Iono mirror fell from 11.0% to 2.5%. The recipe that worked used no entropy bonus, a learning rate of 3e-5 and a target KL of 0.01. It climbed from 17% to 24% over 153,600 steps and then plateaued. The Dragapult ex mirror rose from 7.0% after cloning to 22.5% (95% interval 17.3–28.8), the first per-deck gain from PPO in these experiments that was separated from the clone by its interval.
The 9 July entries were evaluated over 200 seat-swapped games per deck and opponent, with the policy sampling its actions. The pooled figure combines the four mirror pairings.

A specialist for one opponent

Next I narrowed the task to Lucario against the scripted Lucario teacher. A specialist clone followed by PPO reached a reported 82.6% over 800 games. The teacher’s games against itself were balanced, making 50% a useful reference for this matchup.

One opponent: a high win rate, and every audited win a deck-out Learner and scripted teacher both play Lucario. 800 seat-swapped games against that one fixed opponent (reported evaluation, 10 July 2026)

Win rate over 800 games

661 won, 139 lost, 0 drawn (95% interval 79.8–85.1%)

How the wins ended, in a separate 100-game recorded audit

82 won because the opponent ran out of cards to draw (deck-out)
0 won any other way, such as by taking prize cards
18 not won

Lucario specialist against one fixed Lucario teacher: reported results
Games in the evaluation	800
Won / lost / drawn	661 / 139 / 0
Win rate	82.6% (95% interval 79.8–85.1%), and parity with the teacher is 50%
Games in the separate audit	100
Audit wins by opponent deck-out	82 of 82
Audit wins any other way	0

The audit games are sorted by outcome, not shown in the order played.

How the specialist won

I recorded a separate 100-game audit to examine how the games ended. All 82 model wins came from the opponent running out of cards to draw, and the simulator’s own end-of-game reason and a count of the cards left agreed in every game. A later 800-game re-run found the same in all 678 of its wins, and in almost all of them the model was behind on prizes when it won. Deck-out is a legal win, so this isn’t a scoring bug: the model learned one fixed opponent’s habits, and the 82.6% alone doesn’t show how it plays against anyone else.

The specialist's training stages

Stage (Lucario mirror only)	Win rate	Games
Clone trained on all four decks	22.8%	400
Lucario-only clone (4,000 games, 537,922 decisions)	38.8%	400
Plus PPO, cautious recipe	plateau at 45.9–46.8%	400
Plus PPO, learning rate 1e-4 and target KL 0.02	84.0%	200
Final checkpoint	82.6% (79.8–85.1)	800

The Lucario-only clone matched the teacher’s choice 0.934 of the time on held-out games. Choosing the most likely action instead of sampling gave 85.0% (95% interval 79.4–89.3) over 200 games. No evaluation game in this campaign ended in a draw.

What this page shows

The simulator is licensed for competition use only, so this page uses an original diagram and the figures from my dated evaluation reports (9 and 10 July 2026) rather than code, replays or game data. The checkpoints, evaluation logs and audit recordings stay local. My own July re-runs reproduced the headline numbers within their confidence intervals, but nobody else has reproduced them.