marl-stealth-impostor

A learned, stealth-constrained impostor for cooperative multi-agent reinforcement learning on PettingZoo Simple Spread. One agent in an otherwise cooperative team is re-trained to degrade team performance while staying close — under a Jensen-Shannon-divergence (JSD) penalty — to the honest policy distribution it is embedded in.

Companion code for the BSc thesis by Andreas Theos Fernandez and Erik Guerra (KTH Royal Institute of Technology, 2026).

Setup

Requires Python >= 3.10.

git clone https://github.com/Ante042/marl-stealth-impostor.git
cd marl-stealth-impostor
python -m venv .venv
source .venv/bin/activate
pip install -e ".[plotting]"

That installs the impostor package in editable mode together with PyTorch, PettingZoo + mpe2, and the scientific dependencies listed in pyproject.toml. The [plotting] extra adds matplotlib and seaborn, which the scripts in scripts/ require. Omit it if you only need training and evaluation.

To reproduce the honest baseline (Stage B below) you additionally need a clone of EPyMARL at commit cbc38c0 with the two patched files from epymarl-patches/ applied. The impostor training itself (the central contribution of this repo) is fully self-contained.

Repo layout

marl-stealth-impostor/
├── impostor/                       # the Python package
│   ├── networks.py                 # RNNAgent, GRUAgent, QMixer (EPyMARL-compatible)
│   ├── env_wrapper.py              # ImpostorEnv — Simple Spread with a frozen honest team
│   ├── honest_team.py              # HonestTeam — loads + serves the frozen agent.th
│   ├── utils.py                    # jensen_shannon_divergence + RunningMeanStd
│   ├── bc_train.py                 # Stage A: behaviour-clone a Hungarian teacher
│   ├── train_mappo_impostor.py     # PPO impostor with JSD penalty
│   ├── train_qmix_impostor.py      # DQN impostor with JSD penalty
│   ├── eval_jsd_posthoc.py         # post-hoc empirical action-histogram JSD evaluator
│   ├── scenario_env.py             # four structured scenarios (S1–S4) with fixed geometry
│   └── scenario_eval.py            # per-scenario evaluation (return, JSD, action match)
├── checkpoints/
│   └── honest_mappo_bcft_seed42/
│       └── agent.th                # frozen honest team (the one all impostors are trained against)
├── epymarl-patches/                # ppo_learner.py + run.py patched for BC fine-tuning
├── scripts/
│   ├── launch_bcft.sh              # Stage B: BC-anchored MAPPO fine-tune via EPyMARL
│   ├── launch_impostor.sh          # one impostor cell (ALGO LAMBDA SEED)
│   ├── launch_sweep.sh             # full lambda sweep (default 1..10 × {mappo, qmix} × 3 seeds)
│   ├── scenario_summary.py         # aggregate scenario eval results into tables
│   ├── trajectory_l2.py            # trajectory L2 distance metric
│   ├── trajectory_l2_random_init.py # trajectory L2 on random-init eval
│   ├── plot_team_return.py         # team-return vs lambda plots
│   ├── plot_jsd_posthoc.py         # JSD vs lambda plots
│   ├── plot_cross_scenario_bars.py # per-scenario bar charts
│   ├── plot_scenario_panels.py     # scenario panel figures
│   ├── plot_scenario_team_return_bars.py
│   ├── plot_trajectory_l2.py       # trajectory L2 bar charts
│   ├── plot_lambda_sweep_l2.py     # L2 vs lambda sweep plots
│   ├── plot_cell_health.py         # QMIX training-stability grid
│   ├── make_impostor_gif.py        # render impostor rollout as GIF/MP4
│   └── make_scenario_gif.py        # render per-scenario rollout GIF/MP4
├── tests/
│   ├── test_jsd.py                 # unit tests for JSD primitive
│   └── __init__.py
└── pyproject.toml

Honest baseline

The impostor is trained against a fixed honest team. We ship the canonical checkpoint at checkpoints/honest_mappo_bcft_seed42/agent.th; that is the team every reported impostor cell in the thesis is evaluated against. The checkpoint was produced in two stages:

Stage A — behaviour cloning with DAgger refinement. A scripted Hungarian-assignment teacher labels (obs + agent-id one-hot, action) sequences on Simple Spread. A GRU actor with hidden dimension 128 is first trained by cross-entropy on 4000 teacher-rollout episodes. Then 10 DAgger iterations refine the policy: each iteration rolls out 2000 episodes under the student policy, labels them with teacher actions, and retrains on the accumulated dataset. This closes the distribution gap between the teacher's state visitation and the student's. EPyMARL's MAPPO uses exactly this architecture, so the resulting agent.th drops straight into the EPyMARL checkpoint layout.

python -m impostor.bc_train --n-agents 3 \
    --save-dir results/epymarl/mappo_n3_bc_init

Stage B — BC-anchored fine-tune (BCFT). EPyMARL MAPPO is started from the BC actor and KL-anchored toward it for a short warm-up. This avoids the early-training entropy collapse that would otherwise drift the policy away from the assignment-style cooperation we want.

./scripts/launch_bcft.sh 3 42

The patched epymarl-patches/ppo_learner.py adds the KL anchor term; the patched epymarl-patches/run.py wires the BC actor checkpoint into the agent at construction time. Apply both files in place over a clean EPyMARL clone before running.

Impostor sweep

Train an impostor against the frozen honest team by varying the JSD penalty coefficient λ. λ = 0 reproduces an unconstrained impostor; large λ forces the impostor's action distribution close to the honest distribution at its own observation. The full sweep:

./scripts/launch_sweep.sh 3 \
    checkpoints/honest_mappo_bcft_seed42

That launches MAPPO and QMIX impostors at λ ∈ {1, 2, …, 10}, three seeds each, with a default cap of 6 cells running concurrently. Each cell runs t_max = 500000 environment steps. Override the sweep with the optional fourth argument:

./scripts/launch_sweep.sh 3 <honest_path> 6 "0 1 2 5 10"

For a single cell:

./scripts/launch_impostor.sh qmix 3 4 42 <honest_path>

Result directories land at results/impostor/{algo}_n{N}_lambda{L}_seed{S}/ with the trained weights and training_log.json (the args + per-log-interval metrics). MAPPO saves impostor_actor.th + impostor_critic.th; QMIX saves impostor.th. Logs at results/logs/impostor_{algo}_n{N}_lambda{L}_seed{S}.out.

Eval

Random-init evaluation

impostor.eval_jsd_posthoc rolls out an impostor checkpoint against the honest team over many episodes with diverse environment layouts, and reports the empirical action-histogram JSD between the impostor's action distribution and the honest team's distribution at the impostor's observation. This is the metric reported alongside team return in the thesis.

python -m impostor.eval_jsd_posthoc \
    --single-run results/impostor/qmix_n3_lambda4_seed42 \
    --honest-checkpoint checkpoints/honest_mappo_bcft_seed42 \
    --n-episodes 200

To evaluate all runs under results/impostor/ at once, omit --single-run:

python -m impostor.eval_jsd_posthoc \
    --root results/impostor \
    --honest-checkpoint checkpoints/honest_mappo_bcft_seed42

Structured scenario evaluation

impostor.scenario_eval evaluates impostor performance on four fixed scenarios (S1–S4) with predetermined agent and landmark positions. The scenarios vary geometry between clustered and spread layouts, testing whether the impostor generalises across environment configurations.

python -m impostor.scenario_eval \
    --root results/impostor \
    --honest-checkpoint checkpoints/honest_mappo_bcft_seed42 \
    --n-episodes 200

Scenario definitions are in impostor/scenario_env.py. Results are written to results/scenario_eval/.

Hyperparameters

Honest baseline (BCFT, EPyMARL MAPPO):

• t_max = 400000, lr = 0.0001, epochs = 4, entropy_coef = 0.0
• bc_kl_coef = 2.0, bc_warmup_steps = 20
• hidden_dim = 128, GRU actor, decentralized critic
• EPyMARL parallel runner with batch_size = batch_size_run = buffer_size = 10 (episode-batched), which is its standard MPE config
• 3 seeds: {1, 2, 42} (seed 42 is the operative one)

MAPPO impostor:

• t_max = 500000, rollout_length = 2048, ppo_epochs = 4
• lr = 5e-4, gamma = 0.99, gae_lambda = 0.95, eps_clip = 0.2
• entropy_coef = 0.01, hidden_dim = 64, decentralized critic
• Reward standardisation enabled
• JSD term in policy loss: pg = clip_objective + λ · JSD(π_impostor || π_honest)

QMIX (DQN) impostor:

• t_max = 500000, batch_size = 32, buffer_size = 50000
• lr = 5e-5, gamma = 0.99, double Q
• target_update_tau = 0.001 (slow Polyak)
• ε-greedy: 1.0 → 0.05 over 50000 steps
• Reward standardisation enabled, Q-standardisation in the JSD softmax term
• JSD term in TD loss: loss = TD_loss + λ · JSD(softmax(Q) || π_honest)

Citation

If you build on this work please cite the thesis (full reference TBA once the thesis is filed). The maintained citation block will appear here.

License

MIT — see LICENSE.