This repository contains the code, prompt templates, configuration files and logs of experiment associated with the paper "A systematic Evaluation of Large Language Model for Autonomous Cyber Defense". The project assess wether large language models (LLM) can act as autonomous agent in the CAGE 2 environment. It compare LLM agents against a single-agent PPO baseline, submitted during the initial CAGE 2 challenge, across multiples adversaries and network topologies. In addition, several LLM models and prompt strategy are evaluated to higlight the impact of prompt design on LLM performance.
This repository includes the results of our experiments, together with the code required to reproduce the statistical analyses. Fixed seeds are provided for bootstrapping operations to ensure that the exact same numerical results can be reproduced.
It also contains the code for the LLM agent and the RL agent, along with all prompts and scenario configuration files required to reproduce the experiments.
-
LLM Logs
This folder contains the logs generated by the LLM agents. -
Prompts
This folder contains the system prompts, including the prompt variants described in the paper. -
RL Logs
This folder contains the logs generated by the RL agents. -
Scenario Configuration Files
This folder contains the CybORG configuration files used to recreate the network topologies evaluated in the experiments. -
Notebooks
This folder contains the notebooks for running the LLM agents, running the RL agents, and reproducing the statistical analyses presented in the paper.
The code in this project is provided primarily as Jupyter notebooks, which can be run either in Google Colab or locally.
The easiest way to use this repository is to import the notebooks into Google Colab. In this case, all required installation commands are already included in the notebooks, so you can simply open a notebook and run the cells in order.
To fully reproduce the statistical analyses and use the same prompts, logs, and configuration files as in the paper, you will also need to make the prompts and scenario configurations files available inside the Colab runtime. This can be done either by:
- uploading the required files directly into the notebook session, or
- storing them in Google Drive and mounting the drive in Colab.
If you choose the drive option, run the following cell in Colab to mount your Google Drive:
from google.colab import drive
drive.mount('/content/drive')After authorization, your Drive files will be available under /content/drive/MyDrive
If you prefer to run the notebooks locally, we recommend creating a dedicated Python virtual environment, installing the required dependencies from the requirements.txt file, and then launching Jupyter Notebook.
1. Create a virtual environment
python -m venv .venv2. Activate the virtual environment
source .venv/bin/activate3. Install dependencies
pip install -r requirements.txt4. Launch Jupyter
jupyter Notebooks/llm-agent.ipynbFor more details on file formats and notebook usage, see the README files in the relevant subdirectories:
- LLM logs: format specifications
- RL logs: format specifications
- Prompts: prompt descriptions
- Scenario Configuration Files: format specifications
- Notebooks: usage instructions
If you use this repository in your research, please cite the following paper:
@article{jacques_systematic_2026,
title = {A systematic evaluation of large language models for autonomous cyber defense},
volume = {56},
issn = {1573-7497},
url = {https://doi.org/10.1007/s10489-026-07386-9},
doi = {10.1007/s10489-026-07386-9},
number = {11},
journal = {Applied Intelligence},
author = {Jacques, Thibaut},
month = jul,
year = {2026},
pages = {367},
}
Copyright (c) Orange SA. Unless explicitly stated otherwise, all files in this repository (including logs, prompts, and configuration files) are distributed under the MIT license (see LICENSE).