Do deep reinforcement learning agents model intentions?

This is the code for implementing the intention reading and generalization experiments from the paper Do deep reinforcement learning agents model intentions?. It is using the simple_spread environment from the Multi-Agent Particle Environments (MPE).

Installation

• To install, cd into the root directory and type pip install -e .

• Known dependencies: OpenAI gym, tensorflow, numpy, also scikit-learn and matplotlib for plotting.

Re-running the experiments

• Download and install the MPE code here by following the README.

• To run the code, cd into the experiments directory and run:

  • for basic MADDPG agents:
    ./experiment.sh coop_navi_0
  • for MADDPG + shared scheme, all agents use shared model:
    ./experiment.sh coop_navi_shared_0 --shared
  • for MADDPG + shuffle scheme, agents are shuffled for each episode:
    ./experiment.sh coop_navi_shuffle_episode_0 --shuffle episode
  • for MADDPG + ensemble scheme, agents are sampled for each episode:
    ./experiment_ensemble.sh coop_navi_ensemble_episode_0 --ensemble-choice episode

Individual scripts

• train.py - basic training script, also used for evaluation
• ensemble.py - ensemble training script, also used for evaluation
• learning_curve.py - plots learning curve of an experiment
• statistics.py - collects basic benchmark data from evaluation
• prepare.py - simplifies evaluation data for further processing
• prepare_ensemble.py - simplifies evaluation data for further processing, for ensemble results
• accuracy.py - calculates per-timestep target prediction accuracies
• figure.py - plots target prediction accuracies for all agents
• sheldon.py - runs evaluation against Sheldon agents (agents with fixed targets)
• sheldon_ensemble.py - runs evaluation against Sheldon agents, for ensemble results

For usage details refer to experiment.sh, experiment_ensemble.sh and individual files.

Paper citation

If you used this code for your experiments or found it helpful, consider citing the following paper:

@article{matiisen2018do,
  title={Do deep reinforcement learning agents model intentions?},
  author={Matiisen, Tambet and Labash, Aqeel and Majoral, Daniel and Aru, Jaan and Vicente, Raul},
  journal={arXiv preprint arXiv:1805.06020},
  year={2018}
}

Thanks

Thanks to OpenAI for the original paper and for releasing the code.