If someone consider adding a new model to SerpentAI...

[Martyn0324]

One that could be more data efficient and discard mighty datacenter hardwares would be interesting.

Suggestion: https://github.com/mila-iqia/spr

[Martyn0324]

Some games may require Long and Short Term memory, such as strategy and MOBA games.
I don't know about other algorithms, but RainbowDQN, being an upgraded version of DQN, has a mechanism to simulate a memory system - it's the Replay Memory, which you can find in RainbowDQN folder in SerpentAI path.

However, as time passes during a gameplay, the initial states stored in memory can get quite noisy and the performance might be prejudiced.
If you want a duo to play some games with you and you got some money to spend at cloud services, perhaps you might be interested in the following:

In the case of Dota 2, OpenAI Five used a single LSTM layer with 4096 units using a game state extracted from Dota Developer's API:
https://en.wikipedia.org/wiki/OpenAI_Five

However, I find the option of using a Developer's API as...lame. If someone is interested in trying it, then one could take a look at Overwolf API. They can send information from many games, including League of Legends, Valorant, Fortnite, Dota 2, Overwatch and there's even Eternal Return and Minecraft there.
Unfortunately, though, Overwolf API uses JavaScript while Serpent is in Python. One possible solution for this would be creating a PubSub system using redis, which is included in SerpentAI(for python. You'll still have to download the JS module):

JavaScript publisher example with while loop:

const redis = require("redis");
const redis_server = require('redis-server')
const server = new redis_server(6379)

server.open((err) => {
    if (err === null) {
      // You may now connect a client to the Redis
      // server bound to port 6379.
    }
});

const publisher = redis.createClient();

var channel = "game_events"

var i = 0
while(i < 10){
    i++;
    publisher.publish(channel, "testing");
}

Python subscriber example. PS: This while loop might not be necessary.

from redis import StrictRedis

client = StrictRedis(decode_responses=True)

subscriber = client.pubsub()
subscriber.psubscribe('game_events')

while True:
    messages = subscriber.get_message()

    if messages:
        print(messages)

However, DeepMind seems to have created a more interesting model, which was used to play StarCraft 2: AlphaStar. It uses LSTMs, discards the need of an API through using images and is possible to pre-train by visualizing replays of professional players. However, it's obviously more complex than a single LSTM layer with thousands of units.

Here's some suggestions:
https://github.com/kimbring2/AlphaStar_Implementation
https://github.com/liuruoze/mini-AlphaStar

[Martyn0324]

I'm quite eager to try to implement something using LSTM. As far as I've seen, DQN, DDQN and Rainbow tries to create something like a memory process through the computation of different Q values and other things that I simply can't remember because it's too complex. However, I still have my doubts and having an alternative wouldn't be bad.

Unfortunately, I don't expect to have conditions to spend time (and, as it seems, some money) in that for now.

[Martyn0324]

https://arxiv.org/pdf/2209.11553v1.pdf

https://github.com/liuruoze/HierNet-SC2