Input mapping for mouse

[Martyn0324]

So...

We kind of need a way to make it possible to use mouse movement functions without creating a monster.

I tried for loops, but I think that also would be obscene in terms of computation power. Besides, I can't think of a way that wouldn't return something like: (x=1, y=1), (x=2, y=2) [x always equals to y].

Both pyautogui and sneakysnek can move your cursor to the coordinates X and Y. I suppose that, in order to do so, they have to map the screen. Maybe a solution would be making something that could make the models(like RainbowDQN), instead of selecting the whole function within a list [MouseEvent(MouseEvents.MOVE, x=X, y=Y)], simply selecting a X and Y coordinate(removing the need of a whole list) and then creating a function and automatically passing the selected X and Y into that function.

[Martyn0324]

Unfortunately, Mouse inputs not only are too big, but they're also impraticable even with cloud computing:

A NVIDIA Tesla T4 GPU has a memory of 16 Gb. In Google Cloud, one can borrow up to 4 T4 GPUs, which gives, in total, 48 Gb.

However, I like the idea of creating a template for MouveEvent() function.

Hypothesis: create a game input dictionary which is gonna have a "X" key, where all the X values will be stored(ex: from 0 to 1919), and a "Y" key, where all the Y values will be stored(ex: from 0 to 1079). Create a complementary function to RainbowDQNAgent, generate_mouse_actions() where it will select X from the given range based on the current state. The same will happen to Y. Then, instead of returning a list of tuples actions with the structure [(label, action, value)], it will return [MouseEvent(MouseEvents.MOVE, x=X, y=Y)]

[Martyn0324]

    def generate_mouse_coordinates(self, state, **kwargs):
        frames = []

        for game_frame in state.frames:
            frames.append(torch.tensor(torch.from_numpy(game_frame.frame), dtype=torch.float32, device=self.device))

        self.current_state = torch.stack(frames, 0)

        if self.mode == RainbowDQNAgentModes.OBSERVE and self.observe_mode == "RANDOM":
            self.current_action = random.randint(0, len(self.game_inputs[0]["inputs"]) - 1)
        elif self.mode == RainbowDQNAgentModes.OBSERVE and self.observe_mode == "MODEL":
            self.agent.reset_noise()
            self.current_action = self.agent.act(self.current_state)
        elif self.mode == RainbowDQNAgentModes.TRAIN:
            self.agent.reset_noise()
            self.current_action = self.agent.act(self.current_state)

        # Actions template(from generate_actions() function): [(label, [<serpent.input_controller.MouseEvent object at XXXXXXX>], value)]
        # We need to get X and Y and apply to that template ----> Apply X and Y to action in actions template

        label = self.game_inputs_mappings[0][self.current_action]
        number = self.game_inputs[0]["inputs"][label]

        return number

    def generate_mouse_actions(self, X, width, Y, height, **kwargs):
        # X and Y are both obtained through generate_mouse_coordinates() function

        # X has more possible choices than Y (1920 x 1080). However, if they'll be using the same input dictionary,
        # how are we gonna remove 840 choices from Y without prejudicing X?

        # Possibility: proportion. 1919/1920 = k/1080;  1650/1920 = k/1080;  and so on. In this case, k is the Y we want. Having flashbacks from math classes in highschool?

        # PS: Remember that the screen mapping includes 0,0. That means that 1920x1080 is, in our coordinates, 1919x1079.

        if Y > height:
            Y = Y*height/width
            Y = round(Y)
        
        mouse_actions = []

        label = str(X) + "," + str(Y)
        action = [MouseEvent(MouseEvents.MOVE, x=X, y=Y)]
        value = self.game_inputs[0]["value"]
                
        mouse_actions.append((label, action, value))

        print(f"mouse_actions: {mouse_actions}")

        return mouse_actions

This output follows the same template provided by generate_actions. I suppose this means it'll work.

[Martyn0324]

From 216 Gb to 384 Mb. I see this as an absolute win.