[🚀 Feature]: [py] Allow determining the order of CDP and BiDi events

[nxs7]

Description

This request is essentially the same as #13845, but for Python binding.

In WebSocketConnection._process_message, callback invocations are executed in separate threads:

selenium/py/selenium/webdriver/remote/websocket_connection.py

Line 142 in b59cfe0

Thread(target=callback, args=(params,)).start()

This could disrupt the original ordering of callback invocations, which is problematic for certain CDP events. For example, HeapProfiler.addHeapSnapshotChunk provides only a chunk parameter and relies entirely on the order of invocation to correctly reconstruct the snapshot.

It would be helpful if the callback function WebSocketConnection.add_callback accepts could have an additional order parameter to allow enforcing sequential ordering when needed.

Have you considered any alternatives or workarounds?

No response

[selenium-ci]

@nxs7, thank you for creating this issue. We will troubleshoot it as soon as we can.

Selenium Triage Team: remember to follow the Triage Guide

[cgoldberg]

It would be helpful if WebSocketConnection.add_callback could include an additional parameter that allows enforcing sequential ordering of callbacks when needed.?

I can see how enforcing sequential ordering might be useful, but I don't think it can be a parameter for add_callback. What happens if add_callback is called again without this parameter? Would callbacks be processed sequentially or not? Also, add_callback is not called directly by a user. It is used internally in various places like adding event handlers.

I think this feature would need to be implemented as something like a boolean property on the WebSocketConnection class that determines if callbacks are executed sequentially in one thread for that connection.

@nxs7, Can you edit the title of this Issue to reflect what the feature is, not a possible implementation? Something like "Allow sequential WebSocket callbacks" would be more appropriate.

[nxs7]

edit the title of this Issue to reflect what the feature is, not a possible implementation

Sure, my bad.

I don't think it can be a parameter for add_callback

Sorry if my original wording was unclear, I was in a great hurry when submitting the issue.

My intention was that the callback function add_callback accepts could have an additional order parameter, possibly through an overloaded version, rather than adding a new parameter to add_callback itself. This way, the callback function could use the order parameter to determine the sequence in which the event was actually fired.

This is exactly what was implemented in #13921, the PR that resolved the issue mentioned above. I've updated the original description to clarify the intent.

[cgoldberg]

Thanks. The description is more clear now.

My intention was that the callback function add_callback accepts could have an additional order parameter

I still think this is unnecessary. AFAICT, callbacks are already fired in the order they are added. If you want them fired in a different order, you would call the methods that add them in the order you wanted.

The issue is that they are executed by spawning a thread for each, which aren't guaranteed to reach the browser or be processed in order.

The only way I can think of to guarantee the behavior you are suggesting is what I described in my previous comment. Since I don't think this should be default behavior, there would need to be a property on the WebSocketConnection to enable this. When set, all callbacks would be executed sequentially (blocking) in a single thread instead of each one in a different thread.

[nxs7]

callbacks are already fired in the order they are added

Nope, I meant multiple invocations of a single callback, not multiple different callbacks. Consider the CDP event HeapProfiler.addHeapSnapshotChunk, it is fired multiple times, each time providing an additional chunk. However, due to threading, the order of these invocations may become inconsistent, which can corrupt the final snapshot.

I've updated the description again for clarification.

[cgoldberg]

Then I don't fully understand what you are requesting. Can you provide a Pull Request with the proposed changes that I can review?

[nxs7]

The referenced PR, #13921, implements exactly what I need, except that it’s written in Java.

It maintains an internal counter, increments it upon receiving each event, and passes the counter value to the callback function along with the event data. This allows the callback function to use the counter to restore the correct order of events.

[cgoldberg]

Can you provide a Pull Request with this for Python? The Python bindings are significantly different than Java, and I don't really see how to implement what you are asking for.