Improve performance of incoming serial processing

[cpirnat]

Original title (edited by @carlosperate): Program hangs unresponsive during loop in REPL.

---

Steps to reproduce

• Plug in CircuitPlayground Express

• Open REPL

• Generate a infinate loop:

while True:

    print('foo')

Ctrl+C will not kill loop in REPL and GUI hangs. This is in linux on version 1.0.0.beta.14 and 1.0.0.beta.15

[ZanderBrown]

I imagine it's in every version, I don't believe the REPL for m:b/cp has a separate thread so it will take the GUI with it

Raw thoughts: Serial comms thread?

[ntoll]

+1 on serial comms thread (and it shouldn't be too hard to implement with Qt's message passing).

[carlosperate]

The serial data in and out is managed by QSerialPort and propagated via signals, and it's probably a fair assumption to say that Qt has designed that to be non-blocking (although it's an assumption worth contesting).
I tried this in the micro:bit mode, which should be the same as the CP mode, and coudn't replicate in macOS. @ZanderBrown @ntoll could you check if you can replicate on linux?

[ntoll]

I've replicated this with my CPE and micro:bit on Linux. I was under the same impression as @carlosperate when it came to QSerialPort's use of signals etc... but this still appears to block the UI thread. I'm going to take a look and try to work out what's going on.

[ntoll]

So, AFAICT the problem is that the process_bytes method in the MicroPythonREPLPane class simply can't keep up with the flood of data. This method is called in the UI thread since it is updating the UI, but contains a for loop over each byte in the incoming data in order to correctly handle the various control codes used to control the position of the cursor and that sort of thing.

Two solutions:

• Do some flooding detection and warn the user when the connected device is spewing forth data.
• A significant re-write of this functionality so we have a non UI thread to process incoming data and emit signals to tell the widget to react to incoming control codes (move the cursor etc...).

Thoughts on the best approach? The first is simple, the second (much) less so.

[ntoll]

Hmmm... of course, it works on Mac, but not Linux. sigh Perhaps my assumption above isn't correct after all. :-/

Suggestions welcome.

[ZanderBrown]

2 is potentially the better solution, I think the main problem is I/O is evil 🙂

If we had a background worker it could detect a flood and deliver data to UI in batches perhaps sending a warning after a period?

[ntoll]

Quite... sounds like a recipe for spaghetti code. ;-)

[eLEcTRiCZiTy]

A fast microcontroller as STM32F4 (on Adafruit Feather board) is fast enough to send log messages fast as 100Hz or quicker without a problem. I quickly block the GUI and lock the Mu Editor only with 20Hz logging frequency. This behavior isn't right! I can use another program to communicate with the board, but then I can't use Mu editor to debug errors or something because it hangs.

edit: Oh gosh, I realize this problem is more than a year old. Hasn't anyone solved this yet?

Normal console behavior is to buffer incoming messages and write them out as fast as possible. If writing out every message is slow, then write out part or entire buffer. If this is slow, notify the user about flooding of serial input so messages can not be written out. User can enlarge buffer to write out messages after when flooding ends or simply buffer messages and write then in file.

[tjguk]

@dybber is any of this affected by your recent merges?

[dybber]

Hi all.

I looked a lot at the REPL in the spring, those changes are yet to be merged. So far I haven't looked much at optimizing the REPL, but just getting it in a state where it handles the input correctly and doesn't output garbage. Those fixes are in PR #1026.

I agree that it should be optimized, it also annoys me sometimes, and I'd be happy to look at it at some point in the future, but I think other things in Mu have even higher priority. For example, being able to build and notarize for Mac, so we can actually release new versions of Mu again. Things are slowly, but steadily moving forward right now, so hopefully it will be something we can get to in the near future.