Unexpected flushing of empty frames (ESPTOOL-1250)

[g3gg0]

Here is some issue I found while using the stubs on a lower level

Already wrote there

The SLIP start marker (0xC0) which is the same as the end marker (0xC0) already causes a transmit on every first byte written to the buffer.
So any "single-transfer" kind of frame already triggers **three** bulk transfers on the USB-JTAG.

This is the line:

esp-flasher-stub/src/transport.c

Line 54 in 2fa753f

if (c == SLIP_END || transferred_without_flush >= 63) {

Recommendation:

if (((transferred_without_flush > 1) && (c == SLIP_END)) || (transferred_without_flush >= 63)) {

Also i am puzzled about the correct stub loaders to PR/patch/use.

• current ones: deprecated (and no issue system)
• recommended ones: archived (and crashing)
• these: experimental (and crashing)

[g3gg0]

(updated code, as the transferred field is updated before sending)

[g3gg0]

Another finding:
the ROM call uart_tx_one_char() flushes the JTAG UART after every written 0x0A byte as well,
such that we get even more unneccessary bulk transfers which are only partially filled.

[g3gg0]

Another question:
I cannot make out the code that waits for the USB Endpoint buffer to be fetched by the host before it's going to get populated again.
Do the ROM routines and the stub loader just assume that the host is fast enough to fetch the USB Bulk transfers?

[Dzarda7]

Hello @g3gg0, the stub you should still use and esptool uses is the esptool-legacy-flasher-stub. This repo contains new stub that will replace esptool-legacy-flasher-stub and is in active development, but feel free to try it, it should be almost in the working state, few features are missing yet like MD5 check, compressed flashing and 32-bit addressing.

Regarding the issues you have. Does any of it causes you any trouble for you? You are right, each packet transfer triggers at least two flushes. I will fix that. The reason why it flushes also on 0x0A is that it corresponds to /n new line char and it is implemented in ROM that it triggers flush. I was recently checking the way to improve throughput and this was one of these I want to focus on. I thought about implementing it from scratch and taking a buffer so it decreases number of function calls and checks.

Great and deep investigation, thanks for that!

I cannot make out the code that waits for the USB Endpoint buffer to be fetched by the host before it's going to get populated again.
Do the ROM routines and the stub loader just assume that the host is fast enough to fetch the USB Bulk transfers?

There is this check READ_PERI_REG(USB_DEVICE_EP1_CONF_REG) & USB_DEVICE_SERIAL_IN_EP_DATA_FREE) to verify if new data can be put into endpoint buffer with 5 ms timeout, but we probably do not check the return value properly, that is another thing that can be improved, will keep it in mind, thanks.

If you find more issues feel free to let me know, will try to address this.

[g3gg0]

Hi, @Dzarda7 .

Thanks for your reply.
I made a in-browser firmware viewer/editor (blog, direct link) for flashing and reverse engineering.
Also implemented a custom JS-flasher with the goal to be a bit more foolproof when the communication wiring is a bit flimsy.

While the Windows CDC class driver / serial works just fine, I also wanted to get it to run on Android.
And there the rabbit hole started.

Long story short, I am not sure (yet) if the ESP32 drops some frames BECAUSE WinUSB and/or WebUSB is too slow and firmware doesn't care if USB picked up the bulk transfer....
...or if the WinUSB/WebUSB and Android/WebUSB pairs drop bulk transfers because it's enforcing "host queues, then device responds" scheme.

The stub happily sends multiple bulk transfers, we dont know how many and cannot anticipate.

I am still investigating :(

[Dzarda7]

Pretty cool. Do you think it would help you to get rid of the flush on 0x0A? Cannot guarantee, but we can possibly get rid of it as it is unnecessary.

[g3gg0]

I have a "flasher" webinterface here and also built a modified version of the ESP32-C6 loader.
It has a "reliability test", flooding register reads to ensure link is stable enough.
Having a "single bulk transfer" ping-pong style transfer scheme, definitely improves working with the basic commands.

I made a local test stub with a simplified code, directly interfacing USB JTAG as per TRM.

However even with a really proper response to the READ FLASH command, I still see some data being dropped.
I already only read 31 bytes from flash at once - but this again causes three successive bulk transfers again :)

  SLIP: C0 01 D2 02 00 00 00 00 00 00 00 C0
  SLIP: C0 [31 bytes of flash data] C0
  SLIP: C0 [MD5] C0

(why 31? because with escaping all bytes, the max SLIP size would get 64)

But even then sometimes one of these SLIP packets just is missing and my JS reading code never can recover.

As in my local stub I already actively wait for the USB buffer to get picked up properly by the host, there MUST be some further WinUSB/WebUSB issue underneath.
I bet it is something like "only two successive bulk transfers at once" can be buffered, else transfers are just silently dropped.

As a more specific answer to your question:
"Cleaning up" the code to respond with a full bulk as expected solves the first, basic transfers like register read.
Flashing will be possible then, I think.
But flash reading still conceptually seem not to be feasible with my WebUSB approach.

So I probably need to dig deeper into WebUSB's limitations and maybe even build custom stubs with a command
to read flash content with a single bulk response in ping-pong style.

Unstatisfying :D

[g3gg0]

I realized an important detail.

Whenever the ESP32 flushes the EP FIFO with less than the FIFO size of data, this is interpreted as the final packet and the transaction is completed. Because of this, the stub loader should only explicitly flush data after a command has been fully processed.
This doesn't cause problems on the native drivers, as they are fast enough and are not based on a "request-response"-scheme (i guess)
WebUSB in particular expects the communication to follow that ping-pong such that the "device response" is a single(/combined) transaction.

While filling buffers, the stub should rely on the automatic commit of the EP FIFO when it becomes full. If the buffer previously reached exactly 0x40 bytes, an additional empty transaction is effectively a no-op but still signals the end of the transfer.

I tested this and it works now!

If the stub code is changed to follow these rules, everything works with both WebUSB and the native CDC driver:

• rely on the automatic USB serial flush when 0x40 bytes are written
• explicitly flush only after a command has been processed or when a response is expected (e.g. the read-flash command)
• wait until the FIFO has free space before writing

With this, flash reading works correctly.
Conceptually, this approach is cleaner and more streamlined. Parsing payload data and checking for markers like 0xC0 breaks layering and should not be necessary.

Here a (minimal) example for ESP32-C6:

and also in

• cmd_loop() whenever the packets are meant to be "ready for the host"
• handle_flash_read() just before SLIP_recv()

will fork and show the changes needed to sanitize the committing.
yay :)

[g3gg0]

Okay seems to work now on android. Finally :)

[Dzarda7]

Thank you very much for this investigation @g3gg0, I am sorry that you had so much trouble with this. I was aware of this and also pointed it out here and mentioned that we should address this with proper fix. But there was some windows issue, no one reported it for old stub yet and I did not have time to dig deeper into it, so I thought it can wait and approved it with TODO comment.

I cannot guarantee now that we will put it into the old stub or add it here as we mainly focus on this new stub, but as you already did the work, we might add it. Thank you very much, we will keep this open until we fix it here also. Might ask you later if you can try it if that is okay.