Best Practices for App Deployment

[MitchBradley]

My CNC UI project (UI for the FluidNC CNC controller running on an Elecrow 7" ESP32-S3 HMI display) is getting close, so it is time to think about the deployment and code management details.

I'm sure that other considerations will arise, but for now I am thinking about how to manage a custom board definition. Is there a better way to add one than manually adding files to lib/micropython/lib/ports/esp32? Ideally the solution would get the data from another repo, but maybe there is no easy way to do that. Perhaps hot-patching?

[kdschlosser]

Every one of the CONFIG_* parameters that can be set in the board configuration is able to be passed at the command line to the make.py script. Those settings are what control the underlying ESP-IDF SDK. It is what defines the MCU that is being used.

You have the ability to "bake" your program into the firmware by passing the location where you have defined a frozen manifest. This is done by using the FROZEN_MANIFEST= argument of the make.py script.

I need to better know what you are wanting to do when you say board definition. The main purpose of being able to define boards is for the inexperienced user. it is to provide convince functions and variables that align with what a board manufacturer has named things as. They are so a user doesn't have to know if there is SPIRAM or not or how much flash there is. The only use case that would have applied is if a board didn't use a uart repl and instead was using USB-CDC. But even that is able to be controlled using a command line parameter.

The reason why the build script is so large is because of adding the ability to control the build from the command line. The things where the build system in MicroPython fell short have been addressed and by doing that it has also removed the need to define a board.

[MitchBradley]

I was originally thinking that I needed to change the REPL setup and add an additional dupterm slot so I could disable the UART REPL, use the UART for communicating with the CNC controller, and debug on WebREPL. But I no longer need that since I can effectively disconnect the UART0 REPL by redirecting the Tx and Rx pins to UART1. That gives me all that I need, so I can probably use the stock S3..OCT build. And knowing that everything can be overridden from the command line should solve any residual issues that might arise. I do plan to freeze some, but not all, of my application modules. Others will be stored on the file system to provide some measure of user configuration, along with NVS settings. I have a mklittlefs script to prepopulate the vfs. All systems go!

[kdschlosser]

if you need some kind of special changes made to how the repl works lemme know.

Just to let you know MicroPython supports 3 ways of getting a repl prompt. You have the WiFi REPL, the UART REPL and the USB-CDC REPL.

Originally MicroPython would shanghai GPIO 19 & 20 if the ESP32 supported USB-CDC. It would do this while also using the GPIO's for the UART REPL. It would occupy both for the REPL. I stopped that from happening and added a command line switch --usb-otg to reactivate pins 19 & 20 as being used for the USB-CDC REPL. This doesn't deactivate MicroPython using the UART GPIO's for the REPL as well. This is something I have been thinking about patching. in order to deactivate the UART REPL you would need to manually adjust the mpconfigport.h file and add #define MICROPY_HW_ENABLE_UART_REPL 0

This is the original way the initialization was handled.

#if CONFIG_ESP_CONSOLE_USB_SERIAL_JTAG
    usb_serial_jtag_init();
#elif CONFIG_USB_OTG_SUPPORTED
    usb_init();
#endif

#if MICROPY_HW_ENABLE_UART_REPL
    uart_stdout_init();
#endif

CONFIG_USB_OTG_SUPPORTED is not something that is able to be changed. It is set by the ESP-IDF and there is no option to set it. That's the problem is that it is being use to dictate what the pins get used for when maybe the pins are not wanting to get used for that...

and this is how MICROPY_HW_ENABLE_UART_REPL gets set...

#ifndef MICROPY_HW_ENABLE_UART_REPL
    #define MICROPY_HW_ENABLE_UART_REPL (!CONFIG_USB_OTG_SUPPORTED && !CONFIG_ESP_CONSOLE_USB_CDC && !CONFIG_ESP_CONSOLE_USB_SERIAL_JTAG)
#endif

Now the funny thing with this one is CONFIG_ESP_CONSOLE_USB_CDC is something the user is able to set but it is not used when initializing the USB-CDC in micropython.

All of that being said, for some crazy ass reason I think even if you disable the UART REPL MicroPythyon will not allow you to set those pins as as a UART. I have already brought up an issue with the MicroPython authors about this and it will be years before it gets fixed. It took 2 years before they fixed the __dict__ attribute not existing at the module level. I reported that issue in 2020 and it didn't get fixed until 2022.

That is the reason why you are seeing all of this hot patching that I am doing. I am almost to the point of not bothering to open up any more issues at the MicroPython repo. Things never get fixed.. An example is try and set -2147483648 using the NVS for the ESP32. You have to set it using the int32 function. You cannot set that number, it results in an overflow. MicroPython doesn't honor the lower boundaries of signed integers on the ESP32. It shorts them by 1. That is a HUGE bug that should be fixed yesterday.. It is actually a memory leak too, Now what should fit into a 4 byte integer requires 8 bytes of memory. Anything being done about it? NOPE. This requires a massive rewrite of a very large chunk of the MicroPython code in order to fix it. Above my pay grade to correct.

Now you know why I am hot patching the code in MicroPython. Getting things fixed has a laughable time frame. I just had an author of a different repo tell me that I had to pay him to fix bugs in his packaged and listed opensource Python library. Wow, why opensource it if you are expecting to get paid for the work?

Since you did mention using the NVS... I highly recommend doing what you see done here. use struct to pack the settings into a single bytes object and set it into the NVS as a blob. I recommend doing this because the NVS has a crap load of header and footer data that gets placed with the data so if you want to store a 4 byte integer it ends up taking up something on the order of 100 bytes or more. I would have to look again to know exactly how much data is taken up. A blob is able to be stored only taking up that additional amount of storage a single time. It saves a lot of flash space doing that. I also provided functions that change a float into a 32 bit integer so they can be stored as binary data using struct.

[kdschlosser]

I did manage to get them to add a nifty ability to the ESP32.

drop this code into a file names boot.py and upload the file to the ESP32...

import machine

uart = machine.UART(0, baudrate=921600)  # NOQA

then kill the serial connection and change the speed to 921600 and reset the ESP32 and reconnect the serial. Now when you transfer files it will be 10 times faster.

each time you flash you will have to send that boot.py at the lower 115200 speed and do the above again. Not that big of a deal. It is a real time saver when uploading a lot of files.

if you had to upload 4mb of files it would take 4.85 minutes at the factory speed. after adding those couple of lines of code it would take 36.40 seconds. Not real world numbers as the speed is going to vary some. it gives you an idea of the kind of difference that exists.

[MitchBradley]

For the FluidNC project, which is a large C++ codebase, I have been using esptool to flash at 921600 since forever. It works well in most cases, although there are a few ESP32 modules that can't handle it for some reason, possibly related to marginal USP-serial chips. Some Linux setups also seem to reject that baudrate.