Starting code upload crashes the board hard if the currently running code.py has an open & unflushed file on SD card.

[raquo]

CircuitPython version

• Tested on 9.0.2 and 8.2.0 (Bootloader 3.15.0)
• Tested on two different ItsyBitsy M4 boards, two different SD cards (formatted with official SD formatting tool), and two different SD card readers, on breadboard, and on a PCB.

Code/REPL

# this is code.py

import digitalio
import board
import busio
import sdcardio
import storage
import os
import time

ledPin: digitalio.DigitalInOut = digitalio.DigitalInOut(board.D13)
ledPin.direction = digitalio.Direction.OUTPUT

ledPin.value = False

time.sleep(1)

spi = board.SPI()

# # Same effect:
# spi = busio.SPI(
#     clock = board.SCK,
#     MOSI = board.MOSI,
#     MISO = board.MISO
# )

cs = board.D10

sdcard = sdcardio.SDCard(spi, cs)
vfs = storage.VfsFat(sdcard)

storage.mount(vfs, "/sd")

existingFilenames = os.listdir("/sd")

print(existingFilenames)

newFilenames = [
    "FILE1.TXT",
    # "FILE2.TXT"
]

for filename in newFilenames:
    if filename in existingFilenames:
        print("removing {}".format(filename))
        os.remove("/sd/" + filename)
    print("creating {}".format(filename))
    file = open("/sd/" + filename, "w")
    # file.flush()

print("Done!")

ledPin.value = True

Behavior

When you upload this code for the first time and run it, the output is:

Auto-reload is on. Simply save files over USB to run them or enter REPL to disable.
code.py output:
['.Spotlight-V100']
creating FILE1.TXT
Done!

Code done running.

This is as expected. The problem happens when you try to upload a small change (e.g. change "Done!" to "Done!!" in the code) – at this point there is no further output, the board crashes very hard.

Description

Prerequisites:

• I have tested this only with ItsyBitsy M4 express board.
• This code requires you to connect an SD card reader with an SD card in it, formatted as FAT32.
  • I used this Adafruit breakout board on a breadboard (photo attached), and I also tried it with my own custom PCB that has a similar SD card reader, to which ItsyBitsy was connected via headers, not wires.
• You need to create an empty sd directory on your CIRCUITPY drive (for CircuitPython 9.x)

Steps to reproduce:

• Upload the offending code for the first time. It will run without crashes and produce the output above.
• You can reload the board with Ctrl+D in Mu, and it will also run without crashes.
• Note that FILE1.TXT will not actually be created on the SD card.
• Now, open code.py on the CIRCUITPY drive, and make a small change, e.g. change "Done!" to "Done!!", and save the file, so that it uploads to the board.

The board and the CIRCUITPY drive will now completely freeze:

• No LEDs will blink or light up throughout all this
• CIRCUITPY drive will become unresponsive (MacOS cursor becomes a beach ball if you try to open it)
• Other USB devices on the same USB hub will start glitching (keyboard missing or repeating keys, mouse cursor stuttering)
• After 20-30 seconds of this, the drive is unmounted.
• After a few more seconds, the USB glitches stop.
• The board LEDs remain dark.

Disconnecting the board and connecting it again blinks the LEDs initially (even the D13 LED lights up for a fraction of a second), but after that the board goes dark and remains unresponsive. The CIRCUITPY drive never appears in MacOS Finder.app, and attempting to open Serial log in Mu causes the same freezing cycle as described above.

Disconnecting the board from the SD card (e.g. by removing the card from the reader, or disconnecting the SCK wire), and then re-connecting the board to the USB does revive the board and the CIRCUITPY drive. You can re-insert the SD card and Ctrl+D reload the code to get back to the initial conditions. However, the new code was not uploaded. Attempting to upload the code again will result in the same freezing cycle.

Additional information

Uncommenting file.flush() in the code resolves the issue.

So, it appears that having un-flushed open file while starting a file upload somehow kills the board.
Also, I'm not sure if this is expected or not, but the open() call does not seem to create the file immediately, the file is only created on the SD CARD if file.flush() is uncommented.

See also my comment below, that likely gets closer to the root of the problem.

[raquo]

I just found a way to fix the problem without flush()-ing:

I needed to use the VFS object's open method: vfs.open("FILE1.TXT") instead of the global open("/sd/FILE1.TXT"). Then it seems to work fine, no crash.

This doc page says:

You can only have either your computer edit the CIRCUITPY drive files, or CircuitPython. You cannot have both write to the drive at the same time. (Bad Things Will Happen so we do not allow you to do it!)

And indeed, I am not allowed to create a file in the root directory: as expected, open("FILE.TXT", "w") throws OSError: [Errno 30] Read-only filesystem error.

In contrast, it seems that attempting to create a file on the SD card using the "global" open method might be bypassing this safety check, and causing the above-mentioned "Bad Things".

I think open("/sd/FILE1.TXT", "w") should either throw a similar OSError, or delegate to vfs.open if possible – either way will be better than a hard crash.

[jepler]

This looks like a real bug that we should fix.

My gut tells me that it has something to do with attempting to "finalize" all objects when the VM shuts down, which works on objects in an unpredictable order, so for instance it could be the case that the mounted filesystem object has been finalized and then the file object is finalized, leading to an error because it's no longer valid to make calls related to the filesystem object...

The vfs.open workaround is interesting, I wonder why it makes a difference.

However, in the meantime, you may find that your code is more robust if it uses a with statement to handle the lifetime of the file:

with open("/sd/" + filename, "w") as file:
   ... # operate on file, knowing it will be flushed and closed when the block exits

[raquo]

@jepler Sorry, I think I was wrong about vfs.open working.

It seemed to have fixed the issue, but I'm trying it again now, and it's freezing the same as the global open. I probably didn't set up the test correctly yesterday.

--

Thanks for the with suggestion. Just to add a bit of context: the file I'm writing is a log file. I'm keeping it perpetually open, and I write() ~30 bytes to it several times per second, and then flush() the file every few seconds. I figured that was the most efficient way to do it, to reduce the time spent writing data.

I guess I should instead accumulate the bytes-to-write in some variable, and run the entire with-open-write thing once every few seconds, as you said.