After migrating to 0.22, init_psram hangs (esp32s3)

[yanshay]

I migrated an application I'm developing for esp32s3, to 0.22 .

To get it to compile it was all technicalities as in the migration instructions.
To get it to link I had to remove the link with rom_functions.x, I didn't see it documented anywhere but it wasn't found and linked without it.

When I run the code it starts and hangs in init_psram. There isn't much going on before that initialization, this is all the code that runs:

#[esp_hal_embassy::main]
async fn main(spawner: Spawner) {
    esp_println::logger::init_logger_from_env();
    info!("Application Started");

    let mut peripherals = esp_hal::init({
        let mut config = esp_hal::Config::default();
        config.cpu_clock = CpuClock::max();
        config
    });
    info!("peripherals Created");

    // IMPORTANT: PSRAM need to be initialized first, so 'Normal' allocations will use the region
    let (start, size) = psram::init_psram(peripherals.PSRAM, psram::PsramConfig::default());
    info!("Initialized psram");
    init_psram_heap(start, size);
    info!("Initialized psram heap");
...

The log output that comes out is as below, and then the program hangs and doesn't continue further (so "Initialized psram" log isn't reached).

INFO - [src/main.rs:118] Application Started
INFO - [src/main.rs:125] peripherals Created

Anything changed with psram initialization?

[bjoernQ]

Regarding PSRAM the diff doesn't show relevant changes: v0.21.1...v0.22.0

Regarding the rom-functions the migration guide chapter is here: https://github.com/esp-rs/esp-hal/blob/main/esp-wifi/MIGRATING-0.10.md#no-need-to-include-rom_functionsx-manually

[bjoernQ]

it still looks a lot like mis-compilation since running the reproducer with 1.81 works while it doesn't work with 1.82

the fact it apparently only happens somewhere around log and/or esp-println is most probably a red herring

there are some ROM functions like Cache_Invalidate_DCache_All / Cache_Invalidate_ICache_All - but it won't help (actually adding anything might change the generated code so it might even look like it's helpful)

[yanshay]

it still looks a lot like mis-compilation since running the reproducer with 1.81 works while it doesn't work with 1.82

I'll try to see if my application runs with 1.81 as well (though I think esp-Hal 0.21 worked well with 1.82).

It would need to be an mis-compilation that occurs always in the case of init-psram no matter how code is organized around it.

Given what init_psram does which is dealing with cpu memory and cache and such bizarre behaviors I would place my bet more on some memory/cache issues which could cause data corruption to what the cpu sees that could lead to just about anything (including cpu seeing mis-compilation instructions actually).

I saw cases in my tests where data came out in printing but it was garbage data, before hanging.

I also saw in other programs cases where some data made println hang (unrelated to psram). Putting the two together, maybe something with init_psram is causing the cpu read bad data and try to send it over serial and make it hang.

[yanshay]

I compared behavior of 1.81 vs 1.82.
1.81 is better but not perfect.
I have a mode for my app that use either log::{debug!/info!/etc}, let's call it log vs. directly println! .
And then I can control the no-op feature for esp-println

Below are the results of all combinations on both compilers.
The two versions show exactly the opposite symptoms.
I don't know to say what's the reason it hangs in 1.81 where it hangs since with that configuration I don't have logs. It could be something else.

For now it seems I'll stay with 1.81 and see how it goes.

1.81	No-Op set	No-Op not set
Log	Work	Work
Println	Work	Hang
1.82	No-Op set	No-Op not set
Log	Hang	Hang
Println	Hang	Work

Also, calling the cache invalidation functions doesn't seem to help in any way. It even caused some 'LoadProhibited' exception with 1.81.

P.S.
Is there a way to have both 1.81 and 1.82 at the same time with espup? Currently it seems like when I install one it replaces the other (unlike rustup).

[bjoernQ]

I found the cause for this problem at least for the reproducer.

Apply this patch to 0.21.0 to test

diff --git a/esp-hal/src/soc/esp32s3/psram.rs b/esp-hal/src/soc/esp32s3/psram.rs
index 2a6d00a4..2b6557d2 100644
--- a/esp-hal/src/soc/esp32s3/psram.rs
+++ b/esp-hal/src/soc/esp32s3/psram.rs
@@ -125,16 +125,17 @@ pub fn init_psram(_peripheral: crate::peripherals::PSRAM, config: PsramConfig) -
         const MMU_INVALID: u32 = 1 << 14;
         const DR_REG_MMU_TABLE: u32 = 0x600C5000;
 
-        // calculate the PSRAM start address to map
-        let mut start = EXTMEM_ORIGIN;
+        // calculate the PSRAM start address to map - honor the fact there might be
+        // unmapped pages in between
         let mmu_table_ptr = DR_REG_MMU_TABLE as *const u32;
-        for i in 0..FLASH_MMU_TABLE_SIZE {
+        let mut mapped_pages = 0;
+        for i in (0..FLASH_MMU_TABLE_SIZE).rev() {
             if mmu_table_ptr.add(i).read_volatile() != MMU_INVALID {
-                start += MMU_PAGE_SIZE;
-            } else {
+                mapped_pages = i;
                 break;
             }
         }
+        let start = EXTMEM_ORIGIN + MMU_PAGE_SIZE * (mapped_pages + 1) as u32;
         debug!("PSRAM start address = {:x}", start);
 
         // Configure the mode of instruction cache : cache size, cache line size.

The problem is that there are situations where there is an unmapped flash page in between the mapped pages which the current code doesn't handle correctly. That results in .rodata getting unmapped and subsequent reads will get you garbage.

With the reproducer this is the case when compiling with 1.82 but 1.81 produces slightly different code resulting in differently sized segments making the unmapped page disappear. It also explains why random things like enabling debug-asserations or adding / removing unrelated code makes the unmapped page appear / disappear "randomly".

(It's definitely the problem seen in the reproducer - your code might suffer from other or additional problems)

Is there a way to have both 1.81 and 1.82 at the same time with espup? Currently it seems like when I install one it replaces the other (unlike rustup).

Not sure about espup since I don't use it, but you can "install" a toolchain manually by downloading it from https://github.com/esp-rs/rust-build/releases and copy the toolchain to $HOME/.rustup/toolchains (the directoy name is the name of the toolchain, e.g. esp-1.81)

[yanshay]

This is great news (at least looks like it for now).

I applied the changes to 0.22 and my application now runs with 1.82.
I tested with some optimization changes and also both my 'log' and 'println' settings.

@bjoernQ Thank you so much for pursuing this and what looks like the resolution of this issue.
And thanks @relufi as well for the reproducer.

This issue made me worried that I'm at a dead end. So happy to see it resolved. Also a critical issue to resolve for esp-hal in general.