FATAL_GC_ERROR in SVR::gc_heap::sort_mark_list (gc.cpp:10804): region_limit == NULL — mark list contains an address outside any region (regions, Server GC, .NET 10.0.11)

[lingchenyu]

Describe the bug

FATAL_GC_ERROR (COR_E_EXECUTIONENGINE 0x80131506) in SVR::gc_heap::sort_mark_list

Mark list contains a live, marked object whose seg_mapping_table entry
resolves to a region with heap_segment_allocated() == 0.

Fault site: sort_mark_list+620 (after the testq %rbx,%rbx; jne guard at +610)
Config: regions, Server GC, GCConserveMemory=9, .NET 10.0.11

Steps to reproduce

We do not have a minimal or deterministic repro. The crash happens only in our
staging workload after it has been running for about four hour.

What is likely relevant: the application performs periodic assembly hot-reload
approximately every 5 minutes【For Test Only】, for the entire lifetime of the process.

Hot-reload mechanism:

• Plugin assemblies are loaded into a collectible AssemblyLoadContext
  (new AssemblyLoadContext(name, isCollectible: true)), loaded from
• Every ~5 minutes a new ALC instance is created, the new version of the assemblies is
  loaded into it, traffic is switched over, and after 30s the previous ALC is Unload().

Expected behavior

No crash.

Actual behavior

The process aborts with SIGTRAP raised by GCToOSInterface::DebugBreak() from
FATAL_GC_ERROR() in sort_mark_list.

Native stack (lldb bt)

* thread #8, stop reason = signal SIGTRAP
  * frame #0: libc.so.6`__GI___wait4(pid=27076, ...) at wait4.c:27:10
    frame #1: libcoreclr.so`PROCCreateCrashDump(...) at process.cpp:2580:22 [opt]
    frame #2: libcoreclr.so`::PROCCreateCrashDumpIfEnabled(...) at process.cpp:2807:9 [opt]
    frame #3: libcoreclr.so`::PROCAbort(...) at process.cpp:2839:5 [opt]
    frame #4: libcoreclr.so`sigtrap_handler(int, siginfo_t*, void*) [inlined] invoke_previous_action(...) at signal.cpp:447:13 [opt]
    frame #5: libcoreclr.so`sigtrap_handler(code=5, ...) at signal.cpp:753 [opt]
    frame #6: libpthread.so.0`__restore_rt
    frame #7: libcoreclr.so`GCToOSInterface::DebugBreak() at gcenv.unix.cpp:507:1 [opt]
    frame #8: libcoreclr.so`SVR::gc_heap::sort_mark_list() [inlined] SVR::FATAL_GC_ERROR() + 5 [opt]
    frame #9: libcoreclr.so`SVR::gc_heap::sort_mark_list(this=0x0000563f957e8b60) [opt]
    frame #10: libcoreclr.so`SVR::gc_heap::mark_phase(this=0x0000563f957e8b60, condemned_gen_number=0) + 6300 [opt]
    frame #11: libcoreclr.so`SVR::gc_heap::gc1(this=0x0000563f957e8b60) + 561 [opt]
    frame #12: libcoreclr.so`SVR::gc_heap::garbage_collect(this=0x0000563f957e8b60, n=<unavailable>) + 3058 [opt]
    frame #13: libcoreclr.so`SVR::gc_heap::gc_thread_function(this=0x0000563f957e8b60) + 5332 [opt]
    frame #14: libcoreclr.so`SVR::gc_heap::gc_thread_stub(arg=<unavailable>) + 54 [opt]
    frame #15: libcoreclr.so`(anonymous namespace)::CreateNonSuspendableThread(...)::$_0::operator()(void*) const + 78 [opt]
    frame #16: libcoreclr.so`(anonymous namespace)::CreateNonSuspendableThread(...)::$_0::__invoke(void*) [opt]
    frame #17: libcoreclr.so`CorUnix::CPalThread::ThreadEntry(pvParam=0x0000563f95727090) at thread.cpp:1622:16 [opt]
    frame #18: libpthread.so.0`start_thread(arg=<unavailable>) at pthread_create.c:477:8
    frame #19: libc.so.6`__clone at clone.S:95

(lldb) target variable --shlib libcoreclr.so SVR::gc_heap::settings
(SVR::gc_mechanisms) SVR::gc_heap::settings = {
  gc_index = (m_val = 800)
  condemned_generation = 0
  promotion = NO
  compaction = YES
  loh_compaction = NO
  heap_expansion = NO
  concurrent = 0
  demotion = NO
  card_bundles = YES
  gen0_reduction_count = 0
  should_lock_elevation = NO
  elevation_locked_count = 0
  elevation_reduced = NO
  minimal_gc = NO
  reason = reason_alloc_soh
  pause_mode = pause_interactive
  found_finalizers = YES
  background_p = NO
  b_state = bgc_initialized
  stress_induced = NO
  entry_memory_load = 0
  entry_available_physical_mem = 0
  exit_memory_load = 0
}
(lldb) frame select 9
frame #9: 0x00007f46aa520277 libcoreclr.so`SVR::gc_heap::sort_mark_list(this=0x0000563f957e8b60) [opt]
    7e49763ac4e8     7e4bff800028     7e4bffbfff88     7e4bffc00000 0x3fff60 (4194144)   0x400000 (4194304)
    7e49763ac710     7e4c00400028     7e4c007ffda0     7e4c00800000 0x3ffd78 (4193656)   0x400000 (4194304)
    7e49763ac9f0     7e4c01400028     7e4c017fff80     7e4c01800000 0x3fff58 (4194136)   0x400000 (4194304)
    7e49763acb60     7e4c01c00028     7e4c01ffffd0     7e4c02000000 0x3fffa8 (4194216)   0x400000 (4194304)
    7e49763ad1d8     7e4c04000028     7e4c043fffe0     7e4c04400000 0x3fffb8 (4194232)   0x400000 (4194304)
    7e49763ad400     7e4c04c00028     7e4c04fffc80     7e4c05000000 0x3ffc58 (4193368)   0x400000 (4194304)
    7e49763ada78     7e4c07000028     7e4c073fff88     7e4c07400000 0x3fff60 (4194144)   0x400000 (4194304)
    7e49763ae1a8     7e4c09800028     7e4c09c00000     7e4c09c00000 0x3fffd8 (4194264)   0x400000 (4194304)
    7e49763ae540     7e4c0ac00028     7e4c0afffca0     7e4c0b000000 0x3ffc78 (4193400)   0x400000 (4194304)
    7e49763ae820     7e4c0bc00028     7e4c0bc954b8     7e4c0bcb6000 0x95490 (611472)     0xb6000 (745472)
    7e49763aeb00     7e4c0cc00028     7e4c0cfffe60     7e4c0d000000 0x3ffe38 (4193848)   0x400000 (4194304)
    7e49763af2e8     7e4c0f800028     7e4c0fbffea0     7e4c0fc00000 0x3ffe78 (4193912)   0x400000 (4194304)
    7e49763af3a0     7e4c0fc00028     7e4c0fd4bcf8     7e4c0fd6c000 0x14bcd0 (1359056)   0x16c000 (1490944)
    7e49763af510     7e4c10400028     7e4c107ffdb8     7e4c10800000 0x3ffd90 (4193680)   0x400000 (4194304)
    7e49763af738     7e4c11000028     7e4c1111fd28     7e4c11140000 0x11fd00 (1178880)   0x140000 (1310720)
    7e49763afad0     7e4c12400028     7e4c12695f18     7e4c126b6000 0x295ef0 (2711280)   0x2b6000 (2842624)
    7e49763afb88     7e4c12800028     7e4c12bffc40     7e4c12c00000 0x3ffc18 (4193304)   0x400000 (4194304)
    7e49763afdb0     7e4c13400028     7e4c137f32b0     7e4c13800000 0x3f3288 (4141704)   0x400000 (4194304)
    7e49763b0370     7e4c15400028     7e4c157fff80     7e4c15800000 0x3fff58 (4194136)   0x400000 (4194304)
    7e49763b04e0     7e4c15c00028     7e4c15fffe28     7e4c16000000 0x3ffe00 (4193792)   0x400000 (4194304)
    7e49763b07c0     7e4c16c00028     7e4c16fffcd0     7e4c17000000 0x3ffca8 (4193448)   0x400000 (4194304)
    7e49763b0fa8     7e4c19800028     7e4c19bfff48     7e4c19c00000 0x3fff20 (4194080)   0x400000 (4194304)
    7e49763b1060     7e4c19c00028     7e4c19fff6a8     7e4c1a000000 0x3ff680 (4191872)   0x400000 (4194304)
    7e49763b1340     7e4c1ac00028     7e4c1affff80     7e4c1b000000 0x3fff58 (4194136)   0x400000 (4194304)
    7e49763b14b0     7e4c1b400028     7e4c1b7fdab0     7e4c1b800000 0x3fda88 (4184712)   0x400000 (4194304)
    7e49763b1568     7e4c1b800028     7e4c1bbffec8     7e4c1bc00000 0x3ffea0 (4193952)   0x400000 (4194304)
    7e49763b1900     7e4c1cc00028     7e4c1cc3c638     7e4c1cc5d000 0x3c610 (247312)     0x5d000 (380928)
    7e49763b21a0     7e4c1fc00028     7e4c1fffe9d0     7e4c20000000 0x3fe9a8 (4188584)   0x400000 (4194304)
    7e49763b2760     7e4c21c00028     7e4c21e7fe18     7e4c21ea0000 0x27fdf0 (2620912)   0x2a0000 (2752512)
    7e49763b4368     7e4c2b800028     7e4c2bbff9c8     7e4c2bc00000 0x3ff9a0 (4192672)   0x400000 (4194304)
    7e49763b4648     7e4c2c800028     7e4c2cbffe88     7e4c2cc00000 0x3ffe60 (4193888)   0x400000 (4194304)
    7e49763b4cc0     7e4c2ec00028     7e4c2edbcd28     7e4c2eddd000 0x1bcd00 (1821952)   0x1dd000 (1953792)
    7e49763b5338     7e4c31000028     7e4c313fed00     7e4c31400000 0x3fecd8 (4189400)   0x400000 (4194304)
    7e49763b6028     7e4c35800028     7e4c35a74b38     7e4c35a95000 0x274b10 (2575120)   0x295000 (2707456)
    7e49763b60e0     7e4c35c00028     7e4c35fffdc8     7e4c36000000 0x3ffda0 (4193696)   0x400000 (4194304)
    7e49763b6198     7e4c36000028     7e4c3624c390     7e4c3626d000 0x24c368 (2409320)   0x26d000 (2543616)
    7e49763b6a38     7e4c39000028     7e4c393ffd98     7e4c39400000 0x3ffd70 (4193648)   0x400000 (4194304)
    7e49763b6c60     7e4c39c00028     7e4c39fffd20     7e4c3a000000 0x3ffcf8 (4193528)   0x400000 (4194304)
    7e49763b6f40     7e4c3ac00028     7e4c3afffc48     7e4c3b000000 0x3ffc20 (4193312)   0x400000 (4194304)
Large object heap
         segment            begin        allocated        committed allocated size       committed size
    7e4976394c48     7e4b7c800028     7e4b7db11890     7e4b7db32000 0x1311868 (19994728) 0x1332000 (20127744)
Pinned object heap
         segment            begin        allocated        committed allocated size       committed size
    7e49763a0898     7e4bbe000028     7e4bbe020020     7e4bbe021000 0x1fff8 (131064)     0x21000 (135168)
------------------------------
GC Allocated Heap Size:    Size: 0x91862000 (2441486336) bytes.
GC Committed Heap Size:    Size: 0x92d28000 (2463268864) bytes.
(lldb) image lookup --verbose --address $pc
      Address: libcoreclr.so[0x000000000056227c] (libcoreclr.so.PT_LOAD[1]..text + 3771388)
      Summary: libcoreclr.so`SVR::gc_heap::sort_mark_list() + 620 [inlined] SVR::FATAL_GC_ERROR() + 5
               libcoreclr.so`SVR::gc_heap::sort_mark_list() + 615 at 10804
       Module: file = "/usr/share/dotnet/shared/Microsoft.NETCore.App/10.0.11/libcoreclr.so", arch = "x86_64"
  CompileUnit: id = {0x0000018b}, file = "/__w/1/s/src/runtime/src/coreclr/gc/gcsvr.cpp", language = "c++11"
     Function: id = {0x7fffffff05838c22}, name = "SVR::gc_heap::sort_mark_list()", mangled = "_ZN3SVR7gc_heap14sort_mark_listEv", range = [0x00007f46aa520010-0x00007f46aa5204cf)
     FuncType: id = {0x7fffffff05838c22}, byte-size = 0, line = 3269, compiler_type = "size_t (void)"
       Blocks: id = {0x7fffffff05838c22}, range = [0x7f46aa520010-0x7f46aa5204cf)
               id = {0x7fffffff05838e2b}, ranges = [0x7f46aa52020d-0x7f46aa52021b)[0x7f46aa520224-0x7f46aa5202bb)[0x7f46aa5202c3-0x7f46aa52030c)[0x7f46aa5204c5-0x7f46aa5204cf)
               id = {0x7fffffff05838f05}, range = [0x7f46aa520277-0x7f46aa52028e), name = "FATAL_GC_ERROR", line = 111, mangled = _ZN3SVR14FATAL_GC_ERROREv, demangled = SVR::FATAL_GC_ERROR()
       Symbol: id = {0x000061ed}, range = [0x00007f46aa520010-0x00007f46aa5204cf), name="SVR::gc_heap::sort_mark_list()", mangled="_ZN3SVR7gc_heap14sort_mark_listEv"
     Variable: id = {0x7fffffff05838e2f}, name = "region_limit", type = "uint8_t *", location = DW_OP_reg3 RBX, decl =  line 10797
     Variable: id = {0x7fffffff05838e63}, name = "mark_list_piece_start_ptr", type = "uint8_t ***", location = , decl =  line 10807
     Variable: id = {0x7fffffff05838c37}, name = "this", type = "SVR::gc_heap *", location = DW_OP_breg6 RBP-56, decl =
     Variable: id = {0x7fffffff05838c75}, name = "local_mark_list_index", type = "uint8_t **", location = DW_OP_reg15 R15, decl =  line 10706
     Variable: id = {0x7fffffff05838c8f}, name = "x", type = "uint8_t **", location = DW_OP_reg13 R13, decl =  line 10774
(lldb) p this->mark_list
(uint8_t **) $2 = 0x00007e4845654560
(lldb)  p this->mark_list_index
(uint8_t **) $3 = 0x00007e484565fed8
(lldb) p this->mark_list_end
(uint8_t **) $4 = 0x00007e48460bf000
(lldb) p this->heap_number
(int) $5 = 2
(lldb) register read rbx r13 r15
     rbx = 0x0000000000000000
     r13 = 0x00007e4845655a60
     r15 = 0x00007e4845658560
(lldb) memory read -f x -s 8 -c 1 $r13
0x7e4845655a60: 0x00007e4bcadf3c08

(lldb) frame select 9
frame #9: 0x00007f46aa520277 libcoreclr.so`SVR::gc_heap::sort_mark_list(this=0x0000563f957e8b60) [opt]
(lldb) frame variable
(SVR::gc_heap *) this = 0x0000563f957e8b60
(size_t) total_mark_list_size = <variable not available>

(size_t) total_ephemeral_size = <variable not available>

(uint8_t *) low = <variable not available>

(uint8_t *) high = <variable not available>

(uint8_t **) local_mark_list_index = 0x00007e4845658560
(ptrdiff_t) item_count = <variable not available>

(uint8_t **) x = 0x00007e4845655a60
(uint8_t *) region_limit = 0x0000000000000000
(size_t) region_index = <variable not available>

(uint8_t ***) mark_list_piece_end_ptr = <variable not available>

(SVR::heap_segment *) region = <variable not available>

(uint8_t ***) mark_list_piece_start_ptr = <no location, value may have been optimized out>

(lldb) frame variable region_limit x local_mark_list_index
(uint8_t *) region_limit = 0x0000000000000000
(uint8_t **) x = 0x00007e4845655a60
(uint8_t **) local_mark_list_index = 0x00007e4845658560

(lldb) frame select 9
frame #9: 0x00007f46aa520277 libcoreclr.so`SVR::gc_heap::sort_mark_list(this=0x0000563f957e8b60) [opt]
(lldb) p this->mark_list_index
(uint8_t **) $8 = 0x00007e484565fed8
(lldb) p this->mark_list
(uint8_t **) $9 = 0x00007e4845654560
(lldb) p this->heap_number
(int) $10 = 2
(lldb) gcwhere 0x00007e4bcadf3c08
Address 7e4bcadf3c08 not found in the managed heap.
(lldb) p *x
(uint8_t *) $11 = 0x00007e4bcadf3c08 "\xffffff89\xffffffa0\x02+F\x7f"
(lldb) p x[-2]
(uint8_t *) $12 = 0x00007e4bc97f9cd0 "IKW6F\x7f"
(lldb) p x[-1]
(uint8_t *) $13 = 0x00007e4bc97fc228 "IKW6F\x7f"
(lldb)  p x[1]
(uint8_t *) $14 = 0x00007e4bcb0053d8 "IKW6F\x7f"
(lldb) p x[2]
(uint8_t *) $15 = 0x00007e4bcb0066d8 "IKW6F\x7f"

(lldb) disassemble -s $pc-0x60 -c 32
libcoreclr.so`SVR::gc_heap::sort_mark_list:
    0x7f46aa52021c <+524>: cmpl   %edi, %ebp
    0x7f46aa52021e <+526>: jae    0x7f46aa520311            ; <+769>
    0x7f46aa520224 <+532>: movq   (%r13), %rax
    0x7f46aa520228 <+536>: movzbl 0x173b89(%rip), %ecx      ; SVR::gc_heap::min_segment_size_shr
    0x7f46aa52022f <+543>: shrq   %cl, %rax
    0x7f46aa520232 <+546>: movq   0x17308f(%rip), %rdx      ; SVR::seg_mapping_table
    0x7f46aa520239 <+553>: imulq  $0xb8, %rax, %rax
    0x7f46aa520240 <+560>: leaq   (%rdx,%rax), %rsi
    0x7f46aa520244 <+564>: movq   (%rdx,%rax), %rax
    0x7f46aa520248 <+568>: movq   %rax, %rdx
    0x7f46aa52024b <+571>: sarq   $0x3f, %rdx
    0x7f46aa52024f <+575>: andq   %rax, %rdx
    0x7f46aa520252 <+578>: imulq  $0xb8, %rdx, %rax
    0x7f46aa520259 <+585>: movq   0x20(%rax,%rsi), %r12
    0x7f46aa52025e <+590>: shrq   %cl, %r12
    0x7f46aa520261 <+593>: movq   (%rax,%rsi), %rbx
    0x7f46aa520265 <+597>: leaq   0x170f5c(%rip), %rax      ; g_gc_lowest_address
    0x7f46aa52026c <+604>: movq   (%rax), %r14
    0x7f46aa52026f <+607>: shrq   %cl, %r14
    0x7f46aa520272 <+610>: testq  %rbx, %rbx
    0x7f46aa520275 <+613>: jne    0x7f46aa52028e            ; <+638>
    0x7f46aa520277 <+615>: callq  0x7f46aa2aa0f0            ; GCToOSInterface::DebugBreak at gcenv.unix.cpp:501
->  0x7f46aa52027c <+620>: movl   $0x80131506, %edi         ; imm = 0x80131506
    0x7f46aa520281 <+625>: callq  0x7f46aa4535a0            ; GCToEEInterface::HandleFatalError
    0x7f46aa520286 <+630>: movq   -0x30(%rbp), %r11
    0x7f46aa52028a <+634>: movq   -0x38(%rbp), %r10
    0x7f46aa52028e <+638>: subq   %r14, %r12
    0x7f46aa520291 <+641>: movq   0xf0(%r10), %rcx
    0x7f46aa520298 <+648>: movq   0xf8(%r10), %rax
    0x7f46aa52029f <+655>: movq   %r13, (%rcx,%r12,8)
    0x7f46aa5202a3 <+659>: cmpq   %rbx, (%r13)
    0x7f46aa5202a7 <+663>: jae    0x7f46aa520217            ; <+519>

Is this a regression?

this works fine on .NET 8 (SDK 8.0.415). The failure only appears after moving to .NET 10.0.11

Are there any workarounds?

No response

dotnet --info output

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  20.04
 OS Platform: Linux
 RID:         linux-x64
 Base Path:   /usr/share/dotnet/sdk/10.0.303/

.NET workloads installed:
There are no installed workloads to display.
Configured to use workload sets when installing new manifests.
No workload sets are installed. Run "dotnet workload restore" to install a workload set.

Host:
  Version:      10.0.11
  Architecture: x64
  Commit:       e2f47b0110

.NET SDKs installed:
  10.0.303 [/usr/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.24 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 10.0.11 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.24 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 10.0.11 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  None

Environment variables:
  DOTNET_DbgEnableMiniDump                 [1]
  DOTNET_DbgMiniDumpName                   [/data/core.%p]
  DOTNET_DbgMiniDumpType                   [2]
  DOTNET_GCConserveMemory                  [9]
  DOTNET_GCHighMemPercent                  [0x50]
  DOTNET_RUNNING_IN_CONTAINER              [true]
  DOTNET_ThreadPool_UnfairSemaphoreSpinLimit [0x0]
  DOTNET_VERSION                           [6.0.24]
  Detected COMPlus_* environment variable(s). Consider transitioning to DOTNET_* equivalent.

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

IDE version

No response

Other details

No response

[lingchenyu]

This is a runtime GC issue (FATAL_GC_ERROR in SVR::gc_heap::sort_mark_list),
not an SDK issue. Could someone please transfer this to dotnet/runtime,
area-GC-coreclr? Filed here by mistake. Thanks!

[lingchenyu]

---

Update: root cause found on our side — closing.

After further investigation, this turned out to be a stale dependency in our own application, not a runtime/GC bug.

When we migrated the project from .NET 8 to .NET 10, the Refit package was left at 8.0 (the .NET 8-era version) and was never upgraded as part of the migration. That old version interacts badly with our setup on .NET 10 (Server GC + periodic hot reload via collectible AssemblyLoadContext). After upgrading Refit to the current version with .NET 10 support and rebuilding, the FATAL_GC_ERROR in SVR::gc_heap::sort_mark_list no longer reproduces — the same staging workload that previously crashed after ~4 hours has been running stably since.

Apologies for the noise, and thanks to anyone who spent time looking at this. Closing as resolved — no action needed on the SDK/runtime side.

---