[Questions] Classic queue pread eof crash on 4.1.1 (continuation of #12367 and #13758)

[gomoripeti]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I run RabbitMQ 4.x, the only series currently covered by community support
• I promise to provide all relevant information (versions, logs from all nodes, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

4.1.1

Erlang version used

27.3.x

Operating system (distribution) used

ubuntu

How is RabbitMQ deployed?

Debian package

rabbitmq-diagnostics status output

See https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics

# PASTE OUTPUT HERE, BETWEEN BACKTICKS

Logs from node 1 (with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 2 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 3 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

rabbitmq.conf

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location

# PASTE rabbitmq.conf HERE, BETWEEN BACKTICKS

Steps to deploy RabbitMQ cluster

Steps to reproduce the behavior in question

unknown

advanced.config

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location

# PASTE advanced.config HERE, BETWEEN BACKTICKS

Application code

# PASTE CODE HERE, BETWEEN BACKTICKS

Kubernetes deployment file

# Relevant parts of K8S deployment that demonstrate how RabbitMQ is deployed
# PASTE YAML HERE, BETWEEN BACKTICKS

What problem are you trying to solve?

A single node cluster was upgraded to RabbitMQ 4.1.1 about a week ago. Since then the below crash happened two times (on the same queue).

    supervisor: {<0.633.0>,rabbit_amqqueue_sup}
    errorContext: child_terminated
    reason: {function_clause,
                [{rabbit_msg_store,reader_pread_parse,
                     [[[[ eof ]]]],
                     [{file,"src/rabbit_msg_store.erl"},{line,686}]},
                 {rabbit_msg_store,reader_pread,2,
                     [{file,"src/rabbit_msg_store.erl"},{line,682}]},
                 {rabbit_msg_store,read_from_disk,2,
                     [{file,"src/rabbit_msg_store.erl"},{line,642}]},
                 {rabbit_msg_store,client_read3,2,
                     [{file,"src/rabbit_msg_store.erl"},{line,634}]},
                 {rabbit_variable_queue,with_msg_store_state,3,
                     [{file,"src/rabbit_variable_queue.erl"},{line,1057}]},
                 {rabbit_variable_queue,read_msg,5,
                     [{file,"src/rabbit_variable_queue.erl"},{line,1291}]},
                 {rabbit_variable_queue,fetch,2,
                     [{file,"src/rabbit_variable_queue.erl"},{line,573}]},
                 {rabbit_priority_queue,'-fetch/2-fun-0-',4,
                     [{file,"src/rabbit_priority_queue.erl"},{line,256}]}]}

This is not the same crash as reported in #13758 and addressed in #13771 (which happens when trying to read multiple messages from disk into memory via rabbit_msg_store:read_many_file2/4 triggered by rabbit_variable_queue:remove_by_predicate/2)
But more similar to one reported in #12367 (when a single message is fetched, I assume, to be sent to a consumer)

About the setup:
The affected queue has: max-priority 2, expires 2 days
bound to a topic exchange with <key1>.#
topic exchange has 1 more binding with <key1>.#.<key2>.#

This server also experienced the pread within read_many crash addressed by #13771 before upgrading to 4.1.1, so it seems to have a special traffic pattern that is prone to expose issues with the shared message store. I might be able to add some tracing to catch additional info to help get closer to the root cause. What kind of info would be useful?

[gomoripeti]

The same server crashed several times in the past month. Both single read and read_many type of crashes occred. We have backup of the mnesia dir and extra logging for some of the crashes.

The last crash that happened on 2025-08-26 06:26:35.755199+00:00 was particularly interresting.

Logging the file_summary record of an affected rdq file, valid size is ~6MB but file size is only ~4MB, (This is the first time I saw such inconsistent file_summary record)

[{file_summary,340809,6614143,undefined,undefined,4295991,false,undefined}]},

File size is correct, same as on disk

ls
-rw-r----- 1 rabbitmq rabbitmq 4295991 Aug 26 04:40:09 msg_stores/vhosts/86WAU76IUR0OIES3J9T8DFV6Y/msg_store_persistent/340809.rdq

There are zero-holes in the rdq file so the file must have been compacted and truncated (probably at 4:40:09).

These are the locations given to pread. The last 5 entries are beyond the file size (and even beyond valid_size)

[{130491,11870},
 {172663,12598},
 {203369,71595},
 {309250,12336},
 {336103,12601},
 {366857,67359},
 {1035857,10755},
 {1719886,9037},
...
 {3892806,9900},
 {3947218,30117},
 {3992133,29293},
 {4062239,49681},
 {4195506,21161},
 {9458386,23505},
 {9495881,24292},
 {9907941,9380},
 {9934406,81047},
 {12499180,12630}]},

Unfortunately (or fortunately for the user) there were config changes made so the crash will likely won't happen again for further debugging.

The inconsistnt file_summary record is the most disturbing for me.

[lukebakken]

cc @lhoguin

@gomoripeti thanks as always for your detailed investigations. Have you seen these crashes on any upgraded brokers that did not exhibit the crashes while running 3.13? I'm assuming that info may not even be available, but I'm curious. Have a great weekend!

[gomoripeti]

Do you ask if it's a new bug introduced in 4.x?
We don't have the logs way back to answer that question with an automatic query. I could only read through conversations and emails, but this is almost impossible. I can ask around. This particular server was already created on 4.0.5

[lhoguin]

Compaction/truncation doesn't change the valid_total_size so perhaps it was incorrect since node start. Could that be the case?

Otherwise I don't think I have much to suggest except try to reproduce the workload and the bug.

[lhoguin]

PS: don't hesitate to tag me directly on CQ issues I'll answer quicker. :-)

[gomoripeti]

Node was restarted many weeks ago and the rdq files are fresh in the sense that:
The traffic pattern is that there is a flood of publishes to this problematic queue, the queue grows to several 100K messages length, and consumed to empty within a few hours. My colleague tried to reproduce the issue with this pattern without success.

I looked at the routing timestamp in each message. In the problematic rdq file they are 4:39:27-4:39:29. In the following rdq file earliest timestamp is 4:39:31. So the rdq file was freshly created about a minute before it was compacted and filled up within a few seconds.
Actually the valid_size might be consistent with the ETS index. Maybe it was the compaction that had a scanning error and miscalculated the end of file. Unfortunately what was the content of the rdq file before truncation cannot be known.

So I think there is still a bug in the msg_store, but it might be so rare that it might never resurface in which case it remains a purely theoretical issue :)

(thanks Loic for encouraging a direct ping :) )

[trvrnrth]

We are seeing the same issue reported here across a fleet of standalone rabbit 4.1.1 instances with multiple daily crashes across several different queues.

I cannot determine a pattern of behaviour leading to the crashes but perhaps some of the following information may be helpful:

• all impacted queues are durable classic queues (we do not run any quorum queues on these instances)
• self-recovery is not always successful, requiring a delete and re-create of the queue to restore normal operation
• we see crashes more often on a set of priority queues
• the worst affected queues have a pattern of relatively fast publishes followed by a period of slow consumption with some message processing taking upwards of 1 hour by design before ack
• there is no easily discernable correlation between the time of crash and any increased/decreased publish/consume activity

Unfortunately I cannot share full logs as they may contain sensitive data, but I am happy to provide any extra context that may be valuable to investigation of the issue.

[michaelklishin]

@trvrnrth are the non-affected queues also priority queues (have x-max-priority set)? Because internally priority queues are meaningfully different from regular classic queues.

[trvrnrth]

@michaelklishin Unfortunately there's no clear behaviour split:

• Most queues on the instances are stable.
• The "source" queue I describe above is also a priority queue which sees broadly the same payloads and is stable.
• There is another non-priority queue on the instances which we also see crash, although less frequently. It uses the default consumer timeout.
• We also have a separate fleet of instances which host only prioritized queues for a different purpose. We have seen only a couple of crashes on these over the same time period.

[trvrnrth]

One thing which does hold true is that the 3 queues we see crashes on frequently are the only ones on the instances that typically have a large backlog. This will typically be in the 10s of thousands of ready messages and 1200 unacked.

[lukebakken]

1200 unacked

So your consumer prefetch is 100?

[trvrnrth]

1200 unacked

So your consumer prefetch is 100?

Yes, that's correct.

[lukebakken]

In case anyone has suggestions for improvements, or behavior changes, here's what I'm currently trying to use to reproduce the issue:

https://github.com/lukebakken/rabbitmq-server-14181/blob/main/targeted_bug_reproduction.py

@trvrnrth - here's what the publish / ack rates look like when I run 8 instances of the above program in my env:

Queue overview:

I'm curious how close these numbers are to what you see in your env, if they're close at all.

I'm planning to leave the test running for several hours. I'm testing against main using make run-broker.

[lukebakken]

Sorry, it wasn't clear in my second sentence that I was describing the second run, when I started the test script and PerfTest. In that run, the eof error was not triggered.

[mkuratczyk]

I'm trying to use your method on my side. I've had a bunch of timeouts already but no function_clauses so far. I will report back as well

[lukebakken]

I still haven't had success when trying to "speed up" the process with shorter consumer timeouts. I'm back to using the default 30-minute consumer timeout, waiting for two full "cycles" of timeouts to happen (60+) minutes, and only then adding additional load to try and reproduce.

[mkuratczyk]

My machine is resistant to this bug. I had spent a few day trying to reproduce it before and today I had your script running for ~8 hours + perf-test (on and off, sometimes for a a few minutes, sometimes for a few hours) and nothing. Not a single crash. 🤷

[lukebakken]

Yep, I've only reproduced it the one time. The stars must have aligned.

[lukebakken]

@lhoguin I have a question about some code in rabbit_msg_store. In this code...

https://github.com/rabbitmq/rabbitmq-server/blob/main/deps/rabbit/src/rabbit_msg_store.erl#L628-L655

...between the time that index_lookup/2 is called on line 633, and reader_pread/2 is called on line 643, is it possible for compaction to have modified the message store file such that the data read via the index on line 633 is no longer valid, and thus eof is returned for some or all file positions passed to file:pread/2?

The ETS table used for the index is shared with the GC process, so I'm wondering if there could be a race condition between updating that ETS table during GC, and how the ETS table is used in client_read3/2 and elsewhere.

[lukebakken]

@lhoguin since I can't seem to reproduce this again, but it does occur in the wild, what sort of mitigation might be put into place to avoid a queue crash? Can the eof case be handled and those reads re-tried if calling index_lookup/2 returns different positions after an eof?

What logging could be added to assist in finding the root cause of this crash?