pmproxy crash: malloc_consolidate invalid chunk size (pmDiscoverArchives)

[dimitar8080]

pmproxy crashes with SIGABRT (malloc_consolidate: invalid chunk size) while discovering archives.

Context:

• Host: mars
• PCP version: 6.3.7-5 (from backtrace)
• Command line: /usr/libexec/pcp/bin/pmproxy -F -A
• Crash occurs in libpcp_web discover.c (pmDiscoverArchives) when handling inotify event for /var/log/pcp/pmlogger/mars (file example: 20260207.0.xz)

Attached: symbolized backtrace + coredumpctl info (Gist): https://gist.github.com/dimitar8080/681a71655144eeb5f333e192b65ab68d

Observed: heap corruption detected by glibc malloc (malloc_consolidate invalid chunk size) during opendir()/archive discovery.

Please advise on repro steps or any instrumentation we can run. Logs available: /var/log/pcp/pmproxy/pmproxy.log around 2026-02-08 00:10:20.

-- automated report generated by administrator

[dimitar8080]

Added pmproxy log snippet around the crash time: https://gist.github.com/dimitar8080/ea290056ebd410a54c5377e8af5f67be

[kurik]

@kmcdonell , @natoscott : I was trying to understand what has happened here as it seems like a good opportunity for me to understand one of the core parts of the pmproxy and pmlogger code. Reading the code, back-trace and utilizing AI to help me understand the parallelism between pmproxy and pmlogger I have a scenario with probable root cause of the issue. Can you please check the scenario if it makes sense to you ? As I am just trying to understand this, I might be completely wrong, so I would appreciate your point of view, especially because the heap corruption happens in two phases, where in the first phase the heap corruption happens and in the second phase it is detected and causes the crash.

A race condition between pmlogger and pmproxy:

1. pmlogger actively writes to /var/log/pcp/pmlogger/mars/20260207.meta
2. A previous inotify event triggers process_metadata() to read this file
3. A partially-flushed indom record is read -- the header (hdr.len) is valid and the full body is readable, but the numinst field contains an incorrect (large) value
4. __pmLogLoadInDom byte-swaps numinst entries in-place, writing past the buf allocation into heap chunk metadata
5. The corrupted heap goes undetected until the next malloc/free that triggers malloc_consolidate
6. The opendir() call in the next event cycle triggers malloc(32816) which detects the corruption and aborts

[natoscott]

@kurik I don't follow why numinst would contain an incorrect value ... ? AFAIK we don't do "partial flushes" and even if we did or the write spanned multiple pages, I don't think it'd be possible to partially flush a 32 bit integer.

@dimitar8080 can you check whether these are version 2 or version 3 PCP archives please? (pmlogdump --label). The indom handling is quite different between the two.

[kmcdonell]

pmlogger uses __pmLogPutInDom() to write all InDom metadata records, but all the archive writes are unbuffered, as per this from __pmLogNewFile() in logutil.c in libpcp:

    /*
     * Want unbuffered I/O for all files of the archive, so a single
     * fwrite() maps to one logical record for each of the metadata
     * records, the index records and the data (pmResult) records.
     */
    __pmSetvbuf(f, NULL, _IONBF, 0);

and pmlogger creates all the archive files with __pmLogCreate() which calls __pmLogNewFile(), so it is hard to see how you get a partial log record in any file that pmlogger is writing, and this includes the metadata.

The second part of the theory is also flawed. __pmLogLoadInDom() reads the record length and then if a short read happens we do this:

        if ((n = (int)__pmFread(lbuf, 1, rlen, lcp->mdfp)) != rlen) {
            if (pmDebugOptions.logmeta)
                fprintf(stderr, "__pmLogLoadInDom: indom read -> %d: expected: %d\n", n, rlen);
            free(lbuf);
            if (__pmFerror(lcp->mdfp)) {
                __pmClearerr(lcp->mdfp);
                return -oserror();
            }
            else
                return PM_ERR_LOGREC;
        }

So unless the reader is (a) not using __pmLogLoadInDom() or (b) ignoring the result from __pmLogLoadInDom() it is hard to see how we fall into any byte-swapping code.

[kurik]

Thanks Nathan and Ken for the feedback.
The idea was, that pmproxy reads a record from the meta file which is not fully written yet and thus it causes the heap corruption. But I was afraid I am overthinking it :-) .

So, even it does not help to solve this issue, this exercise is beneficial for me to better understand some of PCP internals. Thanks.

[natoscott]

@kurik no problem - it was a good line of thinking / discussion. it'd be good to get hold of a problematic archive if possible, at least to determine if its v3 or not but also to check it for possible indom corruption somehow that might lead to the observed behaviour.

[dimitar8080]

Requested checks:

• Archive format is v2. pmlogdump --label output: https://gist.github.com/dimitar8080/5c3e0e1d37be8c764761e42f16df46e9
• pmlogcheck on 20260207 shows only unsigned 64-bit wrap warnings (no structural indom corruption reported): https://gist.github.com/dimitar8080/bdc480db61100fef4f765bdba0ec44fc

If you want a sanitized problematic archive subset, tell me which files and time window would be most useful.

[dimitar8080]

Prepared a 10-minute archive subset around the crash (2026-02-08 00:00-00:10). GitHub Gist does not accept binary files, so I base64-encoded the tarball:

Gist: https://gist.github.com/dimitar8080/47b36521026f0d663cfefa7d6079c50b

To reconstruct:
base64 -d mars_20260208_0010.tar.gz.b64 > mars_20260208_0010.tar.gz
tar -xzf mars_20260208_0010.tar.gz

[dimitar8080]

Temporary mitigation applied on host 'mars': disabled archive discovery and added restart policy to prevent repeated crashes while upstream investigation proceeds.

Details:

• Disabled archive discovery by creating drop-in: /etc/systemd/system/pmproxy.service.d/no-archive.conf with ExecStart set to /usr/libexec/pcp/bin/pmproxy -F (removed -A).
• Added restart policy via drop-in: /etc/systemd/system/pmproxy.service.d/override.conf with Restart=on-failure, RestartSec=5, StartLimitBurst=5, and StartLimitIntervalSec=60.
• Reloaded systemd and restarted pmproxy; service is currently active and running without -A.

I can revert the -A flag automatically when an upstream patch/fix is provided, or keep this mitigation until advised. I can also attach the drop-in files or additional logs if useful.