fix(telemetry): resolve the process cgroup path and honor ancestor limits [INFP-589]

The v2 limit files were read only at /sys/fs/cgroup, which is correct under
a private cgroup namespace (the modern Docker/K8s default, where the
apparent root is the container's cgroup) but wrong without one — older
runtimes on v2, cgroupns: host, systemd services with unit limits. The v2
root cgroup carries no cpu.max/memory.max, so a limited component reported
no CPU assignment and the whole host's memory as its capacity, breaking the
licensing audit in exactly those environments.

The reader now resolves its own cgroup from the 0:: line of
/proc/self/cgroup and consults every level up to the root, taking the most
restrictive limit; memory usage is read at the level holding the effective
limit, since an ancestor limit is shared with siblings. Resolution failures
fall back to the previous root-only read, and v1 keeps the root read
(runtimes bind-mount the container's own v1 controllers there). Limits
above a private namespace root remain invisible by construction.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: saltas888

backend/infrahub/telemetry/resources.py

@@ -2,9 +2,15 @@
 
 Each process reports the logical CPU count it can see, the CPU quota enforced on
 it by its container control group (``None`` when nothing is enforced), and its
-memory capacity and free memory. The container control group is consulted first
-(cgroup v2, then v1); a host without one — a developer laptop, an unusual mount —
+memory capacity and free memory. The process's own control group is resolved
+from ``/proc/self/cgroup`` and every level up to the root is consulted, because
+a limit may be enforced on an ancestor — under a private cgroup namespace (the
+modern container default) that path collapses to the apparent root, while a host
+namespace or a systemd service exposes the full hierarchy. cgroup v2 is read
+first, then v1; a host with neither — a developer laptop, an unusual mount —
 falls back to the whole-host figures from psutil and reports no CPU quota.
+Limits above a private namespace root (for example a pod-level limit when the
+container itself has none) are invisible from inside and cannot be reported.
 
 The values that cannot change for the lifetime of a process (the host identifier,
 the logical CPU count, the enforced CPU quota and the memory capacity) are read
@@ -20,6 +26,7 @@
 import math
 import socket
 from dataclasses import dataclass
+from operator import itemgetter
 from pathlib import Path
 from typing import TYPE_CHECKING
 
@@ -30,6 +37,7 @@
     from collections.abc import Iterable
 
 CGROUP_ROOT = Path("/sys/fs/cgroup")
+PROC_SELF_CGROUP = Path("/proc/self/cgroup")
 
 # The default CPU period the kernel uses when a cgroup v2 ``cpu.max`` line omits it.
 _DEFAULT_CPU_PERIOD_US = 100000
@@ -132,56 +140,108 @@ def _quota_to_cores(quota: int, period: int) -> int | None:
     return math.ceil(quota / period)
 
 
-def _read_cgroup_cpu_quota(cgroup_root: Path) -> int | None:
-    """Return the enforced CPU limit in whole cores, or ``None`` when unbounded.
+def _own_cgroup_dirs(cgroup_root: Path, proc_cgroup: Path) -> list[Path]:
+    """Return this process's cgroup directory and its ancestors, leaf first.
 
-    cgroup v2 ``cpu.max`` ("<quota> <period>", or "max <period>" when unbounded)
-    is read first; a host on cgroup v1 uses the ``cpu.cfs_quota_us`` /
-    ``cpu.cfs_period_us`` pair, where a quota of ``-1`` means unbounded.
+    Under a private cgroup namespace (the modern container default) the process
+    sits at the apparent root and the list collapses to ``[cgroup_root]``. Without
+    one — an older runtime, an explicit host namespace, a systemd service — the
+    ``0::<path>`` line of the proc file locates the real cgroup, and every level
+    up to the root is returned because a limit may be enforced on any ancestor.
+    An unreadable proc file, a missing v2 line, or a path that does not exist
+    under the root all fall back to ``[cgroup_root]``, preserving the plain read.
     """
-    v2_line = _read_text_file(cgroup_root / "cpu.max")
-    if v2_line is not None:
-        parts = v2_line.split()
-        if not parts or parts[0] == "max":
-            return None
-        try:
-            v2_quota = int(parts[0])
-            v2_period = int(parts[1]) if len(parts) > 1 else _DEFAULT_CPU_PERIOD_US
-        except ValueError:
-            return None
-        return _quota_to_cores(quota=v2_quota, period=v2_period)
+    content = _read_text_file(proc_cgroup)
+    if content is None:
+        return [cgroup_root]
+    for line in content.splitlines():
+        if not line.startswith("0::"):
+            continue
+        relative = line[3:].strip().lstrip("/")
+        if not relative or ".." in relative.split("/"):
+            return [cgroup_root]
+        leaf = cgroup_root / relative
+        if not leaf.is_dir():
+            return [cgroup_root]
+        dirs = [leaf]
+        for parent in leaf.parents:
+            dirs.append(parent)
+            if parent == cgroup_root:
+                break
+        return dirs
+    return [cgroup_root]
+
+
+def _parse_cpu_max(line: str) -> int | None:
+    """Parse one cgroup v2 ``cpu.max`` line ("<quota> <period>", "max" = unbounded)."""
+    parts = line.split()
+    if not parts or parts[0] == "max":
+        return None
+    try:
+        quota = int(parts[0])
+        period = int(parts[1]) if len(parts) > 1 else _DEFAULT_CPU_PERIOD_US
+    except ValueError:
+        return None
+    return _quota_to_cores(quota=quota, period=period)
+
 
-    quota = _read_int_file(cgroup_root / "cpu" / "cpu.cfs_quota_us")
-    period = _read_int_file(cgroup_root / "cpu" / "cpu.cfs_period_us")
+def _read_cgroup_cpu_quota(cgroup_dirs: list[Path]) -> int | None:
+    """Return the enforced CPU limit in whole cores, or ``None`` when unbounded.
+
+    Every level of the process's cgroup path may carry a v2 ``cpu.max``; the
+    effective limit is the most restrictive one. A hierarchy with no readable
+    ``cpu.max`` at any level is treated as cgroup v1, whose ``cpu.cfs_quota_us``
+    / ``cpu.cfs_period_us`` pair (quota ``-1`` = unbounded) lives at the
+    controller mount root inside a container.
+    """
+    v2_lines = [line for directory in cgroup_dirs if (line := _read_text_file(directory / "cpu.max")) is not None]
+    if v2_lines:
+        cores = [value for line in v2_lines if (value := _parse_cpu_max(line)) is not None]
+        return min(cores) if cores else None
+
+    root = cgroup_dirs[-1]
+    quota = _read_int_file(root / "cpu" / "cpu.cfs_quota_us")
+    period = _read_int_file(root / "cpu" / "cpu.cfs_period_us")
     if quota is None or period is None:
         return None
     return _quota_to_cores(quota=quota, period=period)
 
 
-def _read_cgroup_memory_limit(cgroup_root: Path) -> tuple[int | None, Path | None]:
+def _read_cgroup_memory_limit(cgroup_dirs: list[Path]) -> tuple[int | None, Path | None]:
     """Return ``(limit_bytes, current_usage_path)`` for the memory control group.
 
-    ``limit_bytes`` is ``None`` when memory is unbounded or no control group is
-    readable; ``current_usage_path`` points at the file holding current usage so
-    free memory can be recomputed cheaply on each heartbeat. cgroup v2
-    ``memory.max`` ("max" when unbounded) takes precedence over the v1
-    ``memory.limit_in_bytes`` value, which reports a near-``INT64_MAX`` sentinel
-    when unbounded.
+    Every level of the process's cgroup path may carry a v2 ``memory.max``
+    ("max" = unbounded); the effective limit is the smallest, and usage is read
+    from that same level — an ancestor limit is shared with siblings, so free
+    memory within it is the limit minus the whole subtree's usage. A hierarchy
+    with no readable ``memory.max`` at any level is treated as cgroup v1, whose
+    ``memory.limit_in_bytes`` reports a near-``INT64_MAX`` sentinel when
+    unbounded. ``(None, None)`` means no limit is enforced anywhere.
     """
-    v2_max = _read_text_file(cgroup_root / "memory.max")
-    if v2_max is not None:
-        if v2_max == "max":
-            return None, None
+    limits: list[tuple[int, Path]] = []
+    v2_seen = False
+    for directory in cgroup_dirs:
+        raw = _read_text_file(directory / "memory.max")
+        if raw is None:
+            continue
+        v2_seen = True
+        if raw == "max":
+            continue
         try:
-            v2_limit = int(v2_max)
+            limits.append((int(raw), directory))
         except ValueError:
+            continue
+    if v2_seen:
+        if not limits:
             return None, None
-        return v2_limit, cgroup_root / "memory.current"
+        limit, directory = min(limits, key=itemgetter(0))
+        return limit, directory / "memory.current"
 
-    limit = _read_int_file(cgroup_root / "memory" / "memory.limit_in_bytes")
-    if limit is None or limit >= _CGROUP_MEMORY_UNLIMITED_THRESHOLD:
+    root = cgroup_dirs[-1]
+    v1_limit = _read_int_file(root / "memory" / "memory.limit_in_bytes")
+    if v1_limit is None or v1_limit >= _CGROUP_MEMORY_UNLIMITED_THRESHOLD:
         return None, None
-    return limit, cgroup_root / "memory" / "memory.usage_in_bytes"
+    return v1_limit, root / "memory" / "memory.usage_in_bytes"
 
 
 def _host_memory_total() -> int | None:
@@ -200,17 +260,19 @@ class ProcessResources:
     only free memory, which moves with usage.
     """
 
-    def __init__(self, cgroup_root: Path = CGROUP_ROOT) -> None:
+    def __init__(self, cgroup_root: Path = CGROUP_ROOT, proc_cgroup: Path = PROC_SELF_CGROUP) -> None:
         self._cgroup_root = cgroup_root
+        self._proc_cgroup = proc_cgroup
         self._static: _StaticResources | None = None
 
     def _read_static(self) -> _StaticResources:
-        memory_limit, memory_current_path = _read_cgroup_memory_limit(self._cgroup_root)
+        cgroup_dirs = _own_cgroup_dirs(cgroup_root=self._cgroup_root, proc_cgroup=self._proc_cgroup)
+        memory_limit, memory_current_path = _read_cgroup_memory_limit(cgroup_dirs)
         memory_total = memory_limit if memory_limit is not None else _host_memory_total()
         return _StaticResources(
             host=socket.gethostname(),
             processor_available=psutil.cpu_count(logical=True),
-            processor_assigned=_read_cgroup_cpu_quota(self._cgroup_root),
+            processor_assigned=_read_cgroup_cpu_quota(cgroup_dirs),
             memory_total=memory_total,
             memory_limit=memory_limit,
             memory_current_path=memory_current_path,

backend/tests/unit/telemetry/test_resources.py

@@ -158,6 +158,179 @@ def test_cgroup_memory(case: MemoryCase, tmp_path: Path) -> None:
         assert reading.memory_available == case.expected_available
 
 
+@dataclass
+class CgroupPathCase:
+    """A process whose control group is resolved from a ``/proc/self/cgroup`` file.
+
+    ``files`` are laid out under the cgroup root, so a path such as
+    ``system.slice/app.scope/cpu.max`` places a limit on the process's own
+    (non-root) control group, the layout seen without a private cgroup namespace.
+    """
+
+    name: str
+    proc_content: str
+    files: dict[str, str]
+    expected_assigned: int | None
+    expected_memory_total: int | None
+    expected_memory_available: int | None
+
+
+CGROUP_PATH_CASES = [
+    CgroupPathCase(
+        name="private_namespace_reads_the_root",
+        proc_content="0::/\n",
+        files={
+            "cpu.max": "400000 100000",
+            "memory.max": "8589934592",
+            "memory.current": "1073741824",
+        },
+        expected_assigned=4,
+        expected_memory_total=8589934592,
+        expected_memory_available=8589934592 - 1073741824,
+    ),
+    CgroupPathCase(
+        name="host_namespace_reads_the_process_cgroup",
+        proc_content="0::/system.slice/app.scope\n",
+        files={
+            "system.slice/app.scope/cpu.max": "400000 100000",
+            "system.slice/app.scope/memory.max": "8589934592",
+            "system.slice/app.scope/memory.current": "1073741824",
+        },
+        expected_assigned=4,
+        expected_memory_total=8589934592,
+        expected_memory_available=8589934592 - 1073741824,
+    ),
+    CgroupPathCase(
+        name="ancestor_limit_applies_to_an_unlimited_leaf",
+        proc_content="0::/kubepods.slice/pod1.slice/container\n",
+        files={
+            "kubepods.slice/pod1.slice/container/cpu.max": "max 100000",
+            "kubepods.slice/pod1.slice/container/memory.max": "max",
+            "kubepods.slice/pod1.slice/cpu.max": "200000 100000",
+            "kubepods.slice/pod1.slice/memory.max": "4294967296",
+            "kubepods.slice/pod1.slice/memory.current": "1073741824",
+        },
+        expected_assigned=2,
+        expected_memory_total=4294967296,
+        expected_memory_available=4294967296 - 1073741824,
+    ),
+    CgroupPathCase(
+        name="most_restrictive_level_wins",
+        proc_content="0::/a/b\n",
+        files={
+            "a/b/cpu.max": "400000 100000",
+            "a/b/memory.max": "8589934592",
+            "a/b/memory.current": "536870912",
+            "a/cpu.max": "200000 100000",
+            "a/memory.max": "4294967296",
+            "a/memory.current": "1073741824",
+        },
+        expected_assigned=2,
+        expected_memory_total=4294967296,
+        expected_memory_available=4294967296 - 1073741824,
+    ),
+    CgroupPathCase(
+        name="unresolvable_cgroup_path_falls_back_to_the_root",
+        proc_content="0::/vanished.scope\n",
+        files={
+            "cpu.max": "400000 100000",
+            "memory.max": "8589934592",
+            "memory.current": "1073741824",
+        },
+        expected_assigned=4,
+        expected_memory_total=8589934592,
+        expected_memory_available=8589934592 - 1073741824,
+    ),
+    CgroupPathCase(
+        name="v1_only_proc_file_reads_the_root_controllers",
+        proc_content="12:memory:/docker/abc\n3:cpu,cpuacct:/docker/abc\n",
+        files={
+            "cpu/cpu.cfs_quota_us": "200000",
+            "cpu/cpu.cfs_period_us": "100000",
+            "memory/memory.limit_in_bytes": "8589934592",
+            "memory/memory.usage_in_bytes": "2147483648",
+        },
+        expected_assigned=2,
+        expected_memory_total=8589934592,
+        expected_memory_available=8589934592 - 2147483648,
+    ),
+]
+
+
+def _process_resources_for(case: CgroupPathCase, tmp_path: Path) -> ProcessResources:
+    cgroup_root = tmp_path / "cgroup"
+    cgroup_root.mkdir()
+    _write_cgroup_files(cgroup_root, case.files)
+    proc_cgroup = tmp_path / "proc_self_cgroup"
+    proc_cgroup.write_text(case.proc_content)
+    return ProcessResources(cgroup_root=cgroup_root, proc_cgroup=proc_cgroup)
+
+
+@pytest.mark.parametrize("case", CGROUP_PATH_CASES, ids=[case.name for case in CGROUP_PATH_CASES])
+def test_cgroup_path_resolution_cpu(case: CgroupPathCase, tmp_path: Path) -> None:
+    reading = _process_resources_for(case, tmp_path).read()
+
+    assert reading.processor_assigned == case.expected_assigned
+
+
+@pytest.mark.parametrize("case", CGROUP_PATH_CASES, ids=[case.name for case in CGROUP_PATH_CASES])
+def test_cgroup_path_resolution_memory(case: CgroupPathCase, tmp_path: Path) -> None:
+    reading = _process_resources_for(case, tmp_path).read()
+
+    assert reading.memory_total == case.expected_memory_total
+    assert reading.memory_available == case.expected_memory_available
+
+
+def test_unlimited_at_every_level_falls_back_to_host(tmp_path: Path) -> None:
+    case = CgroupPathCase(
+        name="unlimited_everywhere",
+        proc_content="0::/a/b\n",
+        files={
+            "a/b/cpu.max": "max 100000",
+            "a/b/memory.max": "max",
+            "a/cpu.max": "max 100000",
+            "a/memory.max": "max",
+        },
+        expected_assigned=None,
+        expected_memory_total=None,
+        expected_memory_available=None,
+    )
+
+    reading = _process_resources_for(case, tmp_path).read()
+
+    assert reading.processor_assigned is None
+    assert reading.memory_total == psutil.virtual_memory().total
+    assert reading.memory_available is not None
+    assert reading.memory_available >= 0
+
+
+def test_binding_ancestor_usage_refreshes_between_reads(tmp_path: Path) -> None:
+    # Free memory must be recomputed against the level that holds the effective
+    # limit — here the parent — not against the unlimited leaf.
+    case = CgroupPathCase(
+        name="refresh",
+        proc_content="0::/a/b\n",
+        files={
+            "a/b/memory.max": "max",
+            "a/memory.max": "4294967296",
+            "a/memory.current": "1073741824",
+        },
+        expected_assigned=None,
+        expected_memory_total=4294967296,
+        expected_memory_available=4294967296 - 1073741824,
+    )
+    reader = _process_resources_for(case, tmp_path)
+
+    first = reader.read()
+    assert first.memory_available == 4294967296 - 1073741824
+
+    (tmp_path / "cgroup" / "a" / "memory.current").write_text("2147483648")
+    second = reader.read()
+
+    assert second.memory_total == 4294967296
+    assert second.memory_available == 4294967296 - 2147483648
+
+
 def test_host_identifier_is_populated(tmp_path: Path) -> None:
     reading = ProcessResources(cgroup_root=tmp_path).read()
 

dev/specs/infp-589-resource-telemetry/research.md

@@ -32,10 +32,12 @@ Phase 0 decisions. Each resolves an unknown surfaced while planning against the
 - **Rationale**: adopting the existing `memory_*` names/semantics for the new components makes DB, server, and workers byte-for-byte comparable (the naming decision, D14), and matches how the DB JMX (`TotalMemorySize` / `FreeMemorySize`) already reports. Pete's "RAM used" is preserved as a derived value, consistent with the DB.
 - **No `memory_assigned`**: memory has no enforced-limit field — the container memory limit surfaces as `memory_total` capacity. Only `processor_assigned` carries the FR-003 no-fallback-null rule.
 
-## D5 — cgroup v2 primary, v1 fallback, `null` otherwise
+## D5 — cgroup v2 primary, v1 fallback, `null` otherwise; resolve the process's own cgroup path
 
-- **Decision**: Read cgroup v2 first (`/sys/fs/cgroup/cpu.max`, `memory.max`, `memory.current`); fall back to v1 (`/sys/fs/cgroup/cpu/cpu.cfs_quota_us` + `cpu.cfs_period_us`, `/sys/fs/cgroup/memory/memory.limit_in_bytes` + `memory.usage_in_bytes`); return `null` for the affected field where neither is readable (non-Linux dev, unusual mounts).
-- **Rationale**: production runs containers (v2 on current hosts, v1 still common); developer machines are macOS (neither) and correctly report `null` for `assigned`. v1 `memory.limit_in_bytes` reports a sentinel near `INT64_MAX` when unlimited — treat values at/above a high threshold as unlimited → `null`.
+- **Decision**: Read cgroup v2 first (`cpu.max`, `memory.max`, `memory.current`); fall back to v1 (`/sys/fs/cgroup/cpu/cpu.cfs_quota_us` + `cpu.cfs_period_us`, `/sys/fs/cgroup/memory/memory.limit_in_bytes` + `memory.usage_in_bytes`); return `null` for the affected field where neither is readable (non-Linux dev, unusual mounts).
+- **Correction (review finding, post-implementation)**: the v2 files were initially read only at `/sys/fs/cgroup` itself. That is correct under a *private cgroup namespace* (the default on Docker 20.10+/modern K8s, where the apparent root *is* the container's cgroup) but wrong without one (older runtimes on v2, explicit `cgroupns: host`, bare-metal systemd services with unit limits): the v2 root cgroup carries no `cpu.max`/`memory.max` at all, so a limited component reported `processor_assigned = null` and — worse for an audit — the whole **host's** memory as `memory_total`. Fixed by resolving the process's own cgroup from the `0::<path>` line of `/proc/self/cgroup` and consulting **every level up to the root**, taking the most restrictive limit (a limit may be enforced on an ancestor slice); memory usage is read at the level holding the effective limit, since an ancestor limit is shared with siblings. Any resolution failure falls back to the previous root-only read.
+- **Known limitation**: limits set *above* a private namespace root (e.g. a K8s pod-level limit when the container itself has none) are invisible from inside the namespace and cannot be self-reported — a fundamental property of self-observation, not an implementation gap.
+- **Rationale**: production runs containers (v2 on current hosts, v1 still common); developer machines are macOS (neither) and correctly report `null` for `assigned`. v1 `memory.limit_in_bytes` reports a sentinel near `INT64_MAX` when unlimited — treat values at/above a high threshold as unlimited → `null`. v1 keeps the root-level read: container runtimes bind-mount the container's own v1 controller directories at `/sys/fs/cgroup/<controller>`, so the root read is already container-scoped there.
 
 ## D6 — Self-report through the existing heartbeat channel