GROUP BY queries hang indefinitely when the default compute_60 gpu-architecture is used on an H100 (should be compute_90)

[ckeenan-seekr]

Summary

When pg_strom is built with WITH_FATBIN=1 on a host without an NVIDIA GPU
(e.g. a CI container), src/Makefile.cuda silently falls back to
--gpu-architecture=compute_60 and generates SASS for sm_60..sm_90 from
Pascal-era PTX:

__NUM_GPUS := $(shell ls -d /proc/driver/nvidia/gpus/*/information | wc -l)
ifeq ($(__NUM_GPUS),0)
__NVCC_TARGET := --gpu-architecture=compute_60 \
                 --gpu-code=sm_60,sm_61,sm_70,sm_75,sm_80,sm_86,sm_89,sm_90
else
__NVCC_TARGET := --gpu-architecture=native
endif

Per the NVIDIA Volta Tuning Guide, code compiled targeting compute_60 executes
on CC >= 7.0 GPUs with Independent Thread Scheduling disabled (legacy
lockstep warp scheduling) — -arch=compute_60 -code=sm_70 is the documented
ITS opt-out.

The GpuPreAgg GROUP BY path (cuda_gpupreagg.cu, __updateOneTupleGroupBy)
acquires a per-hash-slot lock with __atomic_cas_uint64(hslot, saved, ULONG_MAX)
inside a divergent retry loop. This pattern is starvation-free only under ITS.
With ITS disabled, warp-mates spinning on the CAS can permanently starve the
divergent lock-holding thread → livelock: the kernel spins forever at 100% SM
utilization with 0% memory bandwidth, and the query never returns.

The result is that a WITH_FATBIN=1 package built on a GPU-less builder hangs
every GpuPreAgg query with a non-empty GROUP BY on H100 (and presumably any
CC >= 7.0 GPU), while the identical build performed on a GPU host works
perfectly — an extremely confusing failure mode.

This is reproducible by building two images, one with --gpu-architecture=compute_60
and the other with --gpu-architecture=compute_90 from the same machine that has an H100.
Reproduction scripts are attached.
entrypoint.sh
test.sh
Dockerfile.md (Had to attach as .md, github didn't accept it without the extension but this is just a dockerfile)

The expected behavior here is either an error due to the wrong architecture, or fix
the implemenation to not hang. Hanging in this case is the worst outcome because
its very difficult to debug.

Environment

Component	Version
pg_strom	v6.1 (also reproduced on v5.2.2)
PostgreSQL	16
CUDA toolkit	12.8.2 (nvcc V12.8)
OS	Ubuntu 24.04
GPU (runtime)	NVIDIA H100 80GB HBM3 (CC 9.0)
Build host	container without GPU (GitLab CI / buildkit)

Reproduction

1. Build with WITH_FATBIN=1 inside nvidia/cuda:12.8.2-devel-ubuntu24.04
   without a GPU visible (no /proc/driver/nvidia).

2. Install on an H100 host and run:

   CREATE TABLE probe_tbl (grp TEXT, val FLOAT8);
   INSERT INTO probe_tbl VALUES
     ('alice', 1.0), ('alice', 2.0), ('alice', 3.0),
     ('bob',   4.0), ('bob',   5.0);
   SET pg_strom.enabled = on;
   SET pg_strom.gpu_setup_cost = 0;
   SET pg_strom.cpu_fallback = off;
   SELECT grp, COUNT(*), SUM(val) FROM probe_tbl GROUP BY grp;  -- hangs forever

   SELECT COUNT(*), SUM(val) FROM probe_tbl; (no group key) completes fine —
   the no-groups path takes no hash-slot lock.

3. The same build performed on the H100 host itself
   (--gpu-architecture=native → compute_90) does not hang.
   WITH_FATBIN=0 (runtime JIT on the GPU host) also does not hang.

During the hang: nvidia-smi shows 100% GPU utilization, 0% memory utilization.
A host-side GDB stack shows the GPU service worker blocked in memset() inside
__gpuMemAllocNewSegment — a downstream symptom: the livelocked kernel holds
Unified Memory page ownership, so host access to managed memory blocks behind it.

Evidence that the two builds differ despite identical fatbin filenames

The fatbin filename hash covers source inputs only (Makefile.cuda):

CUDA_MD5SUM = $(shell cat $(CUDA_HEADERS) $(CUDA_SRCS) | md5sum | ...)
CUDA_FATBIN = pgstrom-gpucode-V$(NVCC_VERSION)-$(CUDA_MD5SUM).fatbin

so a GPU-less build and a GPU-host build produce the same filename with
different SASS, and __validate_gpu_fatbin_file accepts both.

Building v6.1 twice in the same GPU-less CUDA 12.8.2 container — once with the
default (no GPU → compute_60 path) and once patched to
--gpu-architecture=compute_90 --gpu-code=sm_90 — yields:

	Build A (no GPU, default Makefile)	Build B (compute_90/sm_90)
Fatbin filename	pgstrom-gpucode-V012080-c406c39…fatbin	identical filename
Size	51,484,072 B	6,669,952 B
md5	cb9c71122a28425786c4f0d64a0523b2	e6d2d4a3066d75e347a8bbaec7835dba
Embedded cubins (cuobjdump --list-elf)	sm_60,61,70,75,80,86,89,90 (no PTX)	sm_90 (no PTX)
sm_90 SASS (cuobjdump -sass -arch sm_90)	differs from B in 117,494 lines	—

sm_90 cubin ELF headers:

Build A: .headerflags @"... EF_CUDA_SM90 EF_CUDA_VIRTUAL_SM(EF_CUDA_SM60)"
         readelf -h: Flags = 0x3c055a   (0x3c = 60)
Build B: .headerflags @"... EF_CUDA_SM90 EF_CUDA_VIRTUAL_SM(EF_CUDA_SM90)"
         readelf -h: Flags = 0x5a055a   (0x5a = 90)

The GPU-less build's sm_90 cubin is stamped VIRTUAL_SM(SM60) — the flag the
CUDA driver uses to select legacy (lockstep) vs. independent thread scheduling.
Since the fatbin embeds no PTX, the driver cannot JIT an ITS-enabled version;
the compute_60-derived SASS is what executes, with ITS disabled.

Suggested fixes

1. Makefile.cuda: when no GPU is detected, use per-target virtual archs
   (e.g. -gencode arch=compute_70,code=sm_70 ... -gencode arch=compute_90,code=sm_90,
   with compute_60 only for sm_60/61) instead of a single compute_60 virtual
   arch for all targets — or at minimum print a prominent warning that the
   resulting binary disables ITS on CC >= 7.0.
2. Runtime validation: include the nvcc target flags in the fatbin
   filename/validation hash so an arch-degraded fatbin triggers a JIT rebuild
   instead of loading.
3. (Optional hardening) Make the hash-slot lock loop in
   __updateOneTupleGroupBy robust under legacy scheduling (e.g. __nanosleep
   backoff), so even ITS-disabled builds fail gracefully rather than livelock.

Live validation on H100

Built pg_strom v6.1 twice from the same Dockerfile in the same
nvidia/cuda:12.8.2-devel-ubuntu24.04 container (no GPU at build time), with
only __NVCC_TARGET differing. Ran each on an H100 80GB HBM3 (CC 9.0):

SQL: SET pg_strom.enabled=on; SET pg_strom.gpu_setup_cost=0;
     SET pg_strom.cpu_fallback=off; SET statement_timeout='60s';
     SELECT grp, COUNT(*), SUM(val) FROM probe GROUP BY grp ORDER BY grp;
     -- 500k rows, 5 groups, GpuPreAgg confirmed in EXPLAIN

compute_60 build:  Elapsed 60s → ERROR: canceling statement due to statement timeout
compute_90 build:  Elapsed  0s → g0|100000|50012.2 … g4|100000|50024.9  (correct)

The query returns instantly with correct results when ITS is enabled, and
livelocks until timeout when ITS is disabled. Validation harness available at
archive/pg-strom-repro/rca/pg-strom-arch-validation/.

Workarounds for users

• Build with WITH_FATBIN=0 (runtime JIT), or
• Build WITH_FATBIN=1 on a host with the target GPU installed, or
• Pass NVCC_FLAGS_CUSTOM="--gpu-architecture=compute_90 --gpu-code=sm_90"
  (adjust to your GPU) to make.