aerospike-client-go/v8 nodeStats.updateOrInsert consumes 44-60% of legacy CPU during mainnet IBD

[oskarszoon]

Summary

The bsv-blockchain/aerospike-client-go/v8@v8.7.1-bsv3 fork's per-record stats tracking (nodeStats.updateOrInsert) consumes 44–60% of the legacy service's CPU during mainnet IBD. Every record in every batch response triggers an atomic-map update; map contention scales with batch size and concurrency, dominating the hot path that should be doing useful work.

Observed

30-second CPU profiles, mainnet IBD on 2026-06-01:

eu-2 legacy — sync rate ~6.6s/block, mixed block sizes:

44.64s total samples
- 43.88% github.com/bsv-blockchain/aerospike-client-go/v8.batchCommandOperate.Execute
  - 43.37% baseMultiCommand.parseResult
    - 42.47% batchCommandOperate.nsIter
      - 42.32% nodeStats.updateOrInsert
        - 39.20% nodeStats.updateOrInsert-range1
          - 20.99% atomic/map.(*Map[int,uint64]).Set

eu-3 legacy — sync rate ~3.3s/block, smaller average blocks:

39.98s total samples
- 59.60% batchCommandOperate.Execute
  - 58.75% parseResult
    - 57.80% nsIter
      - 57.60% nodeStats.updateOrInsert
        - 54.65% nodeStats.updateOrInsert-range1
          - 30.47% atomic/map.Set

Both hosts running teranode v0.15.2-beta-4 (commit e125d1ef8), aerospike client v8.7.1-bsv3.

What the hot path is doing

Per the call graph: for every record returned in a batch response, the parse loop calls nsIter, which calls nodeStats.updateOrInsert(...), which calls into the internal atomic-map Set. With 30+ concurrent batch operations (per-service batchers, partition workers, parallel txMap processing) and thousands of records per batch, the atomic map becomes the contention bottleneck — atomic/map.Set alone is 20–30% of legacy's CPU.

Why this hurts more on BSV/teranode than typical aerospike workloads

• Block-driven burst pattern. Each block triggers len(txs) × inputs reads (batch-decorate previous outputs) + len(txs) writes (createUtxos) + existing-tx-count writes (SetMinedMulti merge). For a typical mainnet block this is tens of thousands of records flowing through one parseRecordResults loop.
• High concurrency, single namespace. Many batches from different goroutines all hammer the same per-node stats map.
• Stats are per-record, not per-batch. So a 1024-record batch incurs 1024 atomic ops to the map.

For a typical workload (smaller batches, mixed namespaces, fewer concurrent batchers) this would be hidden by network/disk latency. Teranode's workload exposes it because the network is local (single-node aerospike on the same host) and the batches are huge.

Fix directions

In rough order of difficulty / impact:

1. Sample stats instead of recording every record. Update on the first record per batch (or 1-in-N) and extrapolate. Most stats consumers (latency histograms, health monitoring) don't need per-record granularity.

2. Sharded counters with periodic aggregation. Replace the global atomic map with one counter per goroutine (or per fixed shard count), aggregate on read. Removes the cross-goroutine atomic contention entirely.

3. Optionally disable per-record stats for high-throughput paths. Expose a client config flag (DisablePerRecordStats: true); teranode would set it for the legacy/pruner clients.

4. Push stats update to the end of the batch loop. If correctness allows: accumulate counts locally, single map update per batch instead of per-record. Cuts atomic ops by batch_size.

5. Audit whether updateOrInsert is using the right structure. If the map only ever has a small bounded set of keys (one per cluster node?), a fixed-size array indexed by node ID would be cheaper than an atomic hashmap.

Verification

Once a candidate fix lands:

• Pre/post CPU profile (30s each) on a busy legacy: % time in batchCommandOperate.parseRecordResults should drop from ~58% to <10%
• Throughput: blocks/second sync rate on eu-2 / eu-3 should rise proportionally
• Latency: batch round-trip time should drop (or stay flat if network-bound)
• Race-test with -race to confirm no new correctness issues

Captured profiles (local)

• probe/eu2-mainnet-sync-2026-06-01/legacy/legacy-cpu30s.pb.gz
• probe/eu3-mainnet-sync-2026-06-01/eu3-prof-1780321364/legacy-cpu30s.pb.gz

Available on request.

Related

• legacy: createUtxos calls SetMinedMulti with unbounded slice — stalls aerospike on fat blocks (regression from #854) #936 — createUtxos chunking (fixed). Without it, batch sizes were even larger and this hotspot would be worse.
• chore(settings): raise aerospike client pool to 128 for docker.m #941 — connection pool size. Larger pool = more concurrent batches in flight = more contention on the same stats map.
• spend circuit breaker counts KEY_NOT_FOUND as infrastructure failure, defeats orphanage during catch-up #953 — spend circuit breaker counts wrong errors. Different layer, but in the same client.

Context: where the fork sits

The fork is github.com/bsv-blockchain/aerospike-client-go/v8 at tag v8.7.1-bsv3. The -bsv3 suffix suggests local modifications layered on top of upstream aerospike/aerospike-client-go/v8. Worth confirming whether nodeStats.updateOrInsert was added by the BSV fork or is also in upstream — that determines whether the fix goes here or needs upstream coordination.

[oskarszoon]

Root cause is worse than the original framing — quadratic per batch, not per-record

Batch commands (operate/get/exists/delete/udf) set getNamespace() == nil, so updateOrInsert takes the loop branch and iterates every record's namespace via nsIter. And updateOrInsert is already called once per record in parseRecordResults. So an N-record batch does N × N = N² counter increments, not N. A 1024-record batch ≈ 1M increments.

This is a general per-batch cost on every batch command — not specific to IBD. Mainnet IBD just exposes it most: largest batches, highest concurrency, and aerospike is local so there is no network latency to mask it. Steady-state block/tx validation pays the same cost, proportionally smaller.

Secondary bugs that fall out of the same path

• Counts inflated ~N×. Because of the N² loop, anything reading DetailedResultCodeCounts over-reports by roughly the batch size. The stats are wrong, not just slow.
• Lost-update race. updateOrInsert does Get (RLock) then Set (Lock) as two separate ops, so concurrent batchers lose increments. The map even has an atomic UpdateOrInsert method that nobody calls.
• The "lock-free" comment is wrong. internal/atomic/map is a sync.RWMutex + plain Go map. The profiled atomic/map.Set at 20–30% is mutex contention, not lock-free atomics.

The granularity buys nothing

In the batch path, all non-zero result codes return/continue before the updateOrInsert call — it only ever records resultCode == 0. So per-record tracking is just "count of successful records per namespace per command type." A single per-batch count would be identical.

Scope

Scans/queries (baseMultiCommand) return a real namespace → O(N), not quadratic, so they are unaffected. Fix is scoped to the 5 batch commands. All of this is fork-added code (nsIter + detailed metrics), so no upstream coordination needed.

Suggested fix

Credit only the current record's namespace, and accumulate counts goroutine-locally during the parse loop, then flush one atomic update per batch (via the map's existing UpdateOrInsert). Result: O(N) parse, ~1 lock acquisition per batch instead of ~N², correct counts, no lost-update race.

Stop-gap shipped

#1003 disables client metrics in teranode (initStats stops calling EnableMetrics). The hot path is gated behind cluster.metricsEnabled, so with it off the path is never entered. Connection-pool and node-count metrics are recorded unconditionally and keep flowing; only the per-command latency histograms and result-code counts go dark until the fork fix lands.

[icellan]

Fix opened in the fork: bsv-blockchain/aerospike-client-go#4 (base feat/teranode-modify-op).

Root cause confirmed. Every batch command's per-record parse loop called updateOrInsert(getNamespace(), getNamespaces(), …). For batch commands getNamespace() returns nil, so the for ns := range namespaces branch runs — and getNamespaces() (nsIter) walks every record in the batch. Since the call is once per record, aggregation is O(N²) per batch, and each record's result code is counted against every namespace in the batch. The counter was also incremented via a non-atomic Get-then-Set on an RWMutex map, which lost ~99% of updates under concurrency.

Fix (fork-only): each record now records against its own namespace (O(1) per record), and the increment uses the map's single-lock UpdateOrInsert (removes the lost-update race). Applied to all nine batch commands (the issue listed the symptom; the PR found one extra — batch_index_command_get — with the same bug).

Benchmark (batchCommandGet parse loop, single namespace):

records	before (O(N²))	after (O(N))	speedup
64	105,922 ns	4,883 ns	21.7×
256	1,655,755 ns	18,980 ns	87×
1024	26,543,849 ns	78,945 ns	336×

-race clean; 76 batch/metrics ginkgo specs pass against a live server with metrics enabled.

Not done here (deliberate): teranode keeps client.EnableMetrics commented out (util/aerospike.go), so this fix has no runtime effect on teranode yet. Re-enabling metrics + bumping the dependency to a new fork tag remains a separate follow-up once the PR merges and is tagged.

Note: the per-command latency/bytes metrics (applyDetailedMetrics*) also iterate getNamespaces(), but run once per command (O(N), not this hotspot) — left for a separate ticket.