Skip to content

Fix GPU device-loss when loading heavy captures (auto-disable Opacity Micromaps) - #11

Closed
skurtyyskirts wants to merge 3 commits into
NVIDIAGameWorks:mainfrom
skurtyyskirts:fix/heavy-capture-device-loss
Closed

Fix GPU device-loss when loading heavy captures (auto-disable Opacity Micromaps)#11
skurtyyskirts wants to merge 3 commits into
NVIDIAGameWorks:mainfrom
skurtyyskirts:fix/heavy-capture-device-loss

Conversation

@skurtyyskirts

Copy link
Copy Markdown
Contributor

What this does

Fixes a GPU device-loss (VK_ERROR_DEVICE_LOST) that crashes/freezes the Toolkit when opening or switching to a very heavy game capture. Reproduced with dense Tomb Raider: Legend captures (bolivia__2.usd ≈18k prims / ≈9k meshes, and the peru captures). During stage realization the path tracer builds geometry / BVH / Opacity-Micromap data for the whole capture at once; on an over-budget capture the OMM working set exceeds the GPU budget and faults the Vulkan device in-process (Aftermath GPU dump generated, no OS-level TDR).

The fix

lightspeed.trex.capture.core.shared now inspects a capture's prim count before it is realized and, if it is at or above a heavy-capture budget (12k prims), loads it in a degraded-safe renderer state:

  • Forces rtx.graphicsPreset = Custom (so the User-layer write wins over the capture's Quality preset), then rtx.opacityMicromap.enable = 0, through the existing hdremix_set_configvar bridge.
  • Applied on project open (the stage-opened event, before the renderer realizes the heavy geometry) and on manual capture switch.
  • Gated by a new setting autoSafeModeOnHeavyCapture (default on); set it to false to always load captures at full quality.

The prim-count inspection is a cheap Sdf.Layer prim-spec walk (no stage composition). The HdRemix dependency is optional and imported defensively, so headless / CLI apps without the renderer are unaffected.

Deliberately minimal and game-agnostic — the whole change is contained in one extension. It does not touch performance tuning, diagnostics, or renderer internals.

Testing

  • format_code and lint_code all: pass.
  • Extension unit tests: 19 pass (OK, 0 failures), including 5 new tests covering prim-count estimation (nested count, missing layer) and the safe-mode push (over-budget pushes graphicsPreset=4 then opacityMicromap.enable=0 in order; under-budget no-op; setting-off no-op).
  • Measured setup.py module coverage: 83% (> 75%).
  • Behavioral verification (open the Tomb Raider: Legend project and confirm the capture realizes with Opacity Micromap: disabled and no device-loss over 2–3 min) should be run on the affected hardware — this is the original repro that motivated the change.

Housekeeping

  • lightspeed.trex.capture.core.shared version bumped 1.4.01.5.0; extension docs/CHANGELOG.md and the root CHANGELOG.md updated.
  • Opened as a draft for self-review. I'll sign the NVIDIA CLA when the bot prompts.

Follow-up (out of scope)

On the original hardware a second, later freeze can occur ~1 minute after a heavy capture realizes, when dxvk-remix re-applies the capture's own graphics preset and re-enables Opacity Micromaps without emitting a settings change. That is a separate issue; the underlying renderer should ideally bound its per-capture GPU allocation and surface a recoverable error on VK_ERROR_DEVICE_LOST rather than losing the device.

Heavy game captures (e.g. dense Tomb Raider: Legend captures such as
bolivia__2 / peru, ~18k prims) overrun the GPU acceleration-structure /
Opacity-Micromap working set during stage realization and fault the Vulkan
device (VK_ERROR_DEVICE_LOST). Before an over-budget capture is realized --
on project open (stage-opened event) and on capture switch -- force
rtx.graphicsPreset=Custom then rtx.opacityMicromap.enable=0 through the
HdRemix bridge so it loads in a degraded-safe state instead of taking down
the GPU. Gated by the new autoSafeModeOnHeavyCapture setting (default on).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 15, 2026

Copy link
Copy Markdown

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Run ID: e949ef36-b997-48fa-bef2-e24de81e4341

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

skurtyyskirts and others added 2 commits July 15, 2026 00:18
The prim-budget safe-mode gate in lightspeed.trex.capture.core.shared ran on the
StageEventType.OPENED event -- after the heavy capture was already realized -- and
never re-asserted, so it could neither pre-empt the initial GPU device-loss nor stop
the capture's own graphics preset from re-enabling Opacity Micromaps ~1 minute later.
Revert it to upstream; the working fix lives in lightspeed.trex.control.stagecraft
(next commit), which disables OMM before open_stage and re-asserts on a watchdog.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Opening a capture project (or switching captures) in StageCraft could fault
the Vulkan device (VK_ERROR_DEVICE_LOST) or hang the main thread seconds after
open_stage. A game capture is thousands of small alpha-tested meshes, and
building Opacity Micromaps (OMM) for all of them during stage realization can
overrun the path tracer's GPU working set -- there is no per-capture GPU budget
ceiling to fall back on, so the device faults instead of degrading.

Prim count does not reliably predict which captures do this (a 599 KB capture
hung while larger ones did not), and OMM is only a render-time optimization for
alpha-tested geometry, so disabling it never changes visuals and never affects
editing or asset replacement. lightspeed.trex.control.stagecraft now disables
OMM (graphicsPreset=Custom, integrateIndirectMode=ReSTIR GI,
opacityMicromap.enable=0, pushed through the HdRemix bridge) for any capture
project before open_stage realizes the stage, and re-asserts it every
opacityMicromapReassertIntervalSeconds (default 5s) via a watchdog -- because
dxvk-remix re-applies the capture's own graphics preset a few seconds after
realization (re-enabling OMM) without touching the carb /rtx/* nodes, so a
one-shot override silently loses and the GPU faults ~1 minute later anyway. The
same override is applied before a switched-to capture is realized.

Gated by autoDisableOpacityMicromaps (default on); torn down on destroy and when
the setting is off. Adds the optional lightspeed.hydra.remix.core dependency for
the bridge and bumps the extension to 1.8.1.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@skurtyyskirts
skurtyyskirts deleted the fix/heavy-capture-device-loss branch July 17, 2026 02:03
@github-actions github-actions Bot locked and limited conversation to collaborators Jul 17, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant