Fix GPU device-loss when loading heavy captures (auto-disable Opacity Micromaps) - #11
Closed
skurtyyskirts wants to merge 3 commits into
Closed
Conversation
Heavy game captures (e.g. dense Tomb Raider: Legend captures such as bolivia__2 / peru, ~18k prims) overrun the GPU acceleration-structure / Opacity-Micromap working set during stage realization and fault the Vulkan device (VK_ERROR_DEVICE_LOST). Before an over-budget capture is realized -- on project open (stage-opened event) and on capture switch -- force rtx.graphicsPreset=Custom then rtx.opacityMicromap.enable=0 through the HdRemix bridge so it loads in a degraded-safe state instead of taking down the GPU. Gated by the new autoSafeModeOnHeavyCapture setting (default on). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
The prim-budget safe-mode gate in lightspeed.trex.capture.core.shared ran on the StageEventType.OPENED event -- after the heavy capture was already realized -- and never re-asserted, so it could neither pre-empt the initial GPU device-loss nor stop the capture's own graphics preset from re-enabling Opacity Micromaps ~1 minute later. Revert it to upstream; the working fix lives in lightspeed.trex.control.stagecraft (next commit), which disables OMM before open_stage and re-asserts on a watchdog. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Opening a capture project (or switching captures) in StageCraft could fault the Vulkan device (VK_ERROR_DEVICE_LOST) or hang the main thread seconds after open_stage. A game capture is thousands of small alpha-tested meshes, and building Opacity Micromaps (OMM) for all of them during stage realization can overrun the path tracer's GPU working set -- there is no per-capture GPU budget ceiling to fall back on, so the device faults instead of degrading. Prim count does not reliably predict which captures do this (a 599 KB capture hung while larger ones did not), and OMM is only a render-time optimization for alpha-tested geometry, so disabling it never changes visuals and never affects editing or asset replacement. lightspeed.trex.control.stagecraft now disables OMM (graphicsPreset=Custom, integrateIndirectMode=ReSTIR GI, opacityMicromap.enable=0, pushed through the HdRemix bridge) for any capture project before open_stage realizes the stage, and re-asserts it every opacityMicromapReassertIntervalSeconds (default 5s) via a watchdog -- because dxvk-remix re-applies the capture's own graphics preset a few seconds after realization (re-enabling OMM) without touching the carb /rtx/* nodes, so a one-shot override silently loses and the GPU faults ~1 minute later anyway. The same override is applied before a switched-to capture is realized. Gated by autoDisableOpacityMicromaps (default on); torn down on destroy and when the setting is off. Adds the optional lightspeed.hydra.remix.core dependency for the bridge and bumps the extension to 1.8.1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Fixes a GPU device-loss (
VK_ERROR_DEVICE_LOST) that crashes/freezes the Toolkit when opening or switching to a very heavy game capture. Reproduced with dense Tomb Raider: Legend captures (bolivia__2.usd≈18k prims / ≈9k meshes, and theperucaptures). During stage realization the path tracer builds geometry / BVH / Opacity-Micromap data for the whole capture at once; on an over-budget capture the OMM working set exceeds the GPU budget and faults the Vulkan device in-process (Aftermath GPU dump generated, no OS-level TDR).The fix
lightspeed.trex.capture.core.sharednow inspects a capture's prim count before it is realized and, if it is at or above a heavy-capture budget (12k prims), loads it in a degraded-safe renderer state:rtx.graphicsPreset = Custom(so the User-layer write wins over the capture's Quality preset), thenrtx.opacityMicromap.enable = 0, through the existinghdremix_set_configvarbridge.autoSafeModeOnHeavyCapture(default on); set it tofalseto always load captures at full quality.The prim-count inspection is a cheap
Sdf.Layerprim-spec walk (no stage composition). The HdRemix dependency is optional and imported defensively, so headless / CLI apps without the renderer are unaffected.Deliberately minimal and game-agnostic — the whole change is contained in one extension. It does not touch performance tuning, diagnostics, or renderer internals.
Testing
format_codeandlint_code all: pass.OK, 0 failures), including 5 new tests covering prim-count estimation (nested count, missing layer) and the safe-mode push (over-budget pushesgraphicsPreset=4thenopacityMicromap.enable=0in order; under-budget no-op; setting-off no-op).setup.pymodule coverage: 83% (> 75%).Opacity Micromap: disabledand no device-loss over 2–3 min) should be run on the affected hardware — this is the original repro that motivated the change.Housekeeping
lightspeed.trex.capture.core.sharedversion bumped1.4.0→1.5.0; extensiondocs/CHANGELOG.mdand the rootCHANGELOG.mdupdated.Follow-up (out of scope)
On the original hardware a second, later freeze can occur ~1 minute after a heavy capture realizes, when dxvk-remix re-applies the capture's own graphics preset and re-enables Opacity Micromaps without emitting a settings change. That is a separate issue; the underlying renderer should ideally bound its per-capture GPU allocation and surface a recoverable error on
VK_ERROR_DEVICE_LOSTrather than losing the device.