feat(server): evict cached model sessions to keep RAM bounded#6
Merged
Conversation
The Electron app was growing past 20 GB because every model the user touched stayed loaded in `_SESSIONS` forever. Three knobs to fix that: - Idle TTL: a background sweeper unloads sessions that have not been used in `RBL_MODEL_IDLE_TTL` seconds (default 600). The pinned model is exempt — it is the one the user is actively working with. - /set_default_model: when the UI changes the global model dropdown, the previous one is evicted from RAM immediately so we don't pile up BiRefNet + ISNet + U2Net at once. - /remove?transient=true: per-image overrides (reprocess) load their model but do not steal the pinned default. The idle TTL reclaims them later. Plus /unload_model for manual freeing without touching the on-disk cache, and /health now reports the pinned model + TTL. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Electron app could grow past 20 GB because every model the user touched stayed loaded in
_SESSIONSforever. This PR caps that:RBL_MODEL_IDLE_TTLseconds (default 600). The pinned model is exempt — it's the one the user is actively working with.POST /set_default_model: the UI calls this when the global model dropdown changes. The previous model is immediately dropped from RAM so we don't pile up BiRefNet + ISNet + U2Net at once.POST /remove?transient=true: per-image reprocess with a different model loads it but does not steal the pinned default. The idle TTL reclaims it later.POST /unload_model: manual free without touching the on-disk.onnxcache.GET /health: now reportspinned_modelandidle_ttl_seconds.UI wired:
pinDefaultModel(model)→ evicts everything else.reprocessJob(job, newModel)flags the request as transient when the model differs from the global one.Test plan
./run_tests.sh— 26 tests pass (10 new covering_evict, sweep,/unload_model,/set_default_model, transient remove, health payload).server.pywithRBL_MODEL_IDLE_TTL=4:/set_default_modelrotates pin and evicts the previous one ✅/removewithtransient=truereturns a valid PNG and does not change the pinned model ✅🤖 Generated with Claude Code