feat(arrs): add automatic queue cleanup for failed/stuck imports #234
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fixed a bug in RAR archive validation where AES-encrypted files were being validated with incorrect parameters, causing misleading error messages about file size mismatches. The issue was that the code was: 1. Manually calculating the encrypted size using aes.EncryptedSize() 2. Passing this encrypted size to ValidateSegmentsForFile 3. But passing Encryption_NONE as the encryption type This caused ValidateSegmentsForFile to treat the encrypted size as the decrypted size, leading to validation failures and confusing error messages like "expected X bytes (decrypted)" when the actual issue was with encryption handling. The fix: - Pass the original decrypted file size to ValidateSegmentsForFile - Determine and pass the correct encryption type (Encryption_AES) - Let ValidateSegmentsForFile handle the encryption size calculation internally based on the encryption type This ensures: - Correct validation logic for AES-encrypted RAR files - Accurate error messages that distinguish between encrypted/decrypted sizes - Proper handling of AES padding calculations (16-byte boundary) Removed the now-unused aes package import.
Replace io.Pipe with nio.Pipe backed by bounded buffers to provide backpressure control during segment streaming. This prevents unbounded memory growth when downloads proceed faster than consumption. - Add djherbis/buffer and djherbis/nio dependencies - Use configurable buffer size per segment (default 1MB) - Update GetSegmentsInRange to accept bufferSize parameter - Update callers in usenet_fs.go and metadata_remote_file.go - Update tests to use new nio pipe with buffer 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The buffer's internal storage was not being released when segments were closed, causing potential memory leaks during streaming. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Removed omitempty from health config integer fields to prevent them from being hidden in the UI when set to 0.
Removed omitempty from health config integer fields to prevent them from being hidden in the UI when set to 0.
- fix category path's - added override to api key - added secure select for real category
missing merge codes
Refactor segment handling to initialize buffers only when needed, reducing upfront memory allocation. This change allows for more efficient memory usage during downloads by deferring buffer creation until the segment is about to be downloaded. - Update segment struct to include buffer size for lazy initialization - Add initBuffer method to handle buffer creation - Modify GetReader and downloadManager to ensure buffer is initialized before use - Adjust timeout duration in UsenetReader close method for improved responsiveness 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This ensures that CalculateVirtualDirectory receives a proper absolute base path, allowing it to correctly calculate relative paths for imported files instead of falling back to string concatenation. This prevents absolute path exposure in the virtual directory, especially when CompleteDir is set to /.
- Add missing default config values for queue cleanup enabled/interval - Expose queue cleanup options in config.sample.yaml - Fix potential path prefix matching issues in isPathManaged (segment-aware) - Add usenet protocol check to Radarr queue cleanup for consistency
…ulation (restored)
…olute path exposure in virtual directory
Contributor
Author
|
Closing this as it has been superseded by the cleaner feature branch in PR #244. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces a new worker to automatically clean up stuck items from Sonarr and Radarr queues. This is particularly useful for maintaining a healthy queue when using
altmountas a download client, where some imports might fail permanently due to mapping issues or file validity checks.Key Changes
internal/arrs/workerthat periodically checks the queue of all enabled Sonarr and Radarr instances.cleanup_automatic_import_failure, defaultfalse).QueueCleanupIntervalSecondsandCleanupAutomaticImportFailuretoArrsconfig.Motivation
Users were experiencing items stuck in "Downloaded - Waiting to Import" state that required manual intervention to remove. This feature automates that maintenance task.