Add 30-minute timeout for MSA server PENDING state - #5
Merged
Conversation
The ColabFold MSA server (api.colabfold.com) can rate-limit or drop jobs, causing them to stay in PENDING state forever. Previously, jobs would wait 12+ hours before the RQ job timeout killed them. This change: - Detects when MSA jobs are stuck in PENDING state - Fails fast after 30 minutes with a clear error message - Advises users to retry (which typically resolves the issue) Root cause: The MSA server appears to expire/deprioritize old job IDs rather than processing them when rate limits free up. Fresh job submissions (from restart) get new IDs that work. See: sokrypton/ColabFold#606
This reverts commit 4a8c7d2.
Problem: MSA server jobs can get stuck in PENDING state forever when the ColabFold server drops or deprioritizes jobs. The client keeps polling the same dead job ID indefinitely (PENDING is a "sink" - it never triggers resubmission like RATELIMIT does). Solution: 1. Detect PENDING state stuck for >10 minutes and fail fast with TimeoutError instead of waiting 12+ hours for RQ job timeout 2. Configure RQ to retry boltz jobs up to 3 times (1min, 2min, 5min intervals) - each retry gets a fresh MSA job ID 3. Upgrade Boltz from 2.1.1 to 2.2.1 (adds auth support, bug fixes) This allows automatic recovery from dropped MSA jobs without manual intervention. See: sokrypton/ColabFold#606
Jacoberts
force-pushed
the
claude/debug-error-message-5ijkC
branch
from
December 27, 2025 18:42
8cab7d2 to
95e1b9c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The ColabFold MSA server (api.colabfold.com) can rate-limit or drop jobs,
causing them to stay in PENDING state forever. Previously, jobs would wait
12+ hours before the RQ job timeout killed them.
This change:
Root cause: The MSA server appears to expire/deprioritize old job IDs
rather than processing them when rate limits free up. Fresh job
submissions (from restart) get new IDs that work.
See: sokrypton/ColabFold#606