fix: filter empty documents in OpenAI rerank client

[yc111233]

Summary

• Rerank providers like DashScope (qwen3-rerank) return HTTP 400 when any document in the batch is an empty string
• This can happen when vector records have empty abstract fields (e.g. due to fix: backfill abstract from file content in vectorize_file #1343)
• Currently the entire rerank call fails, causing fallback to raw vector scores for all results

Fix

Filter out empty/whitespace-only documents before sending to the rerank API, and map scores back to original indices. Empty documents receive a score of 0.0.

valid_indices = [i for i, d in enumerate(documents) if d and d.strip()]
if not valid_indices:
    return [0.0] * len(documents)
# ... call API with filtered_docs ...
# ... map scores back to original positions ...

This acts as a defensive safety net — rerank degrades gracefully (empty docs get score 0.0) instead of failing the entire batch.

Related

• fix: backfill abstract from file content in vectorize_file #1343 fixes the root cause (empty abstracts in vectorize_file)
• This PR provides defense-in-depth for any other path that might produce empty documents

Test plan

• Call rerank_batch with a mix of valid and empty documents
• Verify valid documents get proper rerank scores
• Verify empty documents get score 0.0
• Verify no HTTP 400 errors from DashScope

🤖 Generated with Claude Code

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 1343 - Partially compliant Compliant requirements: Provides defense-in-depth to prevent rerank 400 errors by filtering empty documents Non-compliant requirements: Does not address the root cause (backfilling abstract in vectorize_file) Requires further human verification: Verify that empty documents receive a score of 0.0 Verify that valid documents get proper rerank scores Verify no HTTP 400 errors from rerank providers

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪

🏅 Score: 92

🧪 No relevant tests

🔒 No security concerns identified

✅ No TODO sections

🔀 No multiple PR themes

⚡ No major issues detected

[github-actions]

PR Code Suggestions ✨

No code suggestions found for the PR.

[qin-ctx]

Thanks for the defense-in-depth fix. I found one blocking correctness issue and one non-blocking test gap. The main concern is the all-empty batch path, which currently returns zero scores and suppresses the retriever's existing fallback to vector scores.

[qin-ctx]

[Bug] (blocking) When every input document is empty or whitespace, this returns an all-zero score list instead of signaling rerank failure. HierarchicalRetriever._rerank_scores() treats any numeric list with the expected length as a successful rerank, so this path bypasses fallback to vector scores and can filter out otherwise retrievable results at the rerank threshold. Mixed batches should keep the current behavior, but an all-empty batch should return None or otherwise trigger the existing fallback path.

[qin-ctx]

[Suggestion] (non-blocking) Please add regression tests for the new filtering behavior. The current suite does not cover either a mixed batch like ['doc', '', ' '] with index remapping or the all-empty batch path, so the rerank/fallback semantics introduced here are not locked down.