feat(casework): bulk-submit a batch into the review queue - #444
feat(casework): bulk-submit a batch into the review queue#444gaurav-karki wants to merge 6 commits into
Conversation
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code review is billed via overage credits. To resume reviews, an organization admin can raise the monthly limit at claude.ai/admin-settings/claude-code.
Once credits are available, push a new commit or reopen this pull request to trigger a review.
|
Warning Review limit reached
Next review available in: 54 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (6)
📝 WalkthroughWalkthroughThis change adds review API methods and a ChangesReview workflow
Estimated code review effort: 5 (Critical) | ~90 minutes Possibly related PRs
Suggested labels: Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
PR Reviewer Guide 🔍(Review updated until commit 595865e)Here are some key observations to aid the review process:
🛠️ Relevant configurations:
[config] enable_ai_metadata: False
custom_model_max_tokens: 200000
git_provider: github
output_relevant_configurations: True
model: openai/cx/gpt-5.5
ENABLE_AUTO_APPROVAL: True
custom_reasoning_model: False
fallback_models: ['openai/cx/gpt-5.4-mini']
is_auto_command: True
publish_output: True
publish_output_progress: True
progress_gif_url:
progress_gif_width: 48
verbosity_level: 0
use_extra_bad_extensions: False
log_level: DEBUG
use_wiki_settings_file: True
use_repo_settings_file: True
use_global_settings_file: True
extra_config_url:
disable_auto_feedback: False
ai_timeout: 120
response_language: en-US
repo_context_files: ['AGENTS.md']
repo_context_from_default_branch: True
repo_context_max_lines: 500
max_description_tokens: 500
max_commits_tokens: 500
max_model_tokens: 32000
model_token_count_estimate_factor: 0.3
patch_extension_skip_types: ['.md', '.txt']
allow_dynamic_context: True
max_extra_lines_before_dynamic_context: 10
patch_extra_lines_before: 5
patch_extra_lines_after: 1
cli_mode: False
large_patch_policy: clip
duplicate_prompt_examples: False
seed: -1
temperature: 0.2
ignore_pr_title: ['^\\[Auto\\]', '^Auto', '^Bump ', '^chore\\(deps\\)']
ignore_pr_target_branches: []
ignore_pr_source_branches: []
ignore_pr_labels: []
ignore_pr_authors: []
ignore_repositories: []
ignore_language_framework: []
restricted_mode: False
reasoning_effort: medium
enable_claude_extended_thinking: False
extended_thinking_budget_tokens: 2048
extended_thinking_max_output_tokens: 4096
claude_extended_thinking_models_override: []
extract_issue_from_branch: True
branch_issue_regex:
enable_custom_labels: False
[pr_reviewer] require_ticket_analysis_review: False
require_score_review: False
require_tests_review: True
require_estimate_effort_to_review: True
require_can_be_split_review: False
require_security_review: True
require_estimate_contribution_time_cost: False
require_todo_scan: False
publish_output_no_suggestions: True
persistent_comment: True
extra_instructions: Focus on: logic errors and edge cases; security/authz regressions; missing error handling;
Django/DRF correctness (migrations, N+1 queries, transaction/atomicity, serializer & permission gaps).
Do NOT comment on formatting, import order, or naming — ruff handles those in CI.
num_max_findings: 3
final_update_message: True
enable_review_labels_security: True
enable_review_labels_effort: True
require_all_thresholds_for_incremental_review: False
minimal_commits_for_incremental_review: 0
minimal_minutes_for_incremental_review: 0
enable_intro_text: True
enable_help_text: False
|
PR Code Suggestions ✨Latest suggestions up to 072087b
🛠️ Relevant configurations:
[config] enable_ai_metadata: False
custom_model_max_tokens: 200000
git_provider: github
output_relevant_configurations: True
model: openai/cx/gpt-5.5
ENABLE_AUTO_APPROVAL: True
custom_reasoning_model: False
fallback_models: ['openai/cx/gpt-5.4-mini']
is_auto_command: True
publish_output: True
publish_output_progress: True
progress_gif_url:
progress_gif_width: 48
verbosity_level: 0
use_extra_bad_extensions: False
log_level: DEBUG
use_wiki_settings_file: True
use_repo_settings_file: True
use_global_settings_file: True
extra_config_url:
disable_auto_feedback: False
ai_timeout: 120
response_language: en-US
repo_context_files: ['AGENTS.md']
repo_context_from_default_branch: True
repo_context_max_lines: 500
max_description_tokens: 500
max_commits_tokens: 500
max_model_tokens: 32000
model_token_count_estimate_factor: 0.3
patch_extension_skip_types: ['.md', '.txt']
allow_dynamic_context: True
max_extra_lines_before_dynamic_context: 10
patch_extra_lines_before: 5
patch_extra_lines_after: 1
cli_mode: False
large_patch_policy: clip
duplicate_prompt_examples: False
seed: -1
temperature: 0.2
ignore_pr_title: ['^\\[Auto\\]', '^Auto', '^Bump ', '^chore\\(deps\\)']
ignore_pr_target_branches: []
ignore_pr_source_branches: []
ignore_pr_labels: []
ignore_pr_authors: []
ignore_repositories: []
ignore_language_framework: []
restricted_mode: False
reasoning_effort: medium
enable_claude_extended_thinking: False
extended_thinking_budget_tokens: 2048
extended_thinking_max_output_tokens: 4096
claude_extended_thinking_models_override: []
extract_issue_from_branch: True
branch_issue_regex:
enable_custom_labels: False
[pr_code_suggestions] commitable_code_suggestions: False
dual_publishing_score_threshold: -1
focus_only_on_problems: True
extra_instructions: Prefer a few high-impact, project-specific suggestions over many generic ones.
Skip style/formatting (ruff-enforced) and changes under cases/migrations/.
enable_help_text: False
enable_chat_text: False
persistent_comment: True
max_history_len: 4
publish_output_no_suggestions: True
suggestions_score_threshold: 0
new_score_mechanism: True
new_score_mechanism_th_high: 9
new_score_mechanism_th_medium: 7
auto_extended_mode: True
num_code_suggestions_per_chunk: 3
max_number_of_calls: 3
parallel_calls: True
final_clip_factor: 0.8
decouple_hunks: False
demand_code_suggestions_self_review: False
code_suggestions_self_review_text: **Author self-review**: I have reviewed the PR code suggestions, and addressed the relevant ones.
approve_pr_on_self_review: False
fold_suggestions_on_self_review: True
num_code_suggestions: 4
Previous suggestionsSuggestions up to commit a5ee01d
|
PR Agent Walkthrough 🤖Welcome to the PR Agent, an AI-powered tool for automated pull request analysis, feedback, suggestions and more. Here is a list of tools you can use to interact with the PR Agent:
(1) Note that each tool can be triggered automatically when a new PR is opened, or called manually by commenting on a PR. (2) Tools marked with [*] require additional parameters to be passed. For example, to invoke the |
|
Auto-approved PR |
![jawafdehi-pr-agent[bot]](https://avatars.githubusercontent.com/u/241969530?s=60&v=4){kind=small}
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (4)
tests/casework/test_guard_wiring.py (1)
199-201: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueRecord why
--forceis required in this test.
--forceskips the pre-check GET insubmit_batch. Reads are not write-guarded, so without--forcethe pre-check would reach the patchedurlopenand the test would fail withAssertionErrorinstead of the expectedRuntimeError. The flag is load-bearing, and that is not visible from the call.♻️ Proposed comment
+ # `--force` is load-bearing: it skips the pre-check GET. Reads are not + # write-guarded, so an unforced run would hit `no_sockets` and fail with + # AssertionError before the POST guard ever fires. with pytest.raises(RuntimeError, match="refusing to write to non-loopback"): sr.main(["--batch-csv", str(batch), "--api-base-url", NON_LOOPBACK_BASE_URL, "--api-token", "t", "--apply", "--force"])🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/casework/test_guard_wiring.py` around lines 199 - 201, Add an inline comment beside the --force argument in the sr.main invocation explaining that it skips submit_batch’s unguarded pre-check GET, allowing the test to reach the expected non-loopback write RuntimeError instead of the patched urlopen assertion.tests/casework/test_api.py (1)
1061-1083: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd a case for the bare-list response shape.
reviews_for_slughandles both a paginated envelope and a bare list (casework/common/api.pyline 307). Only the envelope branch is covered. A pagination-setting change would silently break the untested branch.💚 Proposed test
+def test_reviews_for_slug_accepts_an_unpaginated_list(monkeypatch): + api = CaseworkApi("http://127.0.0.1:48010", basic=("u", "p")) + monkeypatch.setattr(api, "get", lambda *a, **kw: [{"id": 1841, "status": "done"}]) + assert api.reviews_for_slug(SLUG) == [{"id": 1841, "status": "done"}]🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/casework/test_api.py` around lines 1061 - 1083, Add a test alongside test_reviews_for_slug_unwraps_the_paginated_envelope that mocks CaseworkApi.get to return a bare list, invokes reviews_for_slug with a slug, and asserts the returned rows match that list. Keep the existing envelope and empty-result coverage unchanged.tests/casework/test_submit_reviews.py (1)
182-190: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueBuild the row with
_rowso the test tracks the real shape.The literal at lines 183-184 includes a
titlekey that_rownever produces andrender_reportnever reads. If_rowgains a column, this literal diverges from the real row shape without failing.♻️ Proposed change
- rows = [{"slug": SLUG_A, "review_id": 1841, "status": "done", "score": 84, - "disposition": "PASS", "duration": 92.4, "title": "", "error": ""}] + rows = [sr._row(SLUG_A, "done", review_id=1841, score=84, + disposition="PASS", duration=92.4)]🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/casework/test_submit_reviews.py` around lines 182 - 190, Update test_the_rendered_report_names_every_case_and_the_totals to construct its row through the existing _row helper instead of an inline dictionary, preserving the same values and assertions while keeping the test aligned with the real row shape.casework/common/api.py (1)
320-326: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueUse the
content_typeparameter of_headers.
_headersacceptscontent_typeand sets the header itself.create_materialat line 539 already uses that form. The manual copy here repeats logic without benefit.♻️ Proposed simplification
url = self.base_url + "/casework/reviews/submit/" body = json.dumps({"slug": slug}).encode("utf-8") - headers = dict(self._headers()) - headers["Content-Type"] = "application/json" - with self._request("POST", url, data=body, headers=headers, + with self._request("POST", url, data=body, + headers=self._headers("application/json"), timeout=timeout) as r: return json.loads(r.read().decode())🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@casework/common/api.py` around lines 320 - 326, Update the review submission request in the relevant API method to pass the JSON content type through _headers(content_type=...) and remove the manual Content-Type assignment, matching the existing create_material pattern while preserving the POST request behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@casework/README.md`:
- Around line 507-511: Update the paragraph in the README so the opening and
closing counts consistently describe the two abort classes: credential rejection
and write-guard refusal. Preserve the existing 401/403 credential details and
explanation.
In `@casework/submit_reviews.py`:
- Around line 373-379: Update main after submit_batch and its existing summary
output to return a non-zero exit code whenever stats["error"] is greater than
zero, while preserving zero for successful runs and intentional dry-run behavior
as defined by the existing contract. Use the error count from stats rather than
changing submit_batch’s per-case handling, and document the convention in the
project README only if partial failures are intentionally kept at exit 0.
---
Nitpick comments:
In `@casework/common/api.py`:
- Around line 320-326: Update the review submission request in the relevant API
method to pass the JSON content type through _headers(content_type=...) and
remove the manual Content-Type assignment, matching the existing create_material
pattern while preserving the POST request behavior.
In `@tests/casework/test_api.py`:
- Around line 1061-1083: Add a test alongside
test_reviews_for_slug_unwraps_the_paginated_envelope that mocks CaseworkApi.get
to return a bare list, invokes reviews_for_slug with a slug, and asserts the
returned rows match that list. Keep the existing envelope and empty-result
coverage unchanged.
In `@tests/casework/test_guard_wiring.py`:
- Around line 199-201: Add an inline comment beside the --force argument in the
sr.main invocation explaining that it skips submit_batch’s unguarded pre-check
GET, allowing the test to reach the expected non-loopback write RuntimeError
instead of the patched urlopen assertion.
In `@tests/casework/test_submit_reviews.py`:
- Around line 182-190: Update
test_the_rendered_report_names_every_case_and_the_totals to construct its row
through the existing _row helper instead of an inline dictionary, preserving the
same values and assertions while keeping the test aligned with the real row
shape.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: e76e7310-d97e-4111-b78f-ca308a17b2f7
📒 Files selected for processing (6)
casework/README.mdcasework/common/api.pycasework/submit_reviews.pytests/casework/test_api.pytests/casework/test_guard_wiring.pytests/casework/test_submit_reviews.py
…w nits CodeRabbit review on #444: - main() returned 0 even when every POST failed, which is the same hole the credential and write-guard aborts exist to close. Now returns 1 if any case errored. This diverges from the sibling enrichers, which always return 0; the divergence is documented in the README. - README said "Two failures" then "All three" of the same list. - submit_review built its headers by hand instead of using _headers's content_type parameter, as create_material already does. - Cover the bare-list branch of reviews_for_slug; pagination is a project-wide DRF setting, so the untested branch could go stale silently. - Build the render_report test row through _row -- the literal still carried the `title` key that was dropped. - Record why --force is load-bearing in the guard-wiring test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Persistent review updated to latest commit 072087b |
|
All six CodeRabbit items addressed in 072087b — 2 actionable, 4 nitpicks, all verified against the code first and all valid.
|
reviews_for_slug answers "has this case been reviewed" off page one of the list endpoint; review_detail carries `error`, which the list rows omit; submit_review POSTs through _request so the host write-guard covers it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A case that already has a review of any status is skipped, so a run that dies part-way resumes instead of re-grading at full LLM cost. 403 aborts the run (the role check fails identically on every remaining case); any other HTTP failure is recorded and the batch continues. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
--report is read-only and reads the score/disposition straight off the review list row. Only a failed row costs a detail fetch, because `error` lives on the detail serializer alone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The broad per-case except swallowed the guard's RuntimeError, so a remote run without --allow-remote-writes logged one error per case and exited 0. Caught by the end-to-end guard-wiring test, which is the only place the real client runs. Also documents submit_reviews in the casework README. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…s 403 Code review found four real defects in submit_reviews: - The pre-check GET sat outside the try/except, so one transient failure on what is half of a run's requests killed the whole batch and discarded the stats and footer. It now costs that case only, like the POST does. - The abort guard checked 403 alone. OIDCAuthentication supplies authenticate_header, so an expired token is a 401 -- the commonest credential failure was counted once per case and still exited 0. - --report logged "mode: APPLY" on a run that writes nothing. - report_rows had the same unguarded read: one blip lost a report over a batch that took hours to grade. Failures are now `unreadable` rows. Also warns when a review comes back filed under a different slug than the one submitted (retired slug via CaseSlugHistory), which otherwise re-grades that case on every run, and drops the report row's unused `title` field. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…w nits CodeRabbit review on #444: - main() returned 0 even when every POST failed, which is the same hole the credential and write-guard aborts exist to close. Now returns 1 if any case errored. This diverges from the sibling enrichers, which always return 0; the divergence is documented in the README. - README said "Two failures" then "All three" of the same list. - submit_review built its headers by hand instead of using _headers's content_type parameter, as create_material already does. - Cover the bare-list branch of reviews_for_slug; pagination is a project-wide DRF setting, so the untested branch could go stale silently. - Build the render_report test row through _row -- the literal still carried the `title` key that was dropped. - Record why --force is load-bearing in the guard-wiring test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Rebased onto Conflict and how it was resolved. One file, Verification. Recovery if this rebase is unwanted: |
There was a problem hiding this comment.
Code review is billed via overage credits. To resume reviews, an organization admin can raise the monthly limit at claude.ai/admin-settings/claude-code.
If your organization is eligible for promotional free reviews, this run could not use one — if free runs remain, retrying may succeed without raising the limit.
Once credits are available — or to retry now — push a new commit or reopen this pull request to trigger a review.
|
Persistent review updated to latest commit 072087b |
072087b to
595865e
Compare
|
Persistent review updated to latest commit 595865e |
PR Agent Walkthrough 🤖Welcome to the PR Agent, an AI-powered tool for automated pull request analysis, feedback, suggestions and more. Here is a list of tools you can use to interact with the PR Agent:
(1) Note that each tool can be triggered automatically when a new PR is opened, or called manually by commenting on a PR. (2) Tools marked with [*] require additional parameters to be passed. For example, to invoke the |
|
Auto-approved PR |
2 participants
User description
Bulk-submits a batch of enriched cases into the casework review queue, and reads the
grades back in a separate read-only pass.
Today the only ways to submit a case for review are one click in the SPA, one MCP call,
or
regrade-all— which only re-queues cases that already carry a review, so it cannever introduce a fresh batch. A 238-case batch means 238 clicks.
What it does
Submitting is instant and tells you nothing — every case comes back
pending. Gradingruns out of process on the jobs queue and takes hours, so the read-back is a separate
run. It writes
work/reviews/<ts>-submit_reviews-<run>.md: status and dispositioncounts, score spread, a per-case table, the failures with their error line, and the
batch slugs that carry no review at all.
Design notes for the reviewer
No case reads. A slug is all the POST needs, and
CaseReviewListSerializeralreadycarries the status, score, disposition and duration the report shows. So submit mode is
2 requests per case and report mode is 1, with no 16-request corpus listing in front of
either. Only a
failedrow costs a third request, for theerrorthe list row omits.Skip on any existing review. The endpoint has no idempotency — every POST creates a
new row, and the job dedup key is the review id, which is new each time. Skipping makes
a run resumable: one that dies at case 120 picks up at 121 instead of re-grading the
first 120 at full LLM cost.
--forceoverrides.An explicit target is required.
--batch-csvor--slug. A bare run is refusedrather than falling through to bulk selection, which would enqueue ~3,000 LLM grading
runs off a forgotten flag. The script deliberately does not call
select_for_run: thatpath hardcodes the DRAFT/IN_REVIEW gate, which is right for an enricher and wrong here,
because a PUBLISHED case is a legitimate review target.
No case state change. Cases stay DRAFT. Promotion to IN_REVIEW is a human decision
and the grade is its input, which is the wrong order to automate before anyone has read
a grade.
Which failures are fatal. A credential rejection (401 expired/invalid token, 403 no
Caseworker role) and the write-guard's refusal abort the run — all three fail identically
on every remaining case. Everything else costs one case and the batch continues. 401
matters specifically:
OIDCAuthenticationsuppliesauthenticate_header, so an expiredtoken is a 401, and a 403-only guard would count the commonest credential failure 238
times and still exit 0.
One known hole, mitigated not closed.
SubmitSerializerresolves retired slugsthrough
CaseSlugHistory; the list endpoint filterscase__slugand sees live slugsonly. A stale batch row therefore submits fine but stays invisible to the skip check and
is re-graded every run. Closing it properly costs a case read per slug — the exact cost
this design avoids — so instead the submit compares the requested slug against the one in
the 201 body (no extra request) and warns, naming both.
Verification
1612 passed, 1 skippedintests/casework/, ruff clean.Smoke-tested end to end against a local DEV_AUTH harness on
127.0.0.1:48010with threesqlite DBs and two seeded cases — dry run wrote nothing, apply created two reviews and two
queued jobs, the re-run skipped both, the report rendered graded/failed/never-submitted,
an unknown slug logged HTTP 400 and the batch continued,
--forcecreated a new reviewand dead-lettered the old job as
superseded, and both refusal paths (no selector, remotewithout
--allow-remote-writes) refused before opening a connection.No production writes were made at any point.
Design and plan:
docs/superpowers/specs/2026-08-10-bulk-review-submit-design.mdanddocs/superpowers/plans/2026-08-10-bulk-review-submit.mdin the meta-repo.🤖 Generated with Claude Code
PR Type
Enhancement, Tests, Documentation
Description
Batch review submission CLI
Review report generation
API review helpers
Guard/error tests
Diagram Walkthrough
File Walkthrough
api.py
Add casework review API helperscasework/common/api.py
reviews_for_slugreview_detailsubmit_reviewPOSTsubmit_reviews.py
Add batch review submit CLIcasework/submit_reviews.py
--forcetest_api.py
Cover review API client helperstests/casework/test_api.py
test_guard_wiring.py
Verify submit_reviews write guard wiringtests/casework/test_guard_wiring.py
submit_reviewsremote POST refusaltest_submit_reviews.py
Cover submit_reviews CLI behaviortests/casework/test_submit_reviews.py
README.md
Document batch review submission workflowcasework/README.md
submit_reviews🛠️ Relevant configurations:
These are the relevant configurations for this tool:
[config]
[pr_description]