feat: add manifest input for paperbanana sweep#184
Open
eureka0928 wants to merge 1 commit intollmsresearch:mainfrom
Open
feat: add manifest input for paperbanana sweep#184eureka0928 wants to merge 1 commit intollmsresearch:mainfrom
eureka0928 wants to merge 1 commit intollmsresearch:mainfrom
Conversation
Closes llmsresearch#182. Sweep today takes eight comma-separated axis flags plus --input and --caption — a 200+ char line of bash that's miserable to diff, share, or commit alongside the paper. Batch already solved this with YAML/JSON manifests; sweep is the symmetric gap. - New `load_sweep_manifest(path)` in `core/sweep.py` parses YAML/JSON manifests: required `input` + `caption`, optional `pdf_pages` and `max_variants`, optional `axes` object with the seven axis lists. Rejects unknown axis keys, wrong types, and missing requireds. Input paths resolve relative to the manifest's parent (mirrors `load_batch_manifest`). - New `--manifest` / `-m` flag on `sweep`, mutually exclusive with the axis flags. When set, `--input` and `--caption` become optional and default from the manifest. Invocation-level flags (--output-dir, --config, --format, --dry-run, --verbose, --auto-download-data) stay as CLI flags because they're invocation concerns, not plan concerns. - New `examples/sweep_manifest.yaml` and README callout. - 15 new tests covering YAML + JSON, relative-path resolution, each validation branch, and unknown-axis rejection.
Contributor
Author
|
Hi @dippatel1994 — tagging you for review when you get a chance. Quick summary:
Happy to squash, rebase, or adjust based on your feedback. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #182
Summary
paperbanana sweeptoday takes eight comma-separated axis flags plus--inputand--caption— a 200+ character line of bash that's miserable to diff, share, or commit alongside the paper.paperbanana batchalready solved this with YAML/JSON manifests; sweep is the symmetric gap.Example manifest:
Implementation
load_sweep_manifest()incore/sweep.py(~50 lines): parses YAML/JSON, validates required keys (input,caption), enforces types on optional keys (pdf_pages: str,max_variants: int >= 1), rejects unknown axis keys, resolves input paths relative to the manifest's parent directory (same convention asload_batch_manifest).--manifest/-mflag onsweep: mutually exclusive with the eight axis flags. When set,--inputand--captionbecome optional and default from the manifest.--output-dir,--config,--format,--dry-run,--verbose,--auto-download-data) because they're invocation concerns, not sweep-plan concerns.examples/sweep_manifest.yamlwith inline commentary on what belongs in the manifest vs. on the CLI.Design notes
vlm_providers) to match CLI flags (--vlm-providers), not the singular internal axis names inbuild_sweep_variants(vlm_provider).typo_axisdon't silently no-op.pdf_pagesfrom manifest only overrides when set — the CLI--pdf-pagesfalls through if the manifest omits it.Test plan
pytest tests/test_core/test_sweep.py— 48 tests pass (15 new)ruff check+ruff format --check— cleanpaperbanana sweep --manifest examples/sweep_manifest.yaml --dry-run— plans 16 variants correctly, writes sweep_report.jsonpaperbanana sweep --manifest ... --vlm-providers gemini— correctly errors with the mutual-exclusion messagecc @dippatel1994 for review.