Skip to content

Add bounded exploratory API comparison and blind judging - #12

Merged
dKosarevsky merged 1 commit into
mainfrom
codex/exploratory-comparison
Sep 18, 2026
Merged

dKosarevsky merged 1 commit into
mainfrom
codex/exploratory-comparison

Conversation

@dKosarevsky

Copy link
Copy Markdown
Collaborator

Summary

  • Add an explicitly exploratory 1,920-generation comparison across Sol, Terra,
    Luna and Astra, four arms, 24 frozen EN/RU cases and five repetitions.
  • Add blinded Sol semantic judging, deterministic hard checks and per-model
    matched concise - if descriptive reports. Incomplete judgment coverage
    suppresses comparisons; failed/missing responses remain visible.
  • Share one USD75 reservation/settlement ledger across generation and judging;
    retain raw attempts and partial artifacts, never retry authentication or
    uncertain calls. No resume or automatic rerun.
  • Add a manual main-only first-attempt workflow, isolated credentials and locked
    dependencies. Existing smoke-pilot defaults and prior evidence are preserved.

This is not the governed confirmatory campaign or publication-ready evidence.
No measured improvement is claimed. The previous successful 32-response pilot
is not pooled into this run. No additional USD75 campaign is implicitly approved.

Verification

  • Focused offline tests cover the new protocol/controller/workflow and pilot
    regression behavior, with synthetic responses and no provider credentials.
  • Scoped Ruff checks and whitespace checks.
  • Independent specification and quality reviews before integration.
  • All protected-main CI jobs must pass before merge; no protection bypass.

Integration boundary

This PR is one exact-file commit on current main. It does not push the historical
experimental branch or activate the unfinished governed Runtime. A live API
dispatch is a separate, user-authorized manual action after merge.

@dKosarevsky
dKosarevsky merged commit ff628ea into main Sep 18, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant