chore: remove deprecated verify_raw / verify instance verification surface

[mikesklar]

Summary

Removes the deprecated verify_raw / verify instance-verification surface. verify_raw POSTed caller-supplied source code to the instance runner's /execute_verifier_function endpoint and used the long-deprecated (before, after, final_answer) -> int DatabaseSnapshot signature. verify was a thin inspect.getsource wrapper over it. The sanctioned replacement is the bundle-based verifier API (/v1/verifiers/check + /v1/verifiers/execute) via @verifier / TaskVerifier.remote(env), or fleet.tasks.verifier_from_string() for a raw code string.

This is a deliberate breaking removal, not a deprecation — no shims, no raising stubs. ValidatorType and ExecuteFunctionResponse were public exports in fleet.instance.__all__; both are gone.

Note: the runner-side POST /execute_verifier_function endpoint itself is NOT removed — it remains the live execution surface used by grading and the verifier API.

Files changed

• fleet/_async/instance/client.py — deleted verify(), verify_raw(), the ValidatorType alias; dropped now-unused imports (inspect, Callable, fleet.verifiers.DatabaseSnapshot, fleet.verifiers.parse.{convert_verifier_string,extract_function_name}, ExecuteFunctionRequest, ExecuteFunctionResponse).
• fleet/instance/client.py — sync mirror of the above (hand-edited to match; see Note on generation below).
• fleet/_async/instance/__init__.py / fleet/instance/__init__.py — removed ValidatorType and ExecuteFunctionResponse from imports and __all__.
• fleet/_async/client.py / fleet/client.py — deleted AsyncEnv.verify/verify_raw (and sync Env.verify/verify_raw) delegating methods; dropped the ValidatorType and ExecuteFunctionResponse imports.
• fleet/instance/models.py — deleted the ExecuteFunctionRequest and ExecuteFunctionResponse pydantic models (no other importers).
• fleet/verifiers/parse.py — deleted. Its only importers were the two instance clients above; confirmed dead via repo-wide grep.
• examples/json_tasks_example.py — ported off await env.verify_raw(...) to fleet._async.tasks.verifier_from_string(...).remote_with_response(env); removed the now-unnecessary local extract_function_name helper.
• examples/gemini_example.py — same treatment for the sync path (fleet.tasks.verifier_from_string(...).remote_with_response(env)); removed the local extract_function_name helper and the now-unused re import.

Migration

Before:

response = env.verify_raw(verifier_code_string, function_name="my_verifier")
if response.success:
    ...

After:

from fleet.tasks import verifier_from_string  # or fleet._async.tasks for async

verifier = verifier_from_string(
    verifier_code_string,
    verifier_id="my_verifier_id",
    verifier_key="my_verifier_key",
)
response = verifier.remote_with_response(env)  # await for the async variant
if response.success:
    ...

verifier.remote(env) is also available if you just want the numeric score back instead of the full VerifiersExecuteResponse.

⚠️ Reviewer decision needed: verify_raw is also the SDK's in-container code door

Auditing the live callers turned up a use case the verifier API does not cover, so this deserves an explicit call before merge.

verify_raw POSTs to the instance runner (https://<instance>.<region>.orchestrator.fleetai.com/api/v1/env/execute_verifier_function), so the code runs inside the instance container. The sanctioned replacement — /v1/verifiers/check + /v1/verifiers/execute via @verifier / verifier_from_string(...) — runs the bundle in an orchestrator warm-pool executor pod, not in the instance. For "grade the final state", those are equivalent and the verifier API is strictly better. For "execute this Python inside the instance", they are not.

Two of the downstream callers depend on the in-container behaviour, and both are actively maintained:

Caller	Last touched	What it uses verify_raw for
theseus-dataminer src/dataminer/tasks/verify_fleet.py:123,128	2026-07-18	Documents itself as "the shipped bridge's in-container door, verify_raw -> /execute_verifier_function -> localhost:9000". Materializes a submission into /app/workspace/submission in-container, then calls the on-box verifier.
theseus-harvey-lab scripts/build_task.py:179	2026-07-15	Calls env.verify_raw(...) from inside a generated verifier to run upstream evaluation/scoring.py extraction in-instance (needs pandoc/pandas/pdfplumber, which the executor pod does not have).

The other two callers are low-risk: theseus-env-tools fli/src/fli/commands/upgrade.py:215 (a sanity-check verifier run — a clean verifier_from_string(...) swap) and fleet-ai/env-qa fullHarness.py:1417 (last touched 2025-07-25, effectively dead).

This does not block the removal — the runner endpoint itself is untouched, so the in-container door remains fully available; callers just have to hit it directly instead of through a convenience method:

# in-container execution, after this PR
resp = env.instance.client.request(
    "POST",
    "/execute_verifier_function",
    json={"function_code": code, "function_name": "verify"},
).json()

So the options are: (a) merge as-is and let those two repos inline the ~5-line POST, or (b) keep an explicitly-named in-container escape hatch (e.g. instance.execute_in_container(code, name)) so the capability keeps a supported SDK surface while the misleading verify_raw name and the dead (before, after, final_answer) signature still go away. This PR implements (a).

Runtime-usage evidence (and an observability gap)

Asked whether anything still calls this in prod. Findings:

• Logfire: 0 records matching execute_verifier_function (sampled 24h windows on 2026-07-26→27 and 07-27→28). This is a null result, not proof of zero usage. verify_raw targets the per-instance subdomain, which is served by the env pod's runner — not by any Logfire-instrumented service. Control: POST /v1/verifiers/execute is captured on orchestrator-public-api at thousands of spans/day, so the modern path is well instrumented and heavily used.
• web-proxy CloudWatch (/ecs/web-proxy, 7d): 0 hits — also not the serving path. Control: /resources (another instance-level route) is likewise 0 there, while the group is otherwise live, so instance-level traffic never traverses web-proxy.
• Env-instance Container Insights: no HTTP access lines shipped for the sampled cluster (control "POST /" over 7d was also empty), so the runner's request log is not captured either.

Net: runner-endpoint invocations are currently unobservable in our telemetry. Worth fixing independently of this PR — it means we cannot empirically retire any instance-runner endpoint, only reason about it statically.

Known downstream callers (NOT updated in this PR)

• fleet-ai/env-qa — fullHarness.py:1417
• fleet-ai/theseus-env-tools — fli/src/fli/commands/upgrade.py:215
• fleet-ai/theseus-dataminer — src/dataminer/tasks/verify_fleet.py, tasks/upstream_footprint/verify_registered_task.py
• fleet-ai/theseus-harvey-lab — scripts/build_task.py

Test plan

• python -c "import fleet; import fleet.client; import fleet._async.client; import fleet.instance; import fleet._async.instance; import fleet.tasks; import fleet.verifiers" succeeds
• python -m compileall -q fleet examples succeeds
• python -m pytest tests/ -q — 741 passed, 11 skipped, 5 failed (same 5 pre-existing failures in tests/track/test_mcp_install.py on baseline main, unrelated to this change)
• git diff --stat reviewed — only files listed above changed, no unrelated churn

Note on sync generation: the sync tree is normally regenerated from fleet/_async/ via make unasync. Running it in this sandbox produced large unrelated diffs across ~15 files (fleet/judge.py, fleet/models.py, fleet/tasks.py, fleet/resources/sqlite.py, etc.) — confirmed via a clean-checkout run that this drift pre-exists on main and is unrelated to this change (likely an unasync tool/config version mismatch). Those files were reverted and the sync mirrors (fleet/client.py, fleet/instance/client.py, fleet/instance/__init__.py) were hand-edited to exactly mirror the async edits instead.

🤖 Created via sandbox session