Skip to content

Commit 59c4f39

Browse files
committed
feat(evaluator): make a stored task runner-polymorphic via kind
A task is an evaluation unit; how it runs is a property of the task, not a different kind of record. The target side already models this — `AgentRunnerTarget` is a `kind`-discriminated union of codex/fabric/harbor — so the stored side now matches, and a user manages every evaluation unit in one place regardless of which runner executes it. Task content moves under a discriminated `spec`: - `EvaluatorTaskDefinition` (kind="evaluator") — intent, inputs, metrics, views - `HarborTaskDefinition` (kind="harbor") — a reference to the task's packaged directory in the Files service, plus Harbor's own config Nested rather than flattened with nullable per-kind fields, so each variant's required fields stay required and the revision digest covers the spec as a unit; two kinds with coincidentally similar metadata cannot collide on content. `kind` is a `Literal`, matching how the runner targets discriminate. The two definitions live in their own modules under `api/task_definitions/`; the shared field types they need moved to `api/fields.py`, since the definitions are imported *by* `schemas` and cannot import back from it. A single model per kind, rather than a stored/input pair: only `metrics` widens on the way in, and the service narrows it to references when storing. That keeps the API surface small at the cost of making the narrowing a service invariant rather than a type-level one. Harbor specifics: - One fileset per task, so a task shared by several tasksets is stored once. - `archive_ref` is shape-validated, so a malformed reference is rejected at publish rather than surfacing as a download failure mid-run. - `config` is stored but excluded from the revision digest. It is a projection of `task.toml`, which lives inside the archive, so a real change already moves `archive_digest`; hashing the projection too would make our revision history sensitive to Harbor's serialization. - Which agent runs a task is not stored: that comes from the run's target, so the same stored task can be evaluated against different agents. Taskset expansion rejects a member whose kind the target cannot run, rather than projecting it onto an agent-eval DTO. A Harbor task's content is a directory of files, not fields — a pure projection would silently produce an empty task. Mixed tasksets remain storable; the mismatch surfaces at submit as a 422. Note for anyone with existing task rows: this is a breaking schema change with no migration. Rows stored in the previous flat shape fail validation on read, which surfaces as a 500 when listing tasks. Clear them before upgrading. Signed-off-by: Sandy Chapman <schapman@nvidia.com>
1 parent 2a18647 commit 59c4f39

22 files changed

Lines changed: 1129 additions & 516 deletions

docs/evaluator/manage-tasks-tasksets.mdx

Lines changed: 35 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -72,29 +72,56 @@ Reference the stored metric with a `MetricRef` (`workspace/name`, or a bare `nam
7272
the task's workspace). The service returns the stored `Task`.
7373

7474
```python
75-
from nemo_evaluator.api.schemas import MetadataItem, MetricRef, TaskInput, TaskInputs
75+
from nemo_evaluator.api.schemas import AgentEvalTaskDefinitionInput, MetadataItem, MetricRef, TaskInput, TaskInputs
7676

7777
task = TaskInput(
78-
intent="Answer the user's geography question with the capital city.",
79-
inputs=TaskInputs(instruction="What is the capital of France?"),
80-
metrics=[MetricRef("default/answer-exact-match")],
78+
spec=AgentEvalTaskDefinitionInput(
79+
intent="Answer the user's geography question with the capital city.",
80+
inputs=TaskInputs(instruction="What is the capital of France?"),
81+
metrics=[MetricRef("default/answer-exact-match")],
82+
),
8183
metadata=[MetadataItem(key="suite", value="geography")],
8284
)
8385

8486
stored = tasks.create("capital-of-france", task=task)
85-
print(stored.id, stored.metrics)
87+
print(stored.id, stored.spec.metrics)
8688
```
8789

8890
### `TaskInput` fields
8991

92+
| Field | Type | Required | Description |
93+
|-------|------|----------|-------------|
94+
| `spec` | `TaskSpecInput` | Yes | The task's content, discriminated by `kind` — see below. |
95+
| `metadata` | `list[MetadataItem]` | No | Key/value annotations. Keys must be unique. |
96+
| `tags` | `list[str]` | No | Tags to point at the revision this request publishes. `latest` is always applied server-side. |
97+
98+
### Task kinds
99+
100+
A task is an evaluation unit; its `kind` says which runner executes it. Both kinds are stored as the
101+
same record type, so a taskset can group them and you manage every evaluation unit in one place.
102+
103+
`AgentEvalTaskDefinitionInput` (`kind="agent_eval"`) — scored by platform metrics:
104+
90105
| Field | Type | Required | Description |
91106
|-------|------|----------|-------------|
92107
| `intent` | `str` | Yes | Human-readable description of the desired agent behavior. |
93108
| `inputs` | `TaskInputs` | No | The task's recognized input fields. `instruction` is the agent's prompt; it falls back to `intent` when unset. |
94109
| `metrics` | `list[MetricRefOrInline]` | No | The metrics that score the task, as `MetricRef` references (`workspace/name`) to stored metrics. Pre-built inline metric bundles (`MetricInline`) are also accepted and are normalized to stored metrics on create. |
95110
| `views` | `dict[str, SemanticView]` | No | Optional reporting views mapping metric outputs into named semantic scores. |
96-
| `metadata` | `list[MetadataItem]` | No | Key/value annotations. Keys must be unique. |
97-
| `tags` | `list[str]` | No | Tags to point at the revision this request publishes. `latest` is always applied server-side. |
111+
112+
`HarborTaskDefinition` (`kind="harbor"`) — a Harbor task, scored by Harbor's own reward:
113+
114+
| Field | Type | Required | Description |
115+
|-------|------|----------|-------------|
116+
| `archive_ref` | `str` | Yes | Files reference to the task's packaged directory (`workspace/fileset#path`). One fileset per task, so a task shared by several tasksets is stored once. |
117+
| `archive_digest` | `str` | Yes | Content hash Harbor computed over the task directory. |
118+
| `instruction` | `str` | No | The task's instruction text, when it has one. |
119+
| `config` | `dict` | No | Harbor's own task configuration (verifier, agent, environment, steps), stored as published. |
120+
121+
<Note>
122+
A run has one target, so it executes one kind. A taskset may group both, but submitting it against a
123+
target whose runner cannot execute a member is rejected with `422` before the run starts.
124+
</Note>
98125

99126
<Note>
100127
A stored task holds **metric references only**. Any inline metric bundle you pass on create is stored
@@ -107,7 +134,7 @@ why `stored.metrics` always comes back as a list of `MetricRef` references.
107134
```python
108135
# Retrieve one task by name (its current content)
109136
task = tasks.retrieve("capital-of-france")
110-
print(task.revision, task.tags) # e.g. 1 {'latest': 1}
137+
print(task.spec.kind, task.revision, task.tags) # e.g. agent_eval 1 {'latest': 1}
111138

112139
# List tasks in the workspace (paginated)
113140
page = tasks.list(page=1, page_size=100, sort="-created_at")

plugins/nemo-evaluator/openapi/openapi.yaml

Lines changed: 167 additions & 49 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)