Commit 59c4f39
committed
feat(evaluator): make a stored task runner-polymorphic via kind
A task is an evaluation unit; how it runs is a property of the task, not a
different kind of record. The target side already models this — `AgentRunnerTarget`
is a `kind`-discriminated union of codex/fabric/harbor — so the stored side now
matches, and a user manages every evaluation unit in one place regardless of
which runner executes it.
Task content moves under a discriminated `spec`:
- `EvaluatorTaskDefinition` (kind="evaluator") — intent, inputs, metrics, views
- `HarborTaskDefinition` (kind="harbor") — a reference to the task's packaged
directory in the Files service, plus Harbor's own config
Nested rather than flattened with nullable per-kind fields, so each variant's
required fields stay required and the revision digest covers the spec as a unit;
two kinds with coincidentally similar metadata cannot collide on content.
`kind` is a `Literal`, matching how the runner targets discriminate. The two
definitions live in their own modules under `api/task_definitions/`; the shared
field types they need moved to `api/fields.py`, since the definitions are
imported *by* `schemas` and cannot import back from it.
A single model per kind, rather than a stored/input pair: only `metrics` widens
on the way in, and the service narrows it to references when storing. That keeps
the API surface small at the cost of making the narrowing a service invariant
rather than a type-level one.
Harbor specifics:
- One fileset per task, so a task shared by several tasksets is stored once.
- `archive_ref` is shape-validated, so a malformed reference is rejected at
publish rather than surfacing as a download failure mid-run.
- `config` is stored but excluded from the revision digest. It is a projection
of `task.toml`, which lives inside the archive, so a real change already
moves `archive_digest`; hashing the projection too would make our revision
history sensitive to Harbor's serialization.
- Which agent runs a task is not stored: that comes from the run's target, so
the same stored task can be evaluated against different agents.
Taskset expansion rejects a member whose kind the target cannot run, rather than
projecting it onto an agent-eval DTO. A Harbor task's content is a directory of
files, not fields — a pure projection would silently produce an empty task.
Mixed tasksets remain storable; the mismatch surfaces at submit as a 422.
Note for anyone with existing task rows: this is a breaking schema change with
no migration. Rows stored in the previous flat shape fail validation on read,
which surfaces as a 500 when listing tasks. Clear them before upgrading.
Signed-off-by: Sandy Chapman <schapman@nvidia.com>1 parent 2a18647 commit 59c4f39
22 files changed
Lines changed: 1129 additions & 516 deletions
File tree
- docs/evaluator
- plugins/nemo-evaluator
- openapi
- src/nemo_evaluator
- api
- service
- task_definitions
- tests
- api
- service
- v2
- integration
- sdk
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
72 | 72 | | |
73 | 73 | | |
74 | 74 | | |
75 | | - | |
| 75 | + | |
76 | 76 | | |
77 | 77 | | |
78 | | - | |
79 | | - | |
80 | | - | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
81 | 83 | | |
82 | 84 | | |
83 | 85 | | |
84 | 86 | | |
85 | | - | |
| 87 | + | |
86 | 88 | | |
87 | 89 | | |
88 | 90 | | |
89 | 91 | | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
90 | 105 | | |
91 | 106 | | |
92 | 107 | | |
93 | 108 | | |
94 | 109 | | |
95 | 110 | | |
96 | | - | |
97 | | - | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
98 | 125 | | |
99 | 126 | | |
100 | 127 | | |
| |||
107 | 134 | | |
108 | 135 | | |
109 | 136 | | |
110 | | - | |
| 137 | + | |
111 | 138 | | |
112 | 139 | | |
113 | 140 | | |
| |||
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
0 commit comments