From ac680a800dea5491747ebb2cca530c3905f00133 Mon Sep 17 00:00:00 2001 From: Matt Kornfield Date: Tue, 18 Aug 2026 10:02:09 -0700 Subject: [PATCH 1/2] fix: address review feedback for submit-only migration MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Clean up incomplete run→submit search/replace leftovers in docs and author skills, restore critical submit CLI regression tests, and fix invalid submit_remote(sdk=...) examples after rebasing onto main. Signed-off-by: Matt Kornfield --- docs/agents/optimization.mdx | 20 ++--- docs/anonymizer/cli.mdx | 20 ++--- docs/data-designer/cli.mdx | 31 +------- docs/evaluator/metrics/llm-as-a-judge.mdx | 10 +-- docs/evaluator/metrics/results.mdx | 2 +- docs/fern/gated-nav.yml | 2 +- docs/fern/versions/latest.yml | 2 +- .../about/host-local-development.mdx | 2 +- docs/safe-synthesizer/getting-started.mdx | 2 +- packages/nemo_platform_plugin/AGENTS.md | 4 +- .../.agents/skills/creating-a-plugin/SKILL.md | 6 +- .../.agents/skills/plugin-function/SKILL.md | 6 +- .../.agents/skills/plugin-job/SKILL.md | 17 ++-- .../nemo_platform_plugin/docs/ARCHITECTURE.md | 8 +- .../nemo_platform_plugin/docs/QUICKSTART.md | 5 +- .../nemo_platform_plugin/run_dependencies.py | 5 +- .../tests/test_cli_hooks.py | 12 ++- .../tests/test_commands.py | 78 +++++++++++++++++++ .../skills/agents-optimize/SKILL.md | 18 ++--- .../skills/anonymizer/SKILL.md | 2 +- .../tests/test_skill_examples.py | 9 +-- .../examples/hermes-optimize/README.md | 12 +-- .../tasks/safe_synthesizer/__main__.py | 2 + skills/nemo-evaluator-plugin/SKILL.md | 9 +-- 24 files changed, 170 insertions(+), 114 deletions(-) diff --git a/docs/agents/optimization.mdx b/docs/agents/optimization.mdx index e8b6aacedb..c8e1e4f5fb 100644 --- a/docs/agents/optimization.mdx +++ b/docs/agents/optimization.mdx @@ -24,7 +24,7 @@ evaluation result before promotion. | ------------------- | ------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | | Model optimization | An agent uses a single frontier model where a smaller model or route split may preserve quality at lower cost | Suggests a model swap or Switchyard random-routing virtual model | | Skill optimization | The agent uses skills and has an evaluation suite | Suggests running `nemo agents optimize-skills` to improve skill files and keep changes that pass evaluation | -| Prompt optimization | The agent has an optimization config and baseline dataset | Suggests `nemo agents optimize run` for Fabric-backed tuning | +| Prompt optimization | The agent has an optimization config and baseline dataset | Suggests `nemo agents optimize submit` for Fabric-backed tuning | | New model scan | Difference between the current model list and the previous optimizer snapshot | Suggests evaluating or auditing newly available models | Optimizer state is stored in the `nemo-agent-optimizer` fileset: @@ -195,7 +195,7 @@ and keeps the change only when the evaluation result improves. ```bash -nemo agents optimize-skills run --spec-file .agent-improver.yml +nemo agents optimize-skills submit --spec-file .agent-improver.yml ``` Set `open_pr: true` in the YAML when you want the loop to prepare a @@ -230,6 +230,7 @@ What it does under the hood: ```python +import os import yaml from pathlib import Path @@ -241,6 +242,7 @@ NemoJobScheduler().submit_remote( OptimizeSkillsJob, spec, workspace="default", + base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"), ) ``` @@ -273,7 +275,7 @@ nemo files list nemo-agent-telemetry ## Run Prompt and Parameter Tuning -The `nemo agents optimize run` command runs Fabric-backed numeric +The `nemo agents optimize submit` command runs Fabric-backed numeric optimization through `agents.optimize` (implementation in `nemo-optimization`). Input must be a Fabric-native agent package (`schema_version: fabric.agent/v1alpha1`). The golden-path harness is @@ -294,7 +296,7 @@ After `uv sync --package nemo-agents-plugin` (and activating `.venv`), invoke ```bash -nemo agents optimize run \ +nemo agents optimize submit \ --optimize-config "$(pwd)/plugins/nemo-optimization/examples/hermes-optimize/optimize-chatonly.yaml" \ --workspace default ``` @@ -306,7 +308,7 @@ Ask your coding agent: > Run prompt tuning on my deployed agent against this optimization config. -The `agents-optimize` skill suggests `nemo agents optimize run` when the +The `agents-optimize` skill suggests `nemo agents optimize submit` when the agent has an optimization config and a baseline dataset. Verify it is installed: @@ -317,7 +319,7 @@ nemo skills show agents-optimize What it does under the hood: - Confirms the agent has a Fabric-native optimization YAML. -- Runs `nemo agents optimize run` (or `submit` for platform jobs). +- Runs `nemo agents optimize submit` (or `submit` for platform jobs). - Compares results against the evaluation baseline and surfaces deltas for review. @@ -349,7 +351,7 @@ result = NemoJobScheduler().submit_remote( "workspace": WORKSPACE, }, workspace=WORKSPACE, - sdk=client, + base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"), ) print(result) ``` @@ -376,7 +378,7 @@ export PHISHING_MCP_BIN="$PHISHING_AGENT_ROOT/.venv/bin/email-phishing-analyzer- # These environment variables are templated into optimize-mcp.yaml. -nemo agents optimize run \ +nemo agents optimize submit \ --optimize-config "$(pwd)/plugins/nemo-optimization/examples/hermes-optimize/optimize-mcp.yaml" \ --workspace default ``` @@ -421,7 +423,7 @@ result = NemoJobScheduler().submit_remote( "workspace": WORKSPACE, }, workspace=WORKSPACE, - sdk=client, + base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"), ) print(result) ``` diff --git a/docs/anonymizer/cli.mdx b/docs/anonymizer/cli.mdx index 5f42d667e7..ed664f1299 100644 --- a/docs/anonymizer/cli.mdx +++ b/docs/anonymizer/cli.mdx @@ -14,9 +14,7 @@ This reference covers the `nemo anonymizer` commands exposed by the Anonymizer p | Command | Source | Description | |----------------------------------|---------------------------------|--------------------------------------------------------------| | `nemo anonymizer validate` | Manual Typer command | Validate an `AnonymizerConfig` (and optional `model_configs`).| -| `nemo anonymizer preview submit` | Generated from `NemoFunction` | Local streaming preview. | | `nemo anonymizer preview submit` | Generated from `NemoFunction` | Remote streaming preview against the plugin service. | -| `nemo anonymizer run submit` | Generated from `NemoJob` | Local job execution in the CLI process. | | `nemo anonymizer run submit` | Generated from `NemoJob` | Submit an `anonymizer.run` job to the NeMo Platform Jobs worker. | | `nemo anonymizer run explain` | Generated from `NemoJob` | Print the job key, submit endpoint, and JSON schemas. | @@ -39,13 +37,9 @@ The command does not accept `data.source`. Input-source validation happens durin ## `nemo anonymizer preview` -Both `preview submit` and `preview submit` take a spec file matching `PreviewRequest`. +`preview submit` takes a spec file matching `PreviewRequest`. ```bash -nemo anonymizer preview submit \ - --spec-file /tmp/anonymizer-preview.yaml \ - --workspace "${NMP_WORKSPACE:-default}" - nemo anonymizer preview submit \ --spec-file /tmp/anonymizer-preview.yaml \ --workspace "${NMP_WORKSPACE:-default}" \ @@ -60,11 +54,11 @@ nemo anonymizer preview submit \ ### Preview source kinds -| Form | `preview submit` | `preview submit` | -|---------------------------------------|---------------|------------------| -| Local path (`/tmp/input.csv`) | yes | no | -| HTTP(S) URL (`https://.../input.csv`) | yes | yes | -| Fileset reference (`fs#path`) | yes | yes | +| Form | `preview submit` | +|---------------------------------------|------------------| +| Local path (`/tmp/input.csv`) | no | +| HTTP(S) URL (`https://.../input.csv`) | yes | +| Fileset reference (`fs#path`) | yes | ### Preview output @@ -152,7 +146,7 @@ Both preview and run specs use the shared `AnonymizerRequest` shape: | `data.text_column`| string | no | Defaults to `text`. | | `data.id_column` | string | no | Optional record identifier column. | | `data.data_summary` | string | no | Optional short description of the data. | -| `model_configs` | list of Data Designer `ModelConfig` | depends | Required for `preview submit` and `run submit`; optional for `preview submit` and `run submit`. | +| `model_configs` | list of Data Designer `ModelConfig` | depends | Required for `preview submit` and `run submit`. | | `selected_models` | object with `detection` / `replace` / `rewrite` | no | Role overrides on top of bundled defaults. Requires `model_configs`. | Preview-only: diff --git a/docs/data-designer/cli.mdx b/docs/data-designer/cli.mdx index b203a1eadc..c9f7f7b501 100644 --- a/docs/data-designer/cli.mdx +++ b/docs/data-designer/cli.mdx @@ -7,7 +7,7 @@ description: "" --- -The NeMo Data Designer plugin adds the `nemo data-designer` command group. Use it to execute Data Designer workloads locally in the CLI process or submit them to NeMo Services. +The NeMo Data Designer plugin adds the `nemo data-designer` command group. Use it to submit preview and create workloads to NeMo Services. ## Configuration Sources @@ -16,6 +16,7 @@ The `preview` and `create` commands accept a configuration source path. The most ```python import data_designer.config as dd + def load_config_builder() -> dd.DataDesignerConfigBuilder: model_configs = [ dd.ModelConfig( @@ -30,41 +31,17 @@ def load_config_builder() -> dd.DataDesignerConfigBuilder: return config_builder ``` -The same configuration source can usually be used with `run` or `submit`. Resource choices determine whether it is compatible with NeMo Services execution; see [Execution Modes](/documentation/design-synthetic-data/execution-modes). - -## Run Versus Submit +The same configuration source is used with `submit`. Resource choices determine whether it is compatible with NeMo Services execution; see [Execution Modes](/documentation/design-synthetic-data/execution-modes). -`run` executes the Data Designer workload locally, in the CLI process. This can be fully local, but it is not an offline-only mode. A local run can still use the Files API, Secrets API, and Inference Gateway API from a running NeMo Services cluster when the configuration references the corresponding resources. +## Submit Versus Platform Execution `submit` sends the workload to NeMo Services. The Data Designer API and Jobs API coordinate execution, job lifecycle, logs, and artifact persistence. The NeMo Services deployment may itself be local or remote. | Command | Workload execution | NeMo Services required? | |---------|--------------------|-------------------------| -| `preview submit` | Local CLI process | Optional | -| `create submit` | Local CLI process | Optional | | `preview submit` | Data Designer API | Yes | | `create submit` | Jobs worker | Yes | -## Preview Locally - -Use local preview for fast iteration: - -```bash -nemo data-designer preview submit product_reviews.py --num-records 5 -``` - -The workload runs in your current Python environment. It can use local-only resources, NeMo resources, or both. - -## Create Locally - -Use local create when you want to generate a larger dataset without submitting work to NeMo Services: - -```bash -nemo data-designer create submit product_reviews.py --num-records 1000 -``` - -This executes the plugin job locally. It is useful for development and for workloads that should stay in the local environment. - ## Submit Preview to NeMo Services Submit preview when you want to exercise the Data Designer API path: diff --git a/docs/evaluator/metrics/llm-as-a-judge.mdx b/docs/evaluator/metrics/llm-as-a-judge.mdx index 999eaf000c..84fc85b6dc 100644 --- a/docs/evaluator/metrics/llm-as-a-judge.mdx +++ b/docs/evaluator/metrics/llm-as-a-judge.mdx @@ -17,11 +17,11 @@ LLM-as-a-Judge evaluation sends each dataset row to a judge LLM and parses the j - **Pre-generated data**: Score existing question-answer pairs or conversations. - **Custom criteria**: Define range scores, rubric scores, prompt templates, and parser behavior. -NeMo Evaluator supports two execution modes through the Evaluator plugin SDK: +Use the Evaluator plugin SDK to submit durable platform jobs: | Mode | Use Case | SDK Call | |------|----------|----------| -| **Durable remote job** | Production workloads that should run as platform jobs | `evaluator.submit(metric=metric, dataset=dataset)` | +| **Platform job** | Durable platform execution with polling and result retrieval | `evaluator.submit(metric=metric, dataset=dataset)` | ## Prerequisites @@ -125,8 +125,6 @@ result = LocalEvaluator().run_sync( }, ], ) -job.wait_until_done() -result = job.get_result() for score in result.aggregate_scores.scores: print(f"{score.name}: mean={score.mean:.2f}, count={score.count}") @@ -230,8 +228,6 @@ result = LocalEvaluator().run_sync( ], aggregate_fields=("rubric_distribution", "mode_category"), ) -job.wait_until_done() -result = job.get_result() print(result.aggregate_scores.model_dump(exclude_none=True)) ``` @@ -251,8 +247,6 @@ result = LocalEvaluator().run_sync( ], aggregate_fields=("std_dev", "variance"), ) -job.wait_until_done() -result = job.get_result() for score in result.aggregate_scores.scores: print(f"{score.name}:") diff --git a/docs/evaluator/metrics/results.mdx b/docs/evaluator/metrics/results.mdx index 80669da6f0..fb89a689e3 100644 --- a/docs/evaluator/metrics/results.mdx +++ b/docs/evaluator/metrics/results.mdx @@ -16,7 +16,7 @@ An `EvaluationResult` contains: - **Aggregate scores**: `result.aggregate_scores.scores`, with statistics such as mean, min, max, count, NaN count, variance, standard deviation, percentiles, and rubric distributions when applicable. - **Row scores**: `result.row_scores`, with the original dataset row, generated sample payload, metric scores, request logs, and row-level errors. -## Get Results from a Local Run +## Get Results Inline ```python from nemo_evaluator_sdk import Evaluator as LocalEvaluator diff --git a/docs/fern/gated-nav.yml b/docs/fern/gated-nav.yml index af78af1ce7..599eae6d18 100644 --- a/docs/fern/gated-nav.yml +++ b/docs/fern/gated-nav.yml @@ -62,7 +62,7 @@ path: ../../safe-synthesizer/about/data-synthesis.mdx - page: Evaluation path: ../../safe-synthesizer/about/evaluation.mdx - - page: Host-Local Development + - page: Runtime Environment path: ../../safe-synthesizer/about/host-local-development.mdx - page: Overview path: ../../safe-synthesizer/about/index.mdx diff --git a/docs/fern/versions/latest.yml b/docs/fern/versions/latest.yml index fa6c701146..57a2b5a93a 100644 --- a/docs/fern/versions/latest.yml +++ b/docs/fern/versions/latest.yml @@ -310,7 +310,7 @@ navigation: path: ../../safe-synthesizer/about/evaluation.mdx - page: Jobs path: ../../safe-synthesizer/about/jobs.mdx - - page: Local and Subprocess Execution + - page: Runtime Environment path: ../../safe-synthesizer/about/host-local-development.mdx - page: Parameters Reference path: ../../safe-synthesizer/about/reference.mdx diff --git a/docs/safe-synthesizer/about/host-local-development.mdx b/docs/safe-synthesizer/about/host-local-development.mdx index e4a2e2884f..f1fbae7379 100644 --- a/docs/safe-synthesizer/about/host-local-development.mdx +++ b/docs/safe-synthesizer/about/host-local-development.mdx @@ -8,7 +8,7 @@ description: "" -NeMo Safe Synthesizer no longer exposes a host-local workload command. Create synthesis jobs through the platform Jobs API or SDK so scheduling, storage, logs, and artifacts all flow through the supported job path. +Create Safe Synthesizer workloads through the platform Jobs API or SDK so scheduling, storage, logs, and artifacts all flow through the supported job path. The `nemo safe-synthesizer` CLI still includes runtime utilities for inspecting or preparing the task runtime used by development and job debugging. diff --git a/docs/safe-synthesizer/getting-started.mdx b/docs/safe-synthesizer/getting-started.mdx index 5e714d21c4..a9e67cda4c 100644 --- a/docs/safe-synthesizer/getting-started.mdx +++ b/docs/safe-synthesizer/getting-started.mdx @@ -15,7 +15,7 @@ Before using NeMo Safe Synthesizer, complete [Setup](/documentation/get-started) NeMo Safe Synthesizer has the following additional requirements: -- A GPU-capable Jobs backend with enough VRAM for the selected model and dataset. +- A GPU-capable Jobs backend with **80GB+ VRAM** recommended for the selected model and dataset (check with `nvidia-smi` on GPU nodes). - Sufficient disk space for generated datasets (50GB+ recommended) For general platform troubleshooting (port conflicts, health checks, and so on), refer to [Setup](/documentation/get-started). diff --git a/packages/nemo_platform_plugin/AGENTS.md b/packages/nemo_platform_plugin/AGENTS.md index cd725bc101..c6871fd0dc 100644 --- a/packages/nemo_platform_plugin/AGENTS.md +++ b/packages/nemo_platform_plugin/AGENTS.md @@ -21,7 +21,7 @@ Before writing any plugin code, load the relevant skill. Skills contain exact im - **`creating-a-plugin`** → starting a new plugin, setting up `pyproject.toml`, registering entry-points, or asking how plugins are discovered - **`plugin-entities`** → defining `NemoEntity` subclasses, CRUD operations, optimistic locking, filter patterns - **`plugin-config`** → adding `NemoConfig` fields, env var naming, test overrides -- **`plugin-job`** → adding `NemoJob` surfaces, the three-verb CLI (`run` / `submit` / `explain`), `spec_schema` / `input_spec_schema` / `to_spec` / `compile`, mounting routes with `add_job_routes`, container execution +- **`plugin-job`** → adding `NemoJob` surfaces, the submit/explain CLI (`submit` / `explain`), `spec_schema` / `input_spec_schema` / `to_spec` / `compile`, mounting routes with `add_job_routes`, container execution - **`plugin-service`** → adding HTTP routes with `NemoService`, `RouterSpec`, response schemas, pagination - **`plugin-authz`** → declaring HTTP authorization on plugin routes: `@path_rule`, `AuthzScope` / `PermissionSet`, caller-kind (`PRINCIPAL` vs `SERVICE_PRINCIPAL`), the `hard_fail` bundle build, migrating off `get_authz_contribution` - **`plugin-controller`** → background reconcile loops with `NemoController`, `on_startup()` patterns, service-principal clients @@ -41,7 +41,7 @@ Before writing any plugin code, load the relevant skill. Skills contain exact im - [`creating-a-plugin`](src/nemo_platform_plugin/.agents/skills/creating-a-plugin/SKILL.md) — Creates a new NeMo plugin from scratch. Use when starting plugin development, setting up a plugin package, registering surfaces via entry points, or asking how plugins are discovered by the platform. _Trigger keywords: create plugin, new plugin, plugin setup, entry-points, plugin structure, get started, plugin discovered, entry point._ - [`plugin-entities`](src/nemo_platform_plugin/.agents/skills/plugin-entities/SKILL.md) — Defines `NemoEntity` subclasses and uses `NemoEntitiesClient` for CRUD in the NeMo Platform entity store. Use when defining a new entity type, storing plugin data in the entity store, handling optimistic locking conflicts, listing entities with filters, or building entity clients for controllers. _Trigger keywords: entity, entity store, NemoEntity, entity_type, NemoEntitiesClient, entity client, store data, optimistic lock, EntityConflictError, EntityNotFoundError._ - [`plugin-config`](src/nemo_platform_plugin/.agents/skills/plugin-config/SKILL.md) — Creates plugin configuration using `NemoConfig` with environment variables and YAML file support. Use when adding plugin configuration fields, reading config values at runtime, setting up test config overrides, or understanding the env var naming formula. _Trigger keywords: config, configuration, NemoConfig, env var, environment variable, plugin_name, NMP_CONFIG, YAML config, config override, test config._ -- [`plugin-job`](src/nemo_platform_plugin/.agents/skills/plugin-job/SKILL.md) — Creates schedulable `NemoJob` surfaces for NeMo Platform plugins. Use when adding a job, declaring `spec_schema` / `input_spec_schema` / `to_spec` / `compile`, mounting job routes with `add_job_routes`, understanding the three CLI verbs (`run` / `submit` / `explain`), or running jobs in containers. _Trigger keywords: job, NemoJob, spec_schema, input_spec_schema, to_spec, compile, add_job_routes, nemo_platform_plugin.jobs, three verbs, run, submit, explain, NemoJobScheduler._ +- [`plugin-job`](src/nemo_platform_plugin/.agents/skills/plugin-job/SKILL.md) — Creates schedulable `NemoJob` surfaces for NeMo Platform plugins. Use when adding a job, declaring `spec_schema` / `input_spec_schema` / `to_spec` / `compile`, mounting job routes with `add_job_routes`, understanding the CLI verbs (`submit` / `explain`), or running jobs in containers. _Trigger keywords: job, NemoJob, spec_schema, input_spec_schema, to_spec, compile, add_job_routes, nemo_platform_plugin.jobs, three verbs, run, submit, explain, NemoJobScheduler._ - [`plugin-service`](src/nemo_platform_plugin/.agents/skills/plugin-service/SKILL.md) — Builds HTTP service surfaces for NeMo Platform plugins using `NemoService`, `RouterSpec`, `NemoListResponse`, and `NemoFilter`. Use when adding REST API routes to a plugin, implementing CRUD endpoints, handling pagination and filtering, or testing FastAPI routes. _Trigger keywords: HTTP routes, REST API, FastAPI, CRUD, endpoint, router, NemoService, pagination, filter, list endpoint, NemoListResponse, RouterSpec._ - [`plugin-authz`](src/nemo_platform_plugin/.agents/skills/plugin-authz/SKILL.md) — Declares HTTP authorization on plugin routes with `@path_rule`, `AuthzScope`, and `PermissionSet`. Use when attaching authz rules to route handlers, picking caller kinds (`PRINCIPAL` vs `SERVICE_PRINCIPAL`), passing `authz=` to `add_job_routes` / `add_function_routes`, granting permissions with no 1:1 route via `extra_permissions` / `extra_role_permissions`, or migrating a plugin off the removed `get_authz_contribution`. _Trigger keywords: authz, authorization, path_rule, AuthzScope, PermissionSet, perm, permission, caller kind, PRINCIPAL, SERVICE_PRINCIPAL, OPA bundle, hard_fail, on_invalid_plugin, extra_permissions, extra_role_permissions, get_authz_contribution._ - [`plugin-controller`](src/nemo_platform_plugin/.agents/skills/plugin-controller/SKILL.md) — Creates background reconcile-loop controllers using `NemoController`. Use when implementing state-machine reconciliation, running periodic background work, managing deployment lifecycle, building service-principal entity clients for background use, or understanding controller startup/shutdown sequence. _Trigger keywords: controller, NemoController, reconcile, background loop, reconcile_one, list_objects, on_startup, state machine, deployment lifecycle, service principal, interval_seconds._ diff --git a/packages/nemo_platform_plugin/src/nemo_platform_plugin/.agents/skills/creating-a-plugin/SKILL.md b/packages/nemo_platform_plugin/src/nemo_platform_plugin/.agents/skills/creating-a-plugin/SKILL.md index 4f3f017d0c..37052a89a5 100644 --- a/packages/nemo_platform_plugin/src/nemo_platform_plugin/.agents/skills/creating-a-plugin/SKILL.md +++ b/packages/nemo_platform_plugin/src/nemo_platform_plugin/.agents/skills/creating-a-plugin/SKILL.md @@ -174,7 +174,7 @@ class SayHelloJob(NemoJob): ... ``` -Entry-point key uses dot: `"my-plugin.say-hello"` under the `nemo.jobs` group. The platform auto-generates `nemo my-plugin say-hello run / submit / explain`. Mount server routes with `add_job_routes(SayHelloJob, authz=AuthzScope("my-plugin"))` from `nemo_platform_plugin.jobs.routes` — the `authz=` kwarg is required, or the generated routes are unruled and fail the OPA bundle build. See the `plugin-job` skill for the full pattern. +Entry-point key uses dot: `"my-plugin.say-hello"` under the `nemo.jobs` group. The platform auto-generates `nemo my-plugin say-hello submit / explain`. Mount server routes with `add_job_routes(SayHelloJob, authz=AuthzScope("my-plugin"))` from `nemo_platform_plugin.jobs.routes` — the `authz=` kwarg is required, or the generated routes are unruled and fail the OPA bundle build. See the `plugin-job` skill for the full pattern. **Add a function:** @@ -199,7 +199,7 @@ class GreetFunction(NemoFunction[GreetSpec]): return GreetResponse(message=f"Hello, {spec.name}!") ``` -Entry-point key uses dot: `"my-plugin.greet"` under the `nemo.functions` group. The platform auto-generates `nemo my-plugin greet run / submit` (two verbs — no `explain`). Mount the HTTP route inside your `NemoService` with `add_function_routes(GreetFunction, authz=AuthzScope("my-plugin"), permission_description="Invoke the greet function")` from `nemo_platform_plugin.functions.routes` — the `authz=` kwarg is required, or the route is unruled and fails the OPA bundle build. Streaming functions return an `AsyncIterator` (one NDJSON frame per line); non-streaming ones return a value. `run` **must be `async def`** — sync work goes through `await asyncio.to_thread(...)`. See the `plugin-function` skill for the full pattern. +Entry-point key uses dot: `"my-plugin.greet"` under the `nemo.functions` group. The platform auto-generates `nemo my-plugin greet submit` (two verbs — no `explain`). Mount the HTTP route inside your `NemoService` with `add_function_routes(GreetFunction, authz=AuthzScope("my-plugin"), permission_description="Invoke the greet function")` from `nemo_platform_plugin.functions.routes` — the `authz=` kwarg is required, or the route is unruled and fails the OPA bundle build. Streaming functions return an `AsyncIterator` (one NDJSON frame per line); non-streaming ones return a value. `run` **must be `async def`** — sync work goes through `await asyncio.to_thread(...)`. See the `plugin-function` skill for the full pattern. **Add a controller:** @@ -246,7 +246,7 @@ Hatchling requires `packages = ["src/nemo_my_plugin"]` — this is critical for | Jobs | `"my-plugin.job-name"` | dot separator | | Functions | `"my-plugin.fn-name"` | dot separator | -Jobs and functions use a dot separator so `discover_jobs()["my-plugin.job-name"]` and `discover_functions()["my-plugin.fn-name"]` resolve unambiguously across plugins. Programmatic job execution goes through `NemoJobScheduler.run_local(job_cls, config)`; functions are the runtime themselves — `await fn_cls().run(spec, ...)`. +Jobs and functions use a dot separator so `discover_jobs()["my-plugin.job-name"]` and `discover_functions()["my-plugin.fn-name"]` resolve unambiguously across plugins. Programmatic job submission goes through `NemoJobScheduler.submit_remote(job_cls, config, base_url=...)`; functions still implement `async def run(...)` as their runtime entrypoint and are invoked via `submit` or HTTP. ## Discovery & Fault Isolation diff --git a/packages/nemo_platform_plugin/src/nemo_platform_plugin/.agents/skills/plugin-function/SKILL.md b/packages/nemo_platform_plugin/src/nemo_platform_plugin/.agents/skills/plugin-function/SKILL.md index 7ae981fabf..6a83dc09aa 100644 --- a/packages/nemo_platform_plugin/src/nemo_platform_plugin/.agents/skills/plugin-function/SKILL.md +++ b/packages/nemo_platform_plugin/src/nemo_platform_plugin/.agents/skills/plugin-function/SKILL.md @@ -3,7 +3,7 @@ # SPDX-License-Identifier: Apache-2.0 name: plugin-function -description: Creates in-process NemoFunction surfaces for NeMo Platform plugins. Use when adding a function, declaring spec_schema, mounting function routes with add_function_routes, understanding the two CLI verbs (run / submit), or streaming NDJSON frames. Trigger keywords - function, NemoFunction, spec_schema, add_function_routes, nemo_platform_plugin.functions, two verbs, run, submit, streaming, NDJSON, FunctionContext. +description: Creates in-process NemoFunction surfaces for NeMo Platform plugins. Use when adding a function, declaring spec_schema, mounting function routes with add_function_routes, understanding the CLI verb submit (plus optional explain only for jobs), or streaming NDJSON frames. Trigger keywords - function, NemoFunction, spec_schema, add_function_routes, nemo_platform_plugin.functions, submit, streaming, NDJSON, FunctionContext. --- # Plugin Functions (NemoFunction) @@ -11,7 +11,7 @@ description: Creates in-process NemoFunction surfaces for NeMo Platform plugins. A `NemoFunction` is the third primitive on a plugin, alongside `NemoResource` and `NemoJob`. It's an in-process request handler — no scheduler, no backend dispatch — that the platform exposes as both a CLI subcommand and an HTTP route automatically. ```text -nemo run [--spec '{...}' | --spec-file FILE] [--workspace W] [...] +nemo submit [--spec '{...}' | --spec-file FILE] [--workspace W] [...] nemo submit [--spec '{...}' | --spec-file FILE] \ [--base-url URL | --cluster URL] \ [--workspace W] [--request-id ID] [...] @@ -24,7 +24,7 @@ nemo submit [--spec '{...}' | --spec-file FILE] \ Every scalar leaf in `spec_schema` becomes a Typer flag automatically. Nested submodels recurse with dotted paths (`--target.url`, `--target.timeout-seconds`). For a function with `spec_schema = GreetSpec(name: str)`: ```text -$ nemo my-plugin greet run --help +$ nemo my-plugin greet submit --help ... Function Spec: --name Name to greet. diff --git a/packages/nemo_platform_plugin/src/nemo_platform_plugin/.agents/skills/plugin-job/SKILL.md b/packages/nemo_platform_plugin/src/nemo_platform_plugin/.agents/skills/plugin-job/SKILL.md index 33d72a67ed..d001aa42a7 100644 --- a/packages/nemo_platform_plugin/src/nemo_platform_plugin/.agents/skills/plugin-job/SKILL.md +++ b/packages/nemo_platform_plugin/src/nemo_platform_plugin/.agents/skills/plugin-job/SKILL.md @@ -3,22 +3,21 @@ # SPDX-License-Identifier: Apache-2.0 name: plugin-job -description: Creates schedulable NemoJob surfaces for NeMo Platform plugins. Use when adding a job, declaring spec_schema / input_spec_schema / to_spec / compile, mounting job routes with add_job_routes, understanding the three CLI verbs (run / submit / explain), or running jobs in containers. Trigger keywords - job, NemoJob, spec_schema, input_spec_schema, to_spec, compile, add_job_routes, nemo_platform_plugin.jobs, three verbs, run, submit, explain, NemoJobScheduler. +description: Creates schedulable NemoJob surfaces for NeMo Platform plugins. Use when adding a job, declaring spec_schema / input_spec_schema / to_spec / compile, mounting job routes with add_job_routes, understanding the submit/explain CLI verbs, or running jobs in containers. Trigger keywords - job, NemoJob, spec_schema, input_spec_schema, to_spec, compile, add_job_routes, nemo_platform_plugin.jobs, submit, explain, NemoJobScheduler. --- # Plugin Jobs (NemoJob) -A `NemoJob` drives three CLI verbs that the platform auto-generates from the class: +A `NemoJob` drives two CLI verbs that the platform auto-generates from the class: ``` -nemo run [--spec '{...}' | --spec-file FILE] nemo submit [--profile

] [--cluster ] \ [--spec '{...}' | --spec-file FILE] \ [-o .= ...] [--options-file FILE] nemo explain [--profile

] ``` -`run` is in-process (no platform); `submit` POSTs to the plugin service, which compiles the spec and hands it off to the Jobs service for cluster execution; `explain` prints the schemas locally. +`submit` POSTs to the plugin service, which compiles the spec and hands it off to the Jobs service for cluster execution; `explain` prints the schemas locally. ## Class Signature @@ -194,15 +193,15 @@ from nemo_platform_plugin.discovery import discover_jobs from nemo_platform_plugin.scheduler import NemoJobScheduler job_cls = discover_jobs()["data-designer.generate"] -result = NemoJobScheduler().run_local( +result = NemoJobScheduler().submit_remote( job_cls, {"num_records": 10, "model": "gpt-oss-120b"}, + base_url="http://localhost:8080", ) ``` -`run_local` is synchronous. It drives `NemoJob.to_spec` once via -`asyncio.run` to produce the canonical spec, then calls `run` directly. -`KeyError` if no job is registered under the key. +`submit_remote` POSTs the job to the plugin service. Pass `base_url` (or rely on +documented env resolution). `KeyError` if no job is registered under the key. ## In Job Containers @@ -245,7 +244,7 @@ No mocking, no FastAPI, no platform needed. See the `plugin-testing` skill for s - **`name` is the suffix only**: For entry-point key `"data-designer.generate"`, `NemoJob.name = "generate"`. Setting the full key logs a warning at startup. - **`spec_schema` is required**: `add_job_routes` raises `TypeError` if `spec_schema` is `None`. Declare it before mounting routes. -- **`compile()` default raises `NotImplementedError`**: Jobs that haven't overridden it fail at `submit` time with a clear 422. Local `run` doesn't need `compile`. +- **`compile()` default raises `NotImplementedError`**: Jobs that haven't overridden it fail at `submit` time with a clear 422. `submit` always goes through the platform compile path. - **`run()` must be synchronous**: Use `asyncio.run()` for async work inside. - **`run()` returns a JSON-serializable dict**: No datetime, no Pydantic models, no custom classes. - **Jobs are stateless**: A new instance is constructed per call; don't store state on `self`. diff --git a/packages/nemo_platform_plugin/src/nemo_platform_plugin/docs/ARCHITECTURE.md b/packages/nemo_platform_plugin/src/nemo_platform_plugin/docs/ARCHITECTURE.md index 28fc0ec7ab..bac20f7de4 100644 --- a/packages/nemo_platform_plugin/src/nemo_platform_plugin/docs/ARCHITECTURE.md +++ b/packages/nemo_platform_plugin/src/nemo_platform_plugin/docs/ARCHITECTURE.md @@ -12,7 +12,7 @@ Every plugin capability is a "surface" — a typed contract registered via a Pyt | **HTTP service** ★ | `nemo.services` | `NemoService` | `/apis//...` | wraps in `NemoServiceAdapter`, mounts FastAPI router | | **CLI** ★ | `nemo.cli` | `NemoCLI` | `nemo ` | calls `get_cli()`, mounts as Typer subcommand | | **Agent CLI** | `nemo.cli.agents` | `NemoCLI` | `nemo agents ` | mounts an agent command group under the shared `agents` namespace | -| **Job** ★ | `nemo.jobs` | `NemoJob` | key: `.` | auto-generates `run` / `submit` / `explain` CLI verbs; the scheduler drives local runs and remote submission | +| **Job** ★ | `nemo.jobs` | `NemoJob` | key: `.` | auto-generates `submit` / `explain` CLI verbs; the scheduler drives remote submission | | **Controller** ★ | `nemo.controllers` | `NemoController` | (background) | wraps in `NemoControllerAdapter`, runs reconcile loop | | SDK | `nemo.sdk` | (any class) | `nemo.` on hub | instantiated as attribute on the `NeMo` hub | | MCP | `nemo.mcp` | `() -> list[dict]` | (MCP tool list) | returns MCP tool definitions | @@ -97,7 +97,11 @@ from nemo_platform_plugin.discovery import discover_jobs from nemo_platform_plugin.scheduler import NemoJobScheduler job_cls = discover_jobs()["example.say-hello"] -NemoJobScheduler().submit_remote(job_cls, {"name": "Alice"}) +NemoJobScheduler().submit_remote( + job_cls, + {"name": "Alice"}, + base_url="http://localhost:8080", +) ``` ## Auto-generated CLI for jobs diff --git a/packages/nemo_platform_plugin/src/nemo_platform_plugin/docs/QUICKSTART.md b/packages/nemo_platform_plugin/src/nemo_platform_plugin/docs/QUICKSTART.md index 95684d84db..c2dfde6608 100644 --- a/packages/nemo_platform_plugin/src/nemo_platform_plugin/docs/QUICKSTART.md +++ b/packages/nemo_platform_plugin/src/nemo_platform_plugin/docs/QUICKSTART.md @@ -146,12 +146,9 @@ class ProcessJob(NemoJob): ... ``` -The platform auto-generates three CLI verbs per job: +The platform auto-generates two CLI verbs per job: ```bash -nemo my-plugin process run --spec '{"input": "hello"}' -# { "status": "done", "result": "HELLO" } - nemo my-plugin process submit --profile default --spec '{"input": "hello"}' # Posts the job to the plugin service; the cluster runs it. diff --git a/packages/nemo_platform_plugin/src/nemo_platform_plugin/run_dependencies.py b/packages/nemo_platform_plugin/src/nemo_platform_plugin/run_dependencies.py index aa0efa5068..c41595069a 100644 --- a/packages/nemo_platform_plugin/src/nemo_platform_plugin/run_dependencies.py +++ b/packages/nemo_platform_plugin/src/nemo_platform_plugin/run_dependencies.py @@ -20,6 +20,9 @@ class RunDependencyError(RuntimeError): """Raised when a required injected ``NemoJob.run`` parameter cannot be bound.""" +LocalRunError = RunDependencyError + + def resolve_run_kwargs( job_cls: type[NemoJob], run: Any, @@ -124,4 +127,4 @@ def _resolve_run_param( return _UNBOUND -__all__ = ["RunDependencyError", "resolve_run_kwargs"] +__all__ = ["LocalRunError", "RunDependencyError", "resolve_run_kwargs"] diff --git a/packages/nemo_platform_plugin/tests/test_cli_hooks.py b/packages/nemo_platform_plugin/tests/test_cli_hooks.py index 1c5c4f6ffb..dbda160419 100644 --- a/packages/nemo_platform_plugin/tests/test_cli_hooks.py +++ b/packages/nemo_platform_plugin/tests/test_cli_hooks.py @@ -17,6 +17,14 @@ from nemo_platform_plugin.job import NemoJob from pydantic import BaseModel from typer.testing import CliRunner +import re + +_ANSI_RE = re.compile(r"\x1b\[[0-9;]*m") + + +def _plain(text: str) -> str: + return _ANSI_RE.sub("", text) + runner = CliRunner() @@ -165,7 +173,7 @@ def submit( app = _app_with_jobs(_GreetJob, cli=_CLI()) help_result = runner.invoke(app, ["greet", "submit", "--help"]) assert help_result.exit_code == 0 - assert "--name" in help_result.output + assert "--name" in _plain(help_result.output) result = runner.invoke(app, ["greet", "submit", "--name", "Wrapped"]) assert result.exit_code == 0 @@ -258,7 +266,7 @@ def submit( app = _app_with_functions(_GreetFunction, cli=_CLI()) help_result = runner.invoke(app, ["greet", "submit", "--help"]) assert help_result.exit_code == 0 - assert "--nickname" in help_result.output + assert "--nickname" in _plain(help_result.output) result = runner.invoke(app, ["greet", "submit", "--nickname", "Wrapped"]) assert result.exit_code == 0 diff --git a/packages/nemo_platform_plugin/tests/test_commands.py b/packages/nemo_platform_plugin/tests/test_commands.py index 5986d212de..867453d4fa 100644 --- a/packages/nemo_platform_plugin/tests/test_commands.py +++ b/packages/nemo_platform_plugin/tests/test_commands.py @@ -6,9 +6,12 @@ from __future__ import annotations import json +import re from pathlib import Path +from types import SimpleNamespace from typing import AsyncIterator +import httpx import pytest import typer from nemo_platform_plugin import commands @@ -19,6 +22,12 @@ from pydantic import BaseModel from typer.testing import CliRunner +_ANSI_RE = re.compile(r"\x1b\[[0-9;]*m") + + +def _plain(text: str) -> str: + return _ANSI_RE.sub("", text) + @pytest.fixture(autouse=True) def clear_discovery_cache(): @@ -274,3 +283,72 @@ def test_function_submit_rejects_removed_run_verb() -> None: assert result.exit_code != 0 assert "No such command" in result.output + + +def test_submit_help_lists_options_flags() -> None: + app = _app_with_jobs(_GreetJob) + result = runner.invoke(app, ["greet", "submit", "--help"]) + + assert result.exit_code == 0 + output = _plain(result.output) + assert "-o" in output + assert "--options-file" in output + assert "--spec-file" in output + assert "--profile" in output + + +def test_submit_malformed_dash_o_exits_cleanly() -> None: + app = _app_with_jobs(_GreetJob) + result = runner.invoke( + app, + ["greet", "submit", "--profile", "research", "-o", "slurm.partition"], + ) + + assert result.exit_code != 0 + combined = (result.output or "") + (result.stderr or "") + assert "KEY=VALUE" in combined or "invalid -o entry" in combined + + +def test_submit_returns_exit_code_2_on_connect_error(monkeypatch: pytest.MonkeyPatch) -> None: + request = httpx.Request("POST", "http://test/apis/plugin/v2/workspaces/default/jobs/greet") + + def _raise_connect(*_args, **_kwargs) -> dict: + raise httpx.ConnectError("Connection refused", request=request) + + monkeypatch.setattr(commands.NemoJobScheduler, "submit_remote", _raise_connect) + app = _app_with_jobs(_GreetJob) + result = runner.invoke(app, ["greet", "submit", "--base-url", "http://test", "--name", "X"]) + + assert result.exit_code == 2 + combined = (result.output or "") + (result.stderr or "") + assert "Connection refused" in combined + assert "Traceback" not in combined + + +def test_submit_passes_cli_auth_headers(monkeypatch: pytest.MonkeyPatch) -> None: + captured: dict[str, object] = {} + + def _capture(_self, _job_cls, _spec, headers=None, **_kwargs) -> dict: + captured["headers"] = headers + return {"id": "job-123"} + + class _State: + def get_sdk_context(self) -> SimpleNamespace: + return SimpleNamespace( + user=SimpleNamespace( + get_client_config=lambda: { + "default_headers": {"Authorization": "Bearer test-token"}, + } + ) + ) + + monkeypatch.setattr(commands.NemoJobScheduler, "submit_remote", _capture) + app = _app_with_jobs(_GreetJob) + result = runner.invoke( + app, + ["greet", "submit", "--base-url", "http://127.0.0.1:8080", "--name", "X"], + obj=_State(), + ) + + assert result.exit_code == 0, result.output + assert captured["headers"] == {"Authorization": "Bearer test-token"} diff --git a/plugins/nemo-agents/src/nemo_agents_plugin/skills/agents-optimize/SKILL.md b/plugins/nemo-agents/src/nemo_agents_plugin/skills/agents-optimize/SKILL.md index 6dcd7683b5..9ee1332c27 100644 --- a/plugins/nemo-agents/src/nemo_agents_plugin/skills/agents-optimize/SKILL.md +++ b/plugins/nemo-agents/src/nemo_agents_plugin/skills/agents-optimize/SKILL.md @@ -72,7 +72,7 @@ model is in use, suggest creating a `random_routing` virtual model with an probabilities, wrong entity IDs in middleware config, and other middleware- silently-bypassed failure modes. - Set `model` on the suggestion to the current source model. Suggested - follow-up actions: `nemo agents evaluate run --agent --eval-config ` + follow-up actions: `nemo agents evaluate submit --agent --eval-config ` against the routed VM before promoting. This is pure config inspection — no telemetry download required. @@ -86,7 +86,7 @@ snapshot, suggest running it. Suggested actions: -- `nemo agents optimize-skills run --spec-file .agent-improver.yml` +- `nemo agents optimize-skills submit --spec-file .agent-improver.yml` (or pass an inline JSON spec via `--spec '{...}'` if no config file exists) - After it returns, apply the resulting skill diff to the agent and redeploy. @@ -101,10 +101,10 @@ snapshot, suggest running it. The job sweeps prompts and hyperparameters via Suggested actions: -- `nemo agents optimize run --agent --optimize-config ` +- `nemo agents optimize submit --agent --optimize-config ` (or submit as a platform job with `nemo agents optimize submit …`) - After it completes, apply the new prompt + hyperparameters to a sibling - agent, deploy it, and run `nemo agents evaluate run` to compare. + agent, deploy it, and run `nemo agents evaluate submit` to compare. ### 4. New model scan @@ -116,7 +116,7 @@ profile (skip guardrails / safety / GLiNER models). Each suggestion sets `model` to the new model name and includes: - Create a sibling agent pointing at ``, then run - `nemo agents evaluate run --agent --eval-config ` to + `nemo agents evaluate submit --agent --eval-config ` to compare against the current model on the baseline dataset. There is no `--model` override flag on `evaluate`; the model swap goes through the sibling agent, or by editing the eval-config YAML. @@ -315,9 +315,9 @@ nemo models list --all-pages # always --all-pages; default paginate nemo models list --filter.name nemotron # find Nemotron candidates # Optimization commands (see also: nemo-agent-skills-optimization skill) -nemo agents evaluate run --agent --eval-config -nemo agents optimize run --agent --optimize-config -nemo agents optimize-skills run --spec-file .agent-improver.yml +nemo agents evaluate submit --agent --eval-config +nemo agents optimize submit --agent --optimize-config +nemo agents optimize-skills submit --spec-file .agent-improver.yml nemo agents evaluate-suite run --spec '{"evals": "

", "agent": ""}' # Files service @@ -360,7 +360,7 @@ nemo auditor audit submit --spec '{"config": "default/", "target": "defa `new_model_scan` suggestions. Other agents' entries in the file are preserved on save. - Look up the latest evaluation result for the selected agent. Run - `nemo agents evaluate run` if no baseline exists and a dataset+config is + `nemo agents evaluate submit` if no baseline exists and a dataset+config is available; otherwise suggest the user create one. - Fetch agents and models in parallel (`nemo agents list`, `nemo models list --all-pages`). diff --git a/plugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/SKILL.md b/plugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/SKILL.md index ed1371932f..bfb0eca6ff 100644 --- a/plugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/SKILL.md +++ b/plugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/SKILL.md @@ -37,7 +37,7 @@ Read **only** the workflow file that matches the selected mode, then follow it: # Rules - Prefer CLI surfaces. Generate YAML specs and run `nemo anonymizer ...` commands unless the user explicitly asks for Python. -- Always iterate via `nemo anonymizer preview submit` or `nemo anonymizer preview submit` before running the full job. Previews are cheap and stream a small sample (default 10 records) with full detection traces. +- Always iterate via `nemo anonymizer preview submit` before running the full job with `nemo anonymizer run submit`. Previews are cheap and stream a small sample (default 10 records) with full detection traces. - When you include `config`, pick exactly one of `replace` (Annotate/Hash/Redact/Substitute) or `rewrite` on the `AnonymizerConfig`. Not both. Do not claim `config` is required for every flow; the Anonymizer library owns default config behavior and strategy semantics. See `references/replace-strategies.md` for plugin request formatting and the [library docs](https://github.com/NVIDIA-NeMo/Anonymizer/tree/main/docs) for semantics. - The input must be a single CSV or Parquet file. `text_column` defaults to `text`; set it explicitly when the free-text column has another name. If the dataset has a stable record id, also set `id_column`. See `references/inputs.md`. - The current plugin-service / Jobs paths (`preview submit`, `run submit`) require `model_configs` so requests route through the NeMo Platform Inference Gateway. See `references/model-configs.md`. diff --git a/plugins/nemo-evaluator/tests/test_skill_examples.py b/plugins/nemo-evaluator/tests/test_skill_examples.py index a83a229049..08a5d0baa1 100644 --- a/plugins/nemo-evaluator/tests/test_skill_examples.py +++ b/plugins/nemo-evaluator/tests/test_skill_examples.py @@ -638,21 +638,20 @@ def test_authored_skill_guidance_uses_submit_for_plugin_jobs() -> None: examples = "\n".join(path.read_text(encoding="utf-8") for path in sorted((root / "assets/examples").glob("*.py"))) guidance = "\n".join([*markdown.values(), examples]) - # The plugin's local execution path is being retired. Prose may name it so the - # agent knows why to avoid it; runnable snippets must never demonstrate it. + # Local plugin `run` must not appear in runnable snippets. retiring = ( "nemo evaluator evaluate run", "nemo evaluator agent-evaluate run", - "client.evaluator." + "run(", + "client.evaluator.run(", ) for path, text in markdown.items(): for block in _fenced_blocks(text): - assert not any(term in block for term in retiring), f"{path.name} demonstrates a retiring run path" + assert not any(term in block for term in retiring), f"{path.name} demonstrates a removed run path" assert not any(term in examples for term in retiring) assert "client.evaluator.create(" not in guidance normalized_skill = " ".join(markdown[root / "SKILL.md"].split()) - assert "is being retired" in normalized_skill + assert "client.evaluator.submit" in normalized_skill or "evaluator.submit" in normalized_skill assert "`nemo_evaluator_sdk.Evaluator`" in normalized_skill assert "Evaluator().run_sync(" in guidance diff --git a/plugins/nemo-optimization/examples/hermes-optimize/README.md b/plugins/nemo-optimization/examples/hermes-optimize/README.md index 7a28a31d05..a04d47105a 100644 --- a/plugins/nemo-optimization/examples/hermes-optimize/README.md +++ b/plugins/nemo-optimization/examples/hermes-optimize/README.md @@ -88,7 +88,7 @@ No MCP, no extra checkouts. Good first smoke for optimize. cd /path/to/nemo-platform source .venv/bin/activate # if not already -nemo agents optimize run \ +nemo agents optimize submit \ --optimize-config "$(pwd)/plugins/nemo-optimization/examples/hermes-optimize/optimize-chatonly.yaml" \ --workspace default ``` @@ -120,7 +120,7 @@ print( OptimizeJob, {"optimize_config": str(optimize_config), "workspace": WORKSPACE}, workspace=WORKSPACE, - sdk=client, + base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"), ) ) ``` @@ -171,7 +171,7 @@ export NEMO_AGENTS_IGW_API_KEY="${NEMO_AGENTS_IGW_API_KEY:-not-used}" ### 2. Run optimize against the stored agent ```bash -nemo agents optimize run \ +nemo agents optimize submit \ --optimize-config "$(pwd)/plugins/nemo-optimization/examples/hermes-optimize/optimize-chatonly-via-agent.yaml" \ --agent hermes-optimize-chatonly \ --workspace default @@ -235,7 +235,7 @@ export PHISHING_AGENT_ROOT="${PHISHING_AGENT_ROOT:-$HOME/work/email-phishing-ana export PHISHING_AGENT_SRC="$PHISHING_AGENT_ROOT/src" export PHISHING_MCP_BIN="$PHISHING_AGENT_ROOT/.venv/bin/email-phishing-analyzer-mcp" -nemo agents optimize run \ +nemo agents optimize submit \ --optimize-config "$(pwd)/plugins/nemo-optimization/examples/hermes-optimize/optimize-mcp.yaml" \ --workspace default ``` @@ -280,7 +280,7 @@ print( OptimizeJob, {"optimize_config": str(optimize_config), "workspace": WORKSPACE}, workspace=WORKSPACE, - sdk=client, + base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"), ) ) ``` @@ -294,7 +294,7 @@ print( | `nemo: command not found` | `source .venv/bin/activate` after `uv sync --package nemo-agents-plugin` | | `No module named hermes_cli` | Re-run the `hermes-agent==0.18.2 --no-deps` install (needed after every fresh `uv sync`) | | `No module named 'nemo_fabric_adapters'` | `export ADAPTER_PYTHON="$(pwd)/.venv/bin/python"` | -| Missing `PHISHING_AGENT_SRC` / MCP binary | Sync the phishing agent checkout; export both env vars before `optimize run` | +| Missing `PHISHING_AGENT_SRC` / MCP binary | Sync the phishing agent checkout; export both env vars before `optimize submit` | | Analyzer / LLM 401 | Confirm `NVIDIA_API_KEY` works on inference-api; keep using `analyzer-inference-api.yaml` | | Dataset / config file not found | Run from the `nemo-platform` repo root | | Agent create fails on fileset size / too many files | Pass `--agent-config` to `agents/chatonly/agent.yaml` (slim dir), not the parent examples folder | diff --git a/plugins/nemo-safe-synthesizer/src/nemo_safe_synthesizer_plugin/tasks/safe_synthesizer/__main__.py b/plugins/nemo-safe-synthesizer/src/nemo_safe_synthesizer_plugin/tasks/safe_synthesizer/__main__.py index f45f30c584..71d2339416 100644 --- a/plugins/nemo-safe-synthesizer/src/nemo_safe_synthesizer_plugin/tasks/safe_synthesizer/__main__.py +++ b/plugins/nemo-safe-synthesizer/src/nemo_safe_synthesizer_plugin/tasks/safe_synthesizer/__main__.py @@ -8,6 +8,7 @@ """ import os +import sys # Disable PyTorch inductor remote cache to avoid Redis warnings from vLLM. # These must be set before any PyTorch imports. @@ -399,6 +400,7 @@ def run_from_env() -> None: def main(argv: list[str] | None = None) -> None: """Run the task entry point from platform environment variables.""" + argv = sys.argv[1:] if argv is None else argv if argv: raise SystemExit("This task module no longer accepts local execution commands.") run_from_env() diff --git a/skills/nemo-evaluator-plugin/SKILL.md b/skills/nemo-evaluator-plugin/SKILL.md index bf8a805af0..20fc5bb5b4 100644 --- a/skills/nemo-evaluator-plugin/SKILL.md +++ b/skills/nemo-evaluator-plugin/SKILL.md @@ -51,11 +51,10 @@ metric for a rubric, RAG workflow, or tool-calling evaluation. | Task-driven platform job | `client.evaluator.submit(tasks=..., target=)` or `nemo evaluator agent-evaluate submit` | | Reusable platform definitions and result indexes | `client.evaluator.metrics`, `.tasks`, `.tasksets`, `.eval_results`, `.agent_eval_results` | -Default to `submit` for every plugin evaluation. The plugin's local execution -path is being retired: the `nemo evaluator ... run` CLI verb still exists but -should not be built on, even though `--help` still lists it. For fast metric -iteration without the platform, use the standalone `nemo_evaluator_sdk.Evaluator` -instead. +Default to `submit` for every plugin evaluation. The plugin does not expose a +local `run` path — use `client.evaluator.submit(...)` or +`nemo evaluator … submit`. For fast metric iteration without the platform, use +the standalone `nemo_evaluator_sdk.Evaluator` instead. - Read [SDK Execution](references/execution.md) for datasets, targets, configuration, field mapping, job lifecycle, and custom metric packaging. From a0ae9433dc01e0800b21fbe9e4d6ea9bd77b0f45 Mon Sep 17 00:00:00 2001 From: Matt Kornfield Date: Tue, 18 Aug 2026 14:46:59 -0700 Subject: [PATCH 2/2] docs(data-designer): address Knepper review on submit-only seeds/providers Clarify that local file/DataFrame seeds remain library-valid but not on the platform path, and that remote validate/submit is IGW-provider-only. Signed-off-by: Matt Kornfield --- docs/data-designer/tutorials/seeding.mdx | 13 +++++++------ plugins/nemo-data-designer/README.md | 4 ++-- 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/docs/data-designer/tutorials/seeding.mdx b/docs/data-designer/tutorials/seeding.mdx index c6abd73886..0045c86cd5 100644 --- a/docs/data-designer/tutorials/seeding.mdx +++ b/docs/data-designer/tutorials/seeding.mdx @@ -11,15 +11,16 @@ This tutorial demonstrates how to use external datasets as seed data for synthet For more detail about seed dataset behavior, see the [open-source library's version](https://docs.nvidia.com/nemo/datadesigner/v0.9.1/tutorials/seeding-with-an-external-dataset) of this tutorial. -## Seed Sources by Execution Mode +## Seed Sources on the Platform -Seed source support depends on platform resource availability: +This tutorial covers platform execution (`nemo data-designer … submit` and the SDK). Seed support there is narrower than the open-source Data Designer library: -| Seed source | CLI `submit` / SDK today | Use case | -|-------------|--------------------------|----------| -| **Local files or DataFrames** | Not supported | Upload local files to Filesets first. | -| **HuggingFace** | Supported | Publicly available datasets or private HuggingFace datasets. | +| Seed source | Platform (`submit` / SDK) | Notes | +|-------------|---------------------------|-------| +| **Local files or DataFrames** | Not supported on the platform path | Still valid in the [open-source library](https://docs.nvidia.com/nemo/datadesigner/v0.9.1/tutorials/seeding-with-an-external-dataset). For platform jobs, upload files to a Fileset first (see Step 1 below). | +| **HuggingFace** | Supported | Public or private HuggingFace datasets (private needs a Secrets API token). | | **Files API Filesets** | Supported | Shared seed data stored through the Files API. | + ### HuggingFace Datasets Use `HuggingFaceSeedSource` to load data from HuggingFace: diff --git a/plugins/nemo-data-designer/README.md b/plugins/nemo-data-designer/README.md index c00238fa92..f537305fa6 100644 --- a/plugins/nemo-data-designer/README.md +++ b/plugins/nemo-data-designer/README.md @@ -27,8 +27,8 @@ The exit code is `0` only when every requested context validates cleanly. JSON o ### Local vs. remote -- **Local** checks library-level compatibility: the engine compiles the config and resolves model providers. Providers can be defined locally **or** referenced by name from the Inference Gateway — both are first-class. -- **Remote** mirrors what `nemo data-designer submit` accepts: unsupported seed types and `tool_configs` are rejected, IGW providers are resolved against the platform, Files-service seeds are looked up, and Nemotron Personas filesets are checked. The remote pass is a client-side simulation of those checks; it does not contact the data-designer service. +- **Local** (`--execution-context local`) checks open-source library compatibility: the engine compiles the config and resolves model providers the library understands (including locally defined providers). This is a library/config check, not a platform execution path — there is no `nemo data-designer … run` verb. +- **Remote** (`--execution-context remote`) mirrors what `nemo data-designer submit` accepts on the platform: local seed types and `tool_configs` are rejected, **only Inference Gateway providers** are accepted, Files-service seeds are looked up, and Nemotron Personas filesets are checked. The remote pass is a client-side simulation of those checks; it does not contact the data-designer service. ### Programmatic use