Skip to content
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 11 additions & 9 deletions docs/agents/optimization.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ evaluation result before promotion.
| ------------------- | ------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| Model optimization | An agent uses a single frontier model where a smaller model or route split may preserve quality at lower cost | Suggests a model swap or Switchyard random-routing virtual model |
| Skill optimization | The agent uses skills and has an evaluation suite | Suggests running `nemo agents optimize-skills` to improve skill files and keep changes that pass evaluation |
| Prompt optimization | The agent has an optimization config and baseline dataset | Suggests `nemo agents optimize run` for Fabric-backed tuning |
| Prompt optimization | The agent has an optimization config and baseline dataset | Suggests `nemo agents optimize submit` for Fabric-backed tuning |
| New model scan | Difference between the current model list and the previous optimizer snapshot | Suggests evaluating or auditing newly available models |

Optimizer state is stored in the `nemo-agent-optimizer` fileset:
Expand Down Expand Up @@ -195,7 +195,7 @@ and keeps the change only when the evaluation result improves.
<Tab title="CLI">

```bash
nemo agents optimize-skills run --spec-file .agent-improver.yml
nemo agents optimize-skills submit --spec-file .agent-improver.yml
```

Set `open_pr: true` in the YAML when you want the loop to prepare a
Expand Down Expand Up @@ -230,6 +230,7 @@ What it does under the hood:
<Tab title="Python SDK">

```python
import os
import yaml
from pathlib import Path

Expand All @@ -241,6 +242,7 @@ NemoJobScheduler().submit_remote(
OptimizeSkillsJob,
spec,
workspace="default",
base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
)
```

Expand Down Expand Up @@ -273,7 +275,7 @@ nemo files list nemo-agent-telemetry

## Run Prompt and Parameter Tuning

The `nemo agents optimize run` command runs Fabric-backed numeric
The `nemo agents optimize submit` command runs Fabric-backed numeric
optimization through `agents.optimize` (implementation in
`nemo-optimization`). Input must be a Fabric-native agent package
(`schema_version: fabric.agent/v1alpha1`). The golden-path harness is
Expand All @@ -294,7 +296,7 @@ After `uv sync --package nemo-agents-plugin` (and activating `.venv`), invoke
<Tab title="CLI">

```bash
nemo agents optimize run \
nemo agents optimize submit \
--optimize-config "$(pwd)/plugins/nemo-optimization/examples/hermes-optimize/optimize-chatonly.yaml" \
--workspace default
```
Expand All @@ -306,7 +308,7 @@ Ask your coding agent:

> Run prompt tuning on my deployed agent against this optimization config.

The `agents-optimize` skill suggests `nemo agents optimize run` when the
The `agents-optimize` skill suggests `nemo agents optimize submit` when the
agent has an optimization config and a baseline dataset. Verify it is
installed:

Expand All @@ -317,7 +319,7 @@ nemo skills show agents-optimize
What it does under the hood:

- Confirms the agent has a Fabric-native optimization YAML.
- Runs `nemo agents optimize run` (or `submit` for platform jobs).
- Runs `nemo agents optimize submit` (or `submit` for platform jobs).
- Compares results against the evaluation baseline and surfaces deltas
for review.

Expand Down Expand Up @@ -349,7 +351,7 @@ result = NemoJobScheduler().submit_remote(
"workspace": WORKSPACE,
},
workspace=WORKSPACE,
sdk=client,
base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
)
print(result)
```
Expand All @@ -376,7 +378,7 @@ export PHISHING_MCP_BIN="$PHISHING_AGENT_ROOT/.venv/bin/email-phishing-analyzer-

# These environment variables are templated into optimize-mcp.yaml.

nemo agents optimize run \
nemo agents optimize submit \
--optimize-config "$(pwd)/plugins/nemo-optimization/examples/hermes-optimize/optimize-mcp.yaml" \
--workspace default
```
Expand Down Expand Up @@ -421,7 +423,7 @@ result = NemoJobScheduler().submit_remote(
"workspace": WORKSPACE,
},
workspace=WORKSPACE,
sdk=client,
base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
)
print(result)
```
Expand Down
20 changes: 7 additions & 13 deletions docs/anonymizer/cli.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,7 @@ This reference covers the `nemo anonymizer` commands exposed by the Anonymizer p
| Command | Source | Description |
|----------------------------------|---------------------------------|--------------------------------------------------------------|
| `nemo anonymizer validate` | Manual Typer command | Validate an `AnonymizerConfig` (and optional `model_configs`).|
| `nemo anonymizer preview submit` | Generated from `NemoFunction` | Local streaming preview. |
| `nemo anonymizer preview submit` | Generated from `NemoFunction` | Remote streaming preview against the plugin service. |
| `nemo anonymizer run submit` | Generated from `NemoJob` | Local job execution in the CLI process. |
| `nemo anonymizer run submit` | Generated from `NemoJob` | Submit an `anonymizer.run` job to the NeMo Platform Jobs worker. |
| `nemo anonymizer run explain` | Generated from `NemoJob` | Print the job key, submit endpoint, and JSON schemas. |

Expand All @@ -39,13 +37,9 @@ The command does not accept `data.source`. Input-source validation happens durin

## `nemo anonymizer preview`

Both `preview submit` and `preview submit` take a spec file matching `PreviewRequest`.
`preview submit` takes a spec file matching `PreviewRequest`.

```bash
nemo anonymizer preview submit \
--spec-file /tmp/anonymizer-preview.yaml \
--workspace "${NMP_WORKSPACE:-default}"

nemo anonymizer preview submit \
--spec-file /tmp/anonymizer-preview.yaml \
--workspace "${NMP_WORKSPACE:-default}" \
Expand All @@ -60,11 +54,11 @@ nemo anonymizer preview submit \

### Preview source kinds

| Form | `preview submit` | `preview submit` |
|---------------------------------------|---------------|------------------|
| Local path (`/tmp/input.csv`) | yes | no |
| HTTP(S) URL (`https://.../input.csv`) | yes | yes |
| Fileset reference (`fs#path`) | yes | yes |
| Form | `preview submit` |
|---------------------------------------|------------------|
| Local path (`/tmp/input.csv`) | no |
| HTTP(S) URL (`https://.../input.csv`) | yes |
| Fileset reference (`fs#path`) | yes |

### Preview output

Expand Down Expand Up @@ -152,7 +146,7 @@ Both preview and run specs use the shared `AnonymizerRequest` shape:
| `data.text_column`| string | no | Defaults to `text`. |
| `data.id_column` | string | no | Optional record identifier column. |
| `data.data_summary` | string | no | Optional short description of the data. |
| `model_configs` | list of Data Designer `ModelConfig` | depends | Required for `preview submit` and `run submit`; optional for `preview submit` and `run submit`. |
| `model_configs` | list of Data Designer `ModelConfig` | depends | Required for `preview submit` and `run submit`. |
| `selected_models` | object with `detection` / `replace` / `rewrite` | no | Role overrides on top of bundled defaults. Requires `model_configs`. |

Preview-only:
Expand Down
31 changes: 4 additions & 27 deletions docs/data-designer/cli.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ description: ""
---
<a id="data-designer-cli"></a>

The NeMo Data Designer plugin adds the `nemo data-designer` command group. Use it to execute Data Designer workloads locally in the CLI process or submit them to NeMo Services.
The NeMo Data Designer plugin adds the `nemo data-designer` command group. Use it to submit preview and create workloads to NeMo Services.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably makes more sense something like...

Use it to generate high-quality synthetic data on NeMo Platform.

(Side note, I see "NeMo Services" already exists... is that a legit proper noun?)


## Configuration Sources

Expand All @@ -16,6 +16,7 @@ The `preview` and `create` commands accept a configuration source path. The most
```python
import data_designer.config as dd


def load_config_builder() -> dd.DataDesignerConfigBuilder:
model_configs = [
dd.ModelConfig(
Expand All @@ -30,41 +31,17 @@ def load_config_builder() -> dd.DataDesignerConfigBuilder:
return config_builder
```

The same configuration source can usually be used with `run` or `submit`. Resource choices determine whether it is compatible with NeMo Services execution; see [Execution Modes](/documentation/design-synthetic-data/execution-modes).

## Run Versus Submit
The same configuration source is used with `submit`. Resource choices determine whether it is compatible with NeMo Services execution; see [Execution Modes](/documentation/design-synthetic-data/execution-modes).

`run` executes the Data Designer workload locally, in the CLI process. This can be fully local, but it is not an offline-only mode. A local run can still use the Files API, Secrets API, and Inference Gateway API from a running NeMo Services cluster when the configuration references the corresponding resources.
## Submit Versus Platform Execution
Comment on lines +34 to +36

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These don't make sense semantically


`submit` sends the workload to NeMo Services. The Data Designer API and Jobs API coordinate execution, job lifecycle, logs, and artifact persistence. The NeMo Services deployment may itself be local or remote.

| Command | Workload execution | NeMo Services required? |
|---------|--------------------|-------------------------|
| `preview submit` | Local CLI process | Optional |
| `create submit` | Local CLI process | Optional |
Comment on lines -43 to -44

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lol at this table as it exists on the target branch, just totally borked. But also this entire table is pointless now, since the platform services (again that "NeMo Services" noun!) are always required.

| `preview submit` | Data Designer API | Yes |
| `create submit` | Jobs worker | Yes |

## Preview Locally

Use local preview for fast iteration:

```bash
nemo data-designer preview submit product_reviews.py --num-records 5
```

The workload runs in your current Python environment. It can use local-only resources, NeMo resources, or both.

## Create Locally

Use local create when you want to generate a larger dataset without submitting work to NeMo Services:

```bash
nemo data-designer create submit product_reviews.py --num-records 1000
```

This executes the plugin job locally. It is useful for development and for workloads that should stay in the local environment.

## Submit Preview to NeMo Services

Submit preview when you want to exercise the Data Designer API path:
Expand Down
10 changes: 2 additions & 8 deletions docs/evaluator/metrics/llm-as-a-judge.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,11 @@ LLM-as-a-Judge evaluation sends each dataset row to a judge LLM and parses the j
- **Pre-generated data**: Score existing question-answer pairs or conversations.
- **Custom criteria**: Define range scores, rubric scores, prompt templates, and parser behavior.

NeMo Evaluator supports two execution modes through the Evaluator plugin SDK:
Use the Evaluator plugin SDK to submit durable platform jobs:

| Mode | Use Case | SDK Call |
|------|----------|----------|
| **Durable remote job** | Production workloads that should run as platform jobs | `evaluator.submit(metric=metric, dataset=dataset)` |
| **Platform job** | Durable platform execution with polling and result retrieval | `evaluator.submit(metric=metric, dataset=dataset)` |

## Prerequisites

Expand Down Expand Up @@ -127,8 +127,6 @@ result = LocalEvaluator().run_sync(
},
],
)
job.wait_until_done()
result = job.get_result()

for score in result.aggregate_scores.scores:
print(f"{score.name}: mean={score.mean:.2f}, count={score.count}")
Expand Down Expand Up @@ -234,8 +232,6 @@ result = LocalEvaluator().run_sync(
],
aggregate_fields=("rubric_distribution", "mode_category"),
)
job.wait_until_done()
result = job.get_result()

print(result.aggregate_scores.model_dump(exclude_none=True))
```
Expand All @@ -255,8 +251,6 @@ result = LocalEvaluator().run_sync(
],
aggregate_fields=("std_dev", "variance"),
)
job.wait_until_done()
result = job.get_result()

for score in result.aggregate_scores.scores:
print(f"{score.name}:")
Expand Down
2 changes: 1 addition & 1 deletion docs/evaluator/metrics/results.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ An `EvaluationResult` contains:
- **Aggregate scores**: `result.aggregate_scores.scores`, with statistics such as mean, min, max, count, NaN count, variance, standard deviation, percentiles, and rubric distributions when applicable.
- **Row scores**: `result.row_scores`, with the original dataset row, generated sample payload, metric scores, request logs, and row-level errors.

## Get Results from a Local Run
## Get Results Inline

```python
from nemo_evaluator_sdk import Evaluator as LocalEvaluator
Expand Down
2 changes: 1 addition & 1 deletion docs/fern/gated-nav.yml
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@
path: ../../safe-synthesizer/about/data-synthesis.mdx
- page: Evaluation
path: ../../safe-synthesizer/about/evaluation.mdx
- page: Host-Local Development
- page: Runtime Environment
path: ../../safe-synthesizer/about/host-local-development.mdx
- page: Overview
path: ../../safe-synthesizer/about/index.mdx
Expand Down
2 changes: 1 addition & 1 deletion docs/fern/versions/latest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -308,7 +308,7 @@ navigation:
path: ../../safe-synthesizer/about/evaluation.mdx
- page: Jobs
path: ../../safe-synthesizer/about/jobs.mdx
- page: Local and Subprocess Execution
- page: Runtime Environment
path: ../../safe-synthesizer/about/host-local-development.mdx
- page: Parameters Reference
path: ../../safe-synthesizer/about/reference.mdx
Expand Down
2 changes: 1 addition & 1 deletion docs/safe-synthesizer/about/host-local-development.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ description: ""

<a id="host-local-development"></a>

NeMo Safe Synthesizer no longer exposes a host-local workload command. Create synthesis jobs through the platform Jobs API or SDK so scheduling, storage, logs, and artifacts all flow through the supported job path.
Create Safe Synthesizer workloads through the platform Jobs API or SDK so scheduling, storage, logs, and artifacts all flow through the supported job path.

The `nemo safe-synthesizer` CLI still includes runtime utilities for inspecting or preparing the task runtime used by development and job debugging.

Expand Down
2 changes: 1 addition & 1 deletion docs/safe-synthesizer/getting-started.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Before using NeMo Safe Synthesizer, complete [Setup](/documentation/get-started)

NeMo Safe Synthesizer has the following additional requirements:

- A GPU-capable Jobs backend with enough VRAM for the selected model and dataset.
- A GPU-capable Jobs backend with **80GB+ VRAM** recommended for the selected model and dataset (check with `nvidia-smi` on GPU nodes).
- Sufficient disk space for generated datasets (50GB+ recommended)

For general platform troubleshooting (port conflicts, health checks, and so on), refer to [Setup](/documentation/get-started).
Expand Down
Loading
Loading