Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/user-guide/docker.md
Original file line number Diff line number Diff line change
Expand Up @@ -244,7 +244,7 @@ docker run --gpus all --shm-size=1g \
nss-gpu:latest run --config /workspace/data/config.yaml --data-source /workspace/data/input.csv
```

See [Environment Variables -- Hugging Face Cache](environment.md#hugging-face-cache)
See [Environment Variables -- Hugging Face cache and offline](environment.md#hugging-face-cache-and-offline)
for details on `HF_HOME`, `HF_HUB_OFFLINE`, and `VLLM_CACHE_ROOT`.

---
Expand Down
362 changes: 189 additions & 173 deletions docs/user-guide/environment.md

Large diffs are not rendered by default.

12 changes: 7 additions & 5 deletions docs/user-guide/running.md
Original file line number Diff line number Diff line change
Expand Up @@ -274,7 +274,7 @@ execute in order (`config` → `dataframe` → `metadata` → `advisory`).
| Check name | Stage | What it validates |
|-------|-------|-------------------|
| `gpu.cuda` | config | PyTorch is importable and a CUDA GPU is visible |
| `env.inference_key` | config | `NSS_INFERENCE_KEY` is set when PII classification is enabled (warning only) |
| `env.inference` | config | Inference config for PII classification: `NSS_INFERENCE_KEY` is set, `NSS_INFERENCE_MODEL` is non-empty, and `NSS_INFERENCE_ENDPOINT` is a valid http(s) URL (warnings only) |
| `env.hf_model_availability` | config | The pretrained model reference is usable locally or can be fetched from Hugging Face; warns about a missing HF token only when online HF access may be needed |
| `dataset.size` | dataframe | Training split meets the hard minimum row count |
| `columns.groupby` | dataframe | `group_training_examples_by` column is present and has no nulls |
Expand Down Expand Up @@ -1230,10 +1230,12 @@ See [`artifacts clean`](#artifacts-clean) in the CLI Commands section for option
## Running in Offline Environments

Pre-cache models by running once with internet access, then set
`HF_HUB_OFFLINE=1` in your target environment. For detailed cache setup
and environment variables (`HF_HOME`, `HF_HUB_OFFLINE`, `LOCAL_FILES_ONLY`,
`VLLM_CACHE_ROOT`), see
[Environment Variables -- Hugging Face Cache](environment.md#hugging-face-cache).
`HF_HUB_OFFLINE=1` in your target environment. Export it before launching
`safe-synthesizer` (or pass `--disable-huggingface-remote`) -- huggingface_hub
reads the value once at import time, so setting it after the process starts has
no effect. For detailed cache setup and environment variables (`HF_HOME`,
`HF_HUB_OFFLINE`, `VLLM_CACHE_ROOT`), see
[Environment Variables -- Hugging Face cache and offline](environment.md#hugging-face-cache-and-offline).

For offline-specific errors, see [Program Runtime](troubleshooting.md).

Expand Down
28 changes: 26 additions & 2 deletions docs/user-guide/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -513,7 +513,9 @@ check of its own.
| `torch_missing` | error | `gpu.cuda` | PyTorch not installed; cannot verify GPU availability |
| `no_gpu` | error | `gpu.cuda` | No CUDA GPU detected (required for training or generation) |
| `low_vram` | warning | `gpu.vram` | Free GPU VRAM may be insufficient |
| `inference_key_missing` | warning | `env.inference_key` | `NSS_INFERENCE_KEY` not set; PII classification degraded |
| `inference_key_missing` | warning | `env.inference` | `NSS_INFERENCE_KEY` not set; PII classification degraded |
| `inference_model_blank` | warning | `env.inference` | `NSS_INFERENCE_MODEL` set but empty; the blank value is ignored and the default model id is used |
| `inference_endpoint_invalid` | error | `env.inference` | `NSS_INFERENCE_ENDPOINT` set but not a valid http(s) URL; classification requests will fail |
| `hf_token_missing` | warning | `env.hf_model_availability` | Neither `HF_TOKEN` nor `HUGGING_FACE_HUB_TOKEN` set, and model loading may need online Hugging Face access |
| `hf_model_not_cached` | warning/error | `env.hf_model_availability` | Hugging Face model is not present in the local cache; severity is error when HF offline mode is enabled |
| `hf_model_cache_incomplete` | warning/error | `env.hf_model_availability` | Cached Hugging Face model snapshot is missing required config, tokenizer, weights, or shards; severity is error when HF offline mode is enabled |
Expand Down Expand Up @@ -545,7 +547,29 @@ The PII replacer downloads the GLiNER NER model on first use. If the download
fails, it raises an exception immediately.

Pre-download the model by running PII replacement once in an environment
with internet access, or set `LOCAL_FILES_ONLY=true` after the model is cached.
with internet access. To force offline use after the model is cached, set
`HF_HUB_OFFLINE=1` or pass `--disable-huggingface-remote`.

### Offline Mode Not Taking Effect

Symptom: `HF_HUB_OFFLINE=1` (or `--disable-huggingface-remote`) is set, yet the
run still attempts a download, or `--enable-huggingface-remote` does not
re-enable downloads.

Cause: huggingface_hub reads `HF_HUB_OFFLINE` once, at import time, and caches
it. If the variable is changed after huggingface_hub has been imported in the
process, the change is ignored.

Fixes:

- CLI: export `HF_HUB_OFFLINE` before launching `safe-synthesizer`, or use
`--enable-huggingface-remote` / `--disable-huggingface-remote`. The CLI
applies the flag before huggingface_hub loads, so the flag always wins over
an inherited environment value.
- Programmatic / SDK: set `HF_HUB_OFFLINE` before importing
`nemo_safe_synthesizer` (or any library that imports huggingface_hub, such as
`transformers` or `datasets`). Setting it afterward has no effect for that
process.

### NER Processing Timeouts

Expand Down
2 changes: 1 addition & 1 deletion script/slurm/slurm_nss_matrix.sh
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ echo "[NSS SLURM] nemo-safe-synthesizer version: $(python -c 'from nemo_safe_syn

# for column classification
export NSS_INFERENCE_ENDPOINT=https://integrate.api.nvidia.com/v1
export NIM_MODEL_ID=qwen/qwen3-next-80b-a3b-instruct
export NSS_INFERENCE_MODEL=qwen/qwen3-next-80b-a3b-instruct

# Extract dataset name for path construction (handles both full paths and simple names)
# e.g., "/path/to/adult.csv" -> "adult", "/path/to/data.parquet" -> "data", "adult" -> "adult"
Expand Down
172 changes: 88 additions & 84 deletions src/nemo_safe_synthesizer/cli/run.py
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,63 @@ def common_run_options(f: Callable[..., object]) -> Callable[..., object]:
"If both env var and CLI option are provided, the CLI option takes precedence.",
)
)
options.append(
click.option(
"--inference-endpoint-url",
type=str,
required=False,
default=None,
help="OpenAI-compatible inference endpoint URL for PII column classification. "
"Can also be set via NSS_INFERENCE_ENDPOINT env var.",
)
)
options.append(
click.option(
"--inference-api-key",
type=str,
required=False,
default=None,
help="API key for the inference endpoint used in PII column classification. "
"Can also be set via NSS_INFERENCE_KEY env var.",
)
)
Comment on lines +172 to +181

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 security API key exposed in process listings and shell history

--inference-api-key is declared as a plain type=str option. Any value passed on the command line is visible in /proc/PID/cmdline, ps aux, shell history files, and CI logs that echo commands — all readable by other users on the same host. The canonical env-var path (NSS_INFERENCE_KEY) avoids this; the help text documents it, but does not warn that the flag itself is the insecure route. Passing --inference-api-key my-secret on a shared Slurm node or in a verbose CI step will leak the key to any co-tenant process with /proc access.

options.append(
click.option(
"--inference-model-id",
type=str,
required=False,
default=None,
help="Model ID sent to the inference endpoint for PII column classification. "
"Can also be set via NSS_INFERENCE_MODEL env var. "
"[default: qwen/qwen3-next-80b-a3b-instruct]",
)
)
options.append(
click.option(
"--enable-huggingface-remote/--disable-huggingface-remote",
"huggingface_remote",
required=False,
default=None,
Comment thread
binaryaaron marked this conversation as resolved.
help="Allow or block Hugging Face remote downloads for both the base model "
"and GLiNER. --disable-huggingface-remote forces a fully offline run by "
"setting HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE; both must already be "
"cached. Equivalent to setting HF_HUB_OFFLINE in the environment. When "
"neither flag is given, the run inherits HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE "
"from the environment (remote downloads enabled when unset). "
"[default: --enable-huggingface-remote]",
)
Comment thread
binaryaaron marked this conversation as resolved.
)
options.append(
click.option(
"--cpu-count",
type=int,
required=False,
default=None,
help="Number of CPU worker processes used for NER (PII replacement). "
"Can also be set via NSS_PII_REPLACER_CPU_COUNT env var. "
"[default: max(1, cpu_count - 1)]",
)
)
# Apply each option decorator in reverse order (decorators apply bottom-up)
for option in reversed(options):
f = option(f)
Expand All @@ -170,6 +227,31 @@ def _parse_run_overrides(kwargs: dict[str, Any]) -> dict[str, Any]:
return parse_overrides(kwargs)


# CLISettings fields populated from common_run_options flags. ``synthesis_overrides``
# is excluded -- it is derived from the leftover pydantic_options kwargs, not bound
# to a single flag. ``observability``/``wandb`` are nested sub-settings with no CLI
# flag, so they never appear in command kwargs.
_CLI_SETTINGS_FIELDS: frozenset[str] = frozenset(CLISettings.model_fields) - {"synthesis_overrides"}


def _settings_from_run_kwargs(kwargs: dict[str, Any]) -> CLISettings:
"""Build ``CLISettings`` from a run command's ``**kwargs``.

``common_run_options`` binds each infrastructure flag to a kwarg whose name
matches a ``CLISettings`` field; those are pulled out here. Everything left
(the ``pydantic_options`` ``--section__field`` options) becomes synthesis
overrides. This keeps the three run commands from re-listing the shared flag
set in both their signature and their settings construction -- adding a flag
now means editing ``common_run_options`` and ``CLISettings`` only.

``kwargs`` is mutated: matched settings keys are popped before the remainder
is parsed into overrides.
"""
settings_kwargs = {name: kwargs.pop(name) for name in _CLI_SETTINGS_FIELDS if name in kwargs}
settings_kwargs["synthesis_overrides"] = _parse_run_overrides(kwargs)
return CLISettings.from_cli_kwargs(**settings_kwargs)


def _set_cli_deployment_type_default() -> None:
"""Default telemetry deployment type for CLI commands without overriding Slurm or explicit settings."""
os.environ.setdefault("NEMO_DEPLOYMENT_TYPE", DeploymentTypeEnum.CLI.value)
Expand Down Expand Up @@ -311,20 +393,8 @@ def _build_validate_render_context(
)
def run(
ctx: click.Context,
config_path: PathT | None,
data_source: str | None,
artifact_path: PathT | None,
run_path: PathT | None,
output_file: PathT | None,
log_file: PathT | None,
log_color: bool | None,
log_format: str | None,
verbose: int = 0,
wandb_mode: str | None = None,
wandb_project: str | None = None,
dataset_registry: str | None = None,
validate: bool = False,
**kwargs: object,
**kwargs: Any,
) -> None:
"""Run the Safe Synthesizer end-to-end pipeline.

Expand All @@ -337,21 +407,7 @@ def run(

_set_cli_deployment_type_default()

settings = CLISettings.from_cli_kwargs(
data_source=data_source,
config_path=config_path,
artifact_path=artifact_path,
run_path=run_path,
output_file=output_file,
log_file=log_file,
log_color=log_color,
log_format=log_format,
verbose=verbose,
wandb_mode=wandb_mode,
wandb_project=wandb_project,
synthesis_overrides=_parse_run_overrides(kwargs),
dataset_registry=dataset_registry,
)
settings = _settings_from_run_kwargs(kwargs)

if validate:
os.environ["NSS_PHASE"] = "process_data"
Expand Down Expand Up @@ -407,20 +463,8 @@ def run(
help="Run pre-flight validation only, then exit without training or generating.",
)
def run_train(
config_path: PathT,
data_source: str | None,
artifact_path: PathT | None,
run_path: PathT | None,
output_file: PathT | None,
log_format: str | None,
log_color: bool | None,
log_file: PathT | None,
verbose: int,
wandb_mode: str | None = None,
wandb_project: str | None = None,
dataset_registry: str | None = None,
validate: bool = False,
**kwargs: object,
**kwargs: Any,
Comment thread
binaryaaron marked this conversation as resolved.
) -> None:
"""Run the training stage only.

Expand All @@ -429,21 +473,7 @@ def run_train(
"""
_set_cli_deployment_type_default()

settings = CLISettings.from_cli_kwargs(
data_source=data_source,
config_path=config_path,
artifact_path=artifact_path,
run_path=run_path,
output_file=output_file,
log_file=log_file,
log_color=log_color,
log_format=log_format,
verbose=verbose,
wandb_mode=wandb_mode,
wandb_project=wandb_project,
synthesis_overrides=_parse_run_overrides(kwargs),
dataset_registry=dataset_registry,
)
settings = _settings_from_run_kwargs(kwargs)

if validate:
os.environ["NSS_PHASE"] = "process_data"
Expand Down Expand Up @@ -498,21 +528,9 @@ def run_train(
)
@pydantic_options(SafeSynthesizerParameters, field_separator=CLI_NESTED_FIELD_SEPARATOR)
def run_generate(
config_path: PathT,
data_source: str | None,
run_path: PathT | None,
artifact_path: PathT | None,
output_file: PathT | None,
log_format: str | None,
log_color: bool | None,
log_file: PathT | None,
verbose: int,
wandb_mode: str | None = None,
wandb_project: str | None = None,
auto_discover_adapter: bool = False,
wandb_resume_job_id: str | None = None,
dataset_registry: str | None = None,
**kwargs: object,
**kwargs: Any,
) -> None:
"""Run the generation stage only.

Expand All @@ -526,21 +544,7 @@ def run_generate(
_set_cli_deployment_type_default()

# Create unified settings from CLI kwargs
settings = CLISettings.from_cli_kwargs(
data_source=data_source,
config_path=config_path,
artifact_path=artifact_path,
run_path=run_path,
output_file=output_file,
log_file=log_file,
log_color=log_color,
log_format=log_format,
verbose=verbose,
wandb_mode=wandb_mode,
wandb_project=wandb_project,
synthesis_overrides=_parse_run_overrides(kwargs),
dataset_registry=dataset_registry,
)
settings = _settings_from_run_kwargs(kwargs)

os.environ["NSS_PHASE"] = "generate"
# Generation always resumes from an existing workdir with a trained model
Expand Down
48 changes: 47 additions & 1 deletion src/nemo_safe_synthesizer/cli/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -118,9 +118,10 @@ class CLISettings(BaseSettings):

log_color: bool | None = Field(
default=None,
validation_alias=AliasChoices("log_color", "NSS_LOG_COLOR"),
description="Whether to colorize console output",
)
"""Whether to colorize console output."""
"""Whether to colorize console output (env variable: ``NSS_LOG_COLOR``)."""

log_file: str | None = Field(
default=None,
Expand Down Expand Up @@ -164,6 +165,51 @@ class CLISettings(BaseSettings):
)
"""URL or path to a dataset registry YAML file (env: ``NSS_DATASET_REGISTRY``)."""

inference_endpoint_url: str | None = Field(
default=None,
validation_alias=AliasChoices("inference_endpoint_url", "NSS_INFERENCE_ENDPOINT"),
description="OpenAI-compatible inference endpoint URL for PII column classification",
)
"""OpenAI-compatible inference endpoint URL for PII column classification
(env: ``NSS_INFERENCE_ENDPOINT``)."""

inference_api_key: str | None = Field(
default=None,
validation_alias=AliasChoices("inference_api_key", "NSS_INFERENCE_KEY"),
description="API key for the inference endpoint used in PII column classification",
)
"""API key for the inference endpoint used in PII column classification
(env: ``NSS_INFERENCE_KEY``)."""

inference_model_id: str | None = Field(
default=None,
validation_alias=AliasChoices("inference_model_id", "NSS_INFERENCE_MODEL"),
description="Model ID sent to the inference endpoint for PII column classification",
)
"""Model ID sent to the inference endpoint for PII column classification
(env: ``NSS_INFERENCE_MODEL``)."""

huggingface_remote: bool | None = Field(
default=None,
validation_alias=AliasChoices("huggingface_remote"),
description="Whether to allow Hugging Face remote downloads (base model and GLiNER)",
)
"""Whether to allow Hugging Face remote downloads for the base model and GLiNER.

``None`` leaves the environment untouched. ``True`` / ``False`` is propagated
to the standard ``HF_HUB_OFFLINE`` and ``TRANSFORMERS_OFFLINE`` variables (the
canonical env switch) by ``_propagate_runtime_settings_to_env``; there is no
separate NSS env var."""

cpu_count: int | None = Field(
default=None,
ge=1,
validation_alias=AliasChoices("cpu_count", "NSS_PII_REPLACER_CPU_COUNT"),
description="Number of CPU worker processes used for NER (PII replacement)",
)
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Comment thread
binaryaaron marked this conversation as resolved.
"""Number of CPU worker processes used for NER (PII replacement)
(env: ``NSS_PII_REPLACER_CPU_COUNT``)."""

@field_validator("wandb_mode", mode="before")
@classmethod
def validate_wandb_mode(cls, v: str | WandbMode | None) -> WandbMode | None:
Expand Down
Loading
Loading