Skip to content
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/user-guide/docker.md
Original file line number Diff line number Diff line change
Expand Up @@ -244,7 +244,7 @@ docker run --gpus all --shm-size=1g \
nss-gpu:latest run --config /workspace/data/config.yaml --data-source /workspace/data/input.csv
```

See [Environment Variables -- Hugging Face Cache](environment.md#hugging-face-cache)
See [Environment Variables -- Hugging Face cache and offline](environment.md#hugging-face-cache-and-offline)
for details on `HF_HOME`, `HF_HUB_OFFLINE`, and `VLLM_CACHE_ROOT`.

---
Expand Down
359 changes: 186 additions & 173 deletions docs/user-guide/environment.md

Large diffs are not rendered by default.

12 changes: 7 additions & 5 deletions docs/user-guide/running.md
Original file line number Diff line number Diff line change
Expand Up @@ -274,7 +274,7 @@ execute in order (`config` → `dataframe` → `metadata` → `advisory`).
| Check name | Stage | What it validates |
|-------|-------|-------------------|
| `gpu.cuda` | config | PyTorch is importable and a CUDA GPU is visible |
| `env.inference_key` | config | `NSS_INFERENCE_KEY` is set when PII classification is enabled (warning only) |
| `env.inference` | config | Inference config for PII classification: `NSS_INFERENCE_KEY` is set, `NSS_INFERENCE_MODEL` is non-empty, and `NSS_INFERENCE_ENDPOINT` is a valid http(s) URL (warnings only) |
| `env.hf_model_availability` | config | The pretrained model reference is usable locally or can be fetched from Hugging Face; warns about a missing HF token only when online HF access may be needed |
| `dataset.size` | dataframe | Training split meets the hard minimum row count |
| `columns.groupby` | dataframe | `group_training_examples_by` column is present and has no nulls |
Expand Down Expand Up @@ -1230,10 +1230,12 @@ See [`artifacts clean`](#artifacts-clean) in the CLI Commands section for option
## Running in Offline Environments

Pre-cache models by running once with internet access, then set
`HF_HUB_OFFLINE=1` in your target environment. For detailed cache setup
and environment variables (`HF_HOME`, `HF_HUB_OFFLINE`, `LOCAL_FILES_ONLY`,
`VLLM_CACHE_ROOT`), see
[Environment Variables -- Hugging Face Cache](environment.md#hugging-face-cache).
`HF_HUB_OFFLINE=1` in your target environment. Export it before launching
`safe-synthesizer` (or pass `--disable-huggingface-remote`) -- huggingface_hub
reads the value once at import time, so setting it after the process starts has
no effect. For detailed cache setup and environment variables (`HF_HOME`,
`HF_HUB_OFFLINE`, `VLLM_CACHE_ROOT`), see
[Environment Variables -- Hugging Face cache and offline](environment.md#hugging-face-cache-and-offline).

For offline-specific errors, see [Program Runtime](troubleshooting.md).

Expand Down
28 changes: 26 additions & 2 deletions docs/user-guide/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -513,7 +513,9 @@ check of its own.
| `torch_missing` | error | `gpu.cuda` | PyTorch not installed; cannot verify GPU availability |
| `no_gpu` | error | `gpu.cuda` | No CUDA GPU detected (required for training or generation) |
| `low_vram` | warning | `gpu.vram` | Free GPU VRAM may be insufficient |
| `inference_key_missing` | warning | `env.inference_key` | `NSS_INFERENCE_KEY` not set; PII classification degraded |
| `inference_key_missing` | warning | `env.inference` | `NSS_INFERENCE_KEY` not set; PII classification degraded |
| `inference_model_blank` | warning | `env.inference` | `NSS_INFERENCE_MODEL` set but empty; the blank value is ignored and the default model id is used |
| `inference_endpoint_invalid` | error | `env.inference` | `NSS_INFERENCE_ENDPOINT` set but not a valid http(s) URL; classification requests will fail |
| `hf_token_missing` | warning | `env.hf_model_availability` | Neither `HF_TOKEN` nor `HUGGING_FACE_HUB_TOKEN` set, and model loading may need online Hugging Face access |
| `hf_model_not_cached` | warning/error | `env.hf_model_availability` | Hugging Face model is not present in the local cache; severity is error when HF offline mode is enabled |
| `hf_model_cache_incomplete` | warning/error | `env.hf_model_availability` | Cached Hugging Face model snapshot is missing required config, tokenizer, weights, or shards; severity is error when HF offline mode is enabled |
Expand Down Expand Up @@ -545,7 +547,29 @@ The PII replacer downloads the GLiNER NER model on first use. If the download
fails, it raises an exception immediately.

Pre-download the model by running PII replacement once in an environment
with internet access, or set `LOCAL_FILES_ONLY=true` after the model is cached.
with internet access. To force offline use after the model is cached, set
`HF_HUB_OFFLINE=1` or pass `--disable-huggingface-remote`.

### Offline Mode Not Taking Effect

Symptom: `HF_HUB_OFFLINE=1` (or `--disable-huggingface-remote`) is set, yet the
run still attempts a download, or `--enable-huggingface-remote` does not
re-enable downloads.

Cause: huggingface_hub reads `HF_HUB_OFFLINE` once, at import time, and caches
it. If the variable is changed after huggingface_hub has been imported in the
process, the change is ignored.

Fixes:

- CLI: export `HF_HUB_OFFLINE` before launching `safe-synthesizer`, or use
`--enable-huggingface-remote` / `--disable-huggingface-remote`. The CLI
applies the flag before huggingface_hub loads, so the flag always wins over
an inherited environment value.
- Programmatic / SDK: set `HF_HUB_OFFLINE` before importing
`nemo_safe_synthesizer` (or any library that imports huggingface_hub, such as
`transformers` or `datasets`). Setting it afterward has no effect for that
process.

### NER Processing Timeouts

Expand Down
2 changes: 1 addition & 1 deletion script/slurm/slurm_nss_matrix.sh
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ echo "[NSS SLURM] nemo-safe-synthesizer version: $(python -c 'from nemo_safe_syn

# for column classification
export NSS_INFERENCE_ENDPOINT=https://integrate.api.nvidia.com/v1
export NIM_MODEL_ID=qwen/qwen3-next-80b-a3b-instruct
export NSS_INFERENCE_MODEL=qwen/qwen3-next-80b-a3b-instruct

# Extract dataset name for path construction (handles both full paths and simple names)
# e.g., "/path/to/adult.csv" -> "adult", "/path/to/data.parquet" -> "data", "adult" -> "adult"
Expand Down
169 changes: 85 additions & 84 deletions src/nemo_safe_synthesizer/cli/run.py
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,60 @@ def common_run_options(f: Callable[..., object]) -> Callable[..., object]:
"If both env var and CLI option are provided, the CLI option takes precedence.",
)
)
options.append(
click.option(
"--inference-endpoint-url",
type=str,
required=False,
default=None,
help="OpenAI-compatible inference endpoint URL for PII column classification. "
"Can also be set via NSS_INFERENCE_ENDPOINT env var.",
)
)
options.append(
click.option(
"--inference-api-key",
type=str,
required=False,
default=None,
help="API key for the inference endpoint used in PII column classification. "
"Can also be set via NSS_INFERENCE_KEY env var.",
)
)
Comment on lines +172 to +181

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 security API key exposed in process listings and shell history

--inference-api-key is declared as a plain type=str option. Any value passed on the command line is visible in /proc/PID/cmdline, ps aux, shell history files, and CI logs that echo commands — all readable by other users on the same host. The canonical env-var path (NSS_INFERENCE_KEY) avoids this; the help text documents it, but does not warn that the flag itself is the insecure route. Passing --inference-api-key my-secret on a shared Slurm node or in a verbose CI step will leak the key to any co-tenant process with /proc access.

options.append(
click.option(
"--inference-model-id",
type=str,
required=False,
default=None,
help="Model ID sent to the inference endpoint for PII column classification. "
"Can also be set via NSS_INFERENCE_MODEL env var. "
"[default: qwen/qwen3-next-80b-a3b-instruct]",
)
)
options.append(
click.option(
"--enable-huggingface-remote/--disable-huggingface-remote",
"huggingface_remote",
required=False,
default=None,
Comment thread
binaryaaron marked this conversation as resolved.
help="Allow or block Hugging Face remote downloads for both the base model "
"and GLiNER. --disable-huggingface-remote forces a fully offline run by "
"setting HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE; both must already be "
"cached. Equivalent to setting HF_HUB_OFFLINE in the environment.",
)
Comment thread
binaryaaron marked this conversation as resolved.
)
options.append(
click.option(
"--cpu-count",
type=int,
required=False,
default=None,
help="Number of CPU worker processes used for NER (PII replacement). "
"Can also be set via NSS_PII_REPLACER_CPU_COUNT env var. "
"[default: max(1, cpu_count - 1)]",
)
)
# Apply each option decorator in reverse order (decorators apply bottom-up)
for option in reversed(options):
f = option(f)
Expand All @@ -170,6 +224,31 @@ def _parse_run_overrides(kwargs: dict[str, Any]) -> dict[str, Any]:
return parse_overrides(kwargs)


# CLISettings fields populated from common_run_options flags. ``synthesis_overrides``
# is excluded -- it is derived from the leftover pydantic_options kwargs, not bound
# to a single flag. ``observability``/``wandb`` are nested sub-settings with no CLI
# flag, so they never appear in command kwargs.
_CLI_SETTINGS_FIELDS: frozenset[str] = frozenset(CLISettings.model_fields) - {"synthesis_overrides"}


def _settings_from_run_kwargs(kwargs: dict[str, Any]) -> CLISettings:
"""Build ``CLISettings`` from a run command's ``**kwargs``.

``common_run_options`` binds each infrastructure flag to a kwarg whose name
matches a ``CLISettings`` field; those are pulled out here. Everything left
(the ``pydantic_options`` ``--section__field`` options) becomes synthesis
overrides. This keeps the three run commands from re-listing the shared flag
set in both their signature and their settings construction -- adding a flag
now means editing ``common_run_options`` and ``CLISettings`` only.

``kwargs`` is mutated: matched settings keys are popped before the remainder
is parsed into overrides.
"""
settings_kwargs = {name: kwargs.pop(name) for name in _CLI_SETTINGS_FIELDS if name in kwargs}
settings_kwargs["synthesis_overrides"] = _parse_run_overrides(kwargs)
return CLISettings.from_cli_kwargs(**settings_kwargs)


def _set_cli_deployment_type_default() -> None:
"""Default telemetry deployment type for CLI commands without overriding Slurm or explicit settings."""
os.environ.setdefault("NEMO_DEPLOYMENT_TYPE", DeploymentTypeEnum.CLI.value)
Expand Down Expand Up @@ -311,20 +390,8 @@ def _build_validate_render_context(
)
def run(
ctx: click.Context,
config_path: PathT | None,
data_source: str | None,
artifact_path: PathT | None,
run_path: PathT | None,
output_file: PathT | None,
log_file: PathT | None,
log_color: bool | None,
log_format: str | None,
verbose: int = 0,
wandb_mode: str | None = None,
wandb_project: str | None = None,
dataset_registry: str | None = None,
validate: bool = False,
**kwargs: object,
**kwargs: Any,
) -> None:
"""Run the Safe Synthesizer end-to-end pipeline.

Expand All @@ -337,21 +404,7 @@ def run(

_set_cli_deployment_type_default()

settings = CLISettings.from_cli_kwargs(
data_source=data_source,
config_path=config_path,
artifact_path=artifact_path,
run_path=run_path,
output_file=output_file,
log_file=log_file,
log_color=log_color,
log_format=log_format,
verbose=verbose,
wandb_mode=wandb_mode,
wandb_project=wandb_project,
synthesis_overrides=_parse_run_overrides(kwargs),
dataset_registry=dataset_registry,
)
settings = _settings_from_run_kwargs(kwargs)

if validate:
os.environ["NSS_PHASE"] = "process_data"
Expand Down Expand Up @@ -407,20 +460,8 @@ def run(
help="Run pre-flight validation only, then exit without training or generating.",
)
def run_train(
config_path: PathT,
data_source: str | None,
artifact_path: PathT | None,
run_path: PathT | None,
output_file: PathT | None,
log_format: str | None,
log_color: bool | None,
log_file: PathT | None,
verbose: int,
wandb_mode: str | None = None,
wandb_project: str | None = None,
dataset_registry: str | None = None,
validate: bool = False,
**kwargs: object,
**kwargs: Any,
Comment thread
binaryaaron marked this conversation as resolved.
) -> None:
"""Run the training stage only.

Expand All @@ -429,21 +470,7 @@ def run_train(
"""
_set_cli_deployment_type_default()

settings = CLISettings.from_cli_kwargs(
data_source=data_source,
config_path=config_path,
artifact_path=artifact_path,
run_path=run_path,
output_file=output_file,
log_file=log_file,
log_color=log_color,
log_format=log_format,
verbose=verbose,
wandb_mode=wandb_mode,
wandb_project=wandb_project,
synthesis_overrides=_parse_run_overrides(kwargs),
dataset_registry=dataset_registry,
)
settings = _settings_from_run_kwargs(kwargs)

if validate:
os.environ["NSS_PHASE"] = "process_data"
Expand Down Expand Up @@ -498,21 +525,9 @@ def run_train(
)
@pydantic_options(SafeSynthesizerParameters, field_separator=CLI_NESTED_FIELD_SEPARATOR)
def run_generate(
config_path: PathT,
data_source: str | None,
run_path: PathT | None,
artifact_path: PathT | None,
output_file: PathT | None,
log_format: str | None,
log_color: bool | None,
log_file: PathT | None,
verbose: int,
wandb_mode: str | None = None,
wandb_project: str | None = None,
auto_discover_adapter: bool = False,
wandb_resume_job_id: str | None = None,
dataset_registry: str | None = None,
**kwargs: object,
**kwargs: Any,
) -> None:
"""Run the generation stage only.

Expand All @@ -526,21 +541,7 @@ def run_generate(
_set_cli_deployment_type_default()

# Create unified settings from CLI kwargs
settings = CLISettings.from_cli_kwargs(
data_source=data_source,
config_path=config_path,
artifact_path=artifact_path,
run_path=run_path,
output_file=output_file,
log_file=log_file,
log_color=log_color,
log_format=log_format,
verbose=verbose,
wandb_mode=wandb_mode,
wandb_project=wandb_project,
synthesis_overrides=_parse_run_overrides(kwargs),
dataset_registry=dataset_registry,
)
settings = _settings_from_run_kwargs(kwargs)

os.environ["NSS_PHASE"] = "generate"
# Generation always resumes from an existing workdir with a trained model
Expand Down
48 changes: 47 additions & 1 deletion src/nemo_safe_synthesizer/cli/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -118,9 +118,10 @@ class CLISettings(BaseSettings):

log_color: bool | None = Field(
default=None,
validation_alias=AliasChoices("log_color", "NSS_LOG_COLOR"),
description="Whether to colorize console output",
)
"""Whether to colorize console output."""
"""Whether to colorize console output (env variable: ``NSS_LOG_COLOR``)."""

log_file: str | None = Field(
default=None,
Expand Down Expand Up @@ -164,6 +165,51 @@ class CLISettings(BaseSettings):
)
"""URL or path to a dataset registry YAML file (env: ``NSS_DATASET_REGISTRY``)."""

inference_endpoint_url: str | None = Field(
default=None,
validation_alias=AliasChoices("inference_endpoint_url", "NSS_INFERENCE_ENDPOINT"),
description="OpenAI-compatible inference endpoint URL for PII column classification",
)
"""OpenAI-compatible inference endpoint URL for PII column classification
(env: ``NSS_INFERENCE_ENDPOINT``)."""

inference_api_key: str | None = Field(
default=None,
validation_alias=AliasChoices("inference_api_key", "NSS_INFERENCE_KEY"),
description="API key for the inference endpoint used in PII column classification",
)
"""API key for the inference endpoint used in PII column classification
(env: ``NSS_INFERENCE_KEY``)."""

inference_model_id: str | None = Field(
default=None,
validation_alias=AliasChoices("inference_model_id", "NSS_INFERENCE_MODEL"),
description="Model ID sent to the inference endpoint for PII column classification",
)
"""Model ID sent to the inference endpoint for PII column classification
(env: ``NSS_INFERENCE_MODEL``)."""

huggingface_remote: bool | None = Field(
default=None,
validation_alias=AliasChoices("huggingface_remote"),
description="Whether to allow Hugging Face remote downloads (base model and GLiNER)",
)
"""Whether to allow Hugging Face remote downloads for the base model and GLiNER.

``None`` leaves the environment untouched. ``True`` / ``False`` is propagated
to the standard ``HF_HUB_OFFLINE`` and ``TRANSFORMERS_OFFLINE`` variables (the
canonical env switch) by ``_propagate_runtime_settings_to_env``; there is no
separate NSS env var."""

cpu_count: int | None = Field(
default=None,
ge=1,
validation_alias=AliasChoices("cpu_count", "NSS_PII_REPLACER_CPU_COUNT"),
description="Number of CPU worker processes used for NER (PII replacement)",
)
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Comment thread
binaryaaron marked this conversation as resolved.
"""Number of CPU worker processes used for NER (PII replacement)
(env: ``NSS_PII_REPLACER_CPU_COUNT``)."""

@field_validator("wandb_mode", mode="before")
@classmethod
def validate_wandb_mode(cls, v: str | WandbMode | None) -> WandbMode | None:
Expand Down
Loading
Loading