Skip to content

Commit 8e99a30

Browse files
authored
chore(cli): rename debug to info (#342)
- rename debug cmd to info - include key files into displayed job infomrmation - fix --help to not include export flags --------- Signed-off-by: Meriem B. <[email protected]>
1 parent a1f49ee commit 8e99a30

File tree

6 files changed

+264
-147
lines changed

6 files changed

+264
-147
lines changed

docs/get-started/quickstart/index.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ All paths require:
1717
| List benchmarks | `nemo-evaluator-launcher ls tasks` |
1818
| Run evaluation | `nemo-evaluator-launcher run --config-dir packages/nemo-evaluator-launcher/examples --config-name <config>` |
1919
| Check status | `nemo-evaluator-launcher status <invocation_id>` |
20-
| Debug job | `nemo-evaluator-launcher debug <invocation_id>` |
20+
| Job info | `nemo-evaluator-launcher info <invocation_id>` |
2121
| Export results | `nemo-evaluator-launcher export <invocation_id> --dest local --format json` |
2222
| Dry run | Add `--dry-run` to any run command |
2323
| Test with limited samples | Add `-o +config.params.limit_samples=3` |

docs/libraries/nemo-evaluator-launcher/cli.md

Lines changed: 60 additions & 62 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,8 @@ nemo-evaluator-launcher --version # Show version information
2121
- Run evaluations with specified configuration
2222
* - `status`
2323
- Check status of jobs or invocations
24+
* - `info`
25+
- Show detailed job(s) information
2426
* - `kill`
2527
- Kill a job or invocation
2628
* - `ls`
@@ -29,8 +31,6 @@ nemo-evaluator-launcher --version # Show version information
2931
- Export evaluation results to various destinations
3032
* - `version`
3133
- Show version information
32-
* - `debug`
33-
- Show detailed job(s) information
3434
```
3535

3636
## run - Run Evaluations
@@ -156,6 +156,62 @@ abc12345.1 | success | container124 | <output_dir>/task2/...
156156
]
157157
```
158158

159+
## info - Job information and navigation
160+
161+
Display detailed job information, including metadata, configuration, and paths to logs/artifacts with descriptions of key result files. Supports copying results locally from both local and remote jobs.
162+
163+
### Basic usage
164+
```bash
165+
# Show job info for one or more IDs (job or invocation)
166+
nemo-evaluator-launcher info <job_or_invocation_id>
167+
nemo-evaluator-launcher info <inv1> <inv2>
168+
```
169+
170+
### Show configuration
171+
```bash
172+
nemo-evaluator-launcher info <id> --config
173+
```
174+
175+
### Show paths
176+
```bash
177+
# Show artifact locations
178+
nemo-evaluator-launcher info <id> --artifacts
179+
# Show log locations
180+
nemo-evaluator-launcher info <id> --logs
181+
```
182+
183+
### Copy files locally
184+
```bash
185+
# Copy logs
186+
nemo-evaluator-launcher info <id> --copy-logs [DIR]
187+
188+
# Copy artifacts
189+
nemo-evaluator-launcher info <id> --copy-artifacts [DIR]
190+
```
191+
192+
### Example (Slurm)
193+
```text
194+
nemo-evaluator-launcher info <inv_id>
195+
196+
Job <inv_id>.0
197+
├── Executor: slurm
198+
├── Created: <timestamp>
199+
├── Task: <task_name>
200+
├── Artifacts: user@host:/shared/.../<job_id>/task_name/artifacts (remote)
201+
│ └── Key files:
202+
│ ├── results.yml - Benchmark scores, task results and resolved run configuration.
203+
│ ├── eval_factory_metrics.json - Response + runtime stats (latency, tokens count, memory)
204+
│ ├── metrics.json - Harness/benchmark metric and configuration
205+
│ ├── report.html - Request-Response Pairs samples in HTML format (if enabled)
206+
│ ├── report.json - Report data in json format, if enabled
207+
├── Logs: user@host:/shared/.../<job_id>/task_name/logs (remote)
208+
│ └── Key files:
209+
│ ├── client-{SLURM_JOB_ID}.out - Evaluation container/process output
210+
│ ├── slurm-{SLURM_JOB_ID}.out - SLURM scheduler stdout/stderr (batch submission, export steps).
211+
│ ├── server-{SLURM_JOB_ID}.out - Model server logs when a deployment is used.
212+
├── Slurm Job ID: <SLURM_JOB_ID>
213+
```
214+
159215
## kill - Kill Jobs
160216

161217
Stop running evaluations.
@@ -303,64 +359,6 @@ nemo-evaluator-launcher version
303359
nemo-evaluator-launcher --version
304360
```
305361

306-
## debug - Job Information and Debugging helper functionalities
307-
308-
Display detailed job information including metadata, configuration, and locations of logs and artifacts. The debug command is useful for troubleshooting job issues, inspecting configurations, and retrieving artifacts from both local and remote jobs.
309-
310-
### Basic Usage
311-
312-
```bash
313-
# Show job metadata and information for a single or multiple jobs
314-
nemo-evaluator-launcher debug <invocation_id>
315-
316-
nemo-evaluator-launcher debug <invocation_id1> <invocation_id2>
317-
```
318-
319-
### Show Configuration
320-
321-
```bash
322-
# Display the job configuration in YAML format
323-
nemo-evaluator-launcher debug <invocation_id> --config
324-
```
325-
326-
### Show Paths
327-
328-
```bash
329-
# Show only artifact locations
330-
nemo-evaluator-launcher debug <invocation_id> --artifacts
331-
332-
# Show only log locations
333-
nemo-evaluator-launcher debug <invocation_id> --logs
334-
```
335-
336-
For remote jobs (Slurm), paths are shown in the format `user@host:/path`.
337-
338-
### Copy Files Locally
339-
340-
```bash
341-
# Copy logs to local directory (works for both local and remote jobs)
342-
nemo-evaluator-launcher debug <invocation_id> --copy-logs [destination_dir]
343-
344-
# Copy artifacts to local directory (works for both local and remote jobs)
345-
nemo-evaluator-launcher debug <invocation_id> --copy-artifacts [destination_dir]
346-
347-
# If no destination is specified, defaults to current directory
348-
nemo-evaluator-launcher debug <invocation_id> --copy-logs
349-
```
350-
351-
352-
### Debug example for a slurm job
353-
354-
```bash
355-
# Shows remote paths and Slurm job ID
356-
nemo-evaluator-launcher debug abc12345
357-
# Output includes:
358-
# ├── Artifacts: user@host:/shared/results/artifacts (remote)
359-
# ├── Logs: user@host:/shared/results/logs (remote)
360-
# ├── Slurm Job ID: 12345678
361-
362-
```
363-
364362
## Environment Variables
365363

366364
The CLI respects environment variables for logging and task-specific authentication:
@@ -474,9 +472,9 @@ nemo-evaluator-launcher run --config-dir packages/nemo-evaluator-launcher/exampl
474472
```bash
475473
# Command-specific help
476474
nemo-evaluator-launcher run --help
477-
nemo-evaluator-launcher export --help
475+
nemo-evaluator-launcher info --help
478476
nemo-evaluator-launcher ls --help
479-
nemo-evaluator-launcher debug --help
477+
nemo-evaluator-launcher export --help
480478

481479
# General help
482480
nemo-evaluator-launcher --help

0 commit comments

Comments
 (0)