feat: resume interrupted dataset generation runs (sync + async engine)#526

Open

przemekboruta wants to merge 21 commits intoNVIDIA-NeMo:mainfrom

przemekboruta:main

Commits on May 4, 2026

docs: add implementation plan for resume mechanism
przemekboruta
committed
feat(storage): add resume flag and clear_partial_results()
przemekboruta
committed
feat(batch-manager): add start_batch param to start()
przemekboruta
committed
feat(builder): implement resume logic in DatasetBuilder
przemekboruta
committed
feat(interface): expose resume on DataDesigner.create()
przemekboruta
committed
test: add tests for resume mechanism
przemekboruta
committed
feat(builder): extend resume to async engine (DATA_DESIGNER_ASYNC_ENGINE=1)
przemekboruta
committed
fix(builder): skip after-generation processors when resume finds dataset already complete
przemekboruta
committed
fix(builder): use filesystem count for initial_total_num_batches on async resume
przemekboruta
committed
feat(results): add export() method and --output-format CLI flag
przemekboruta
committed
fix(builder): handle resume when metadata.json missing (interrupted before first batch)
przemekboruta
committed
docs(interface): fix resume docstring — async engine is supported
przemekboruta
committed
fix(builder): derive initial_actual_num_records from filesystem in async resume
przemekboruta
committed
feat(resume): replace resume: bool with ResumeMode enum (NEVER/ALWAYS/IF_POSSIBLE)
przemekboruta
committed
fix(resume): invalidate resolved_dataset_name cache when IF_POSSIBLE downgrades to NEVER
przemekboruta
committed
fix(builder): move partial-completion warning before return in _build_async
przemekboruta
committed
fix(builder): IF_POSSIBLE now starts fresh when no dataset directory exists
przemekboruta
committed
fix(builder): use original target_num_records in async resume record count
przemekboruta
committed
fix(builder): IF_POSSIBLE starts fresh on empty dataset directory
przemekboruta
committed
fix(builder): ALWAYS raises DatasetGenerationError on config fingerprint mismatch
przemekboruta
committed

Commits on May 5, 2026

Merge branch 'main' into main
nabinchha
authored

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: resume interrupted dataset generation runs (sync + async engine)#526

feat: resume interrupted dataset generation runs (sync + async engine)#526
przemekboruta wants to merge 21 commits intoNVIDIA-NeMo:mainfrom
przemekboruta:main

Commits on May 4, 2026

docs: add implementation plan for resume mechanism

feat(storage): add resume flag and clear_partial_results()

feat(batch-manager): add start_batch param to start()

feat(builder): implement resume logic in DatasetBuilder

feat(interface): expose resume on DataDesigner.create()

test: add tests for resume mechanism

feat(builder): extend resume to async engine (DATA_DESIGNER_ASYNC_ENGINE=1)

fix(builder): skip after-generation processors when resume finds dataset already complete

fix(builder): use filesystem count for initial_total_num_batches on async resume

feat(results): add export() method and --output-format CLI flag

fix(builder): handle resume when metadata.json missing (interrupted before first batch)

docs(interface): fix resume docstring — async engine is supported

fix(builder): derive initial_actual_num_records from filesystem in async resume

feat(resume): replace resume: bool with ResumeMode enum (NEVER/ALWAYS/IF_POSSIBLE)

fix(resume): invalidate resolved_dataset_name cache when IF_POSSIBLE downgrades to NEVER

fix(builder): move partial-completion warning before return in _build_async

fix(builder): IF_POSSIBLE now starts fresh when no dataset directory exists

fix(builder): use original target_num_records in async resume record count

fix(builder): IF_POSSIBLE starts fresh on empty dataset directory

fix(builder): ALWAYS raises DatasetGenerationError on config fingerprint mismatch

Commits on May 5, 2026

Merge branch 'main' into main