Skip to content

Commit fa5b53b

Browse files
RobotSailclaude
andauthored
Rename checkpoint trigger file to be product-agnostic (#704)
* Rename checkpoint trigger file to be product-agnostic The on-demand checkpoint sentinel file in /dev/shm was named "instructlab_checkpoint_requested" — rename it to the generic "checkpoint_requested" so the mechanism is not tied to a specific product name. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Allow checkpoint trigger filename to be overridden via env var Read CHECKPOINT_TRIGGER_FILENAME from os.environ at call time in _get_trigger_path() instead of using a module-level constant. This lets users customize the trigger filename for environments where multiple training jobs share the same /dev/shm. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 73ecf57 commit fa5b53b

3 files changed

Lines changed: 15 additions & 5 deletions

File tree

docs/on_demand_checkpointing.md

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -135,9 +135,13 @@ The trigger file is always at a fixed path. To trigger a checkpoint
135135
(e.g. via `kubectl exec` into the training pod):
136136

137137
```bash
138-
touch /dev/shm/instructlab_checkpoint_requested
138+
touch /dev/shm/checkpoint_requested
139139
```
140140

141+
The default filename is `checkpoint_requested`. To use a custom filename,
142+
set the `CHECKPOINT_TRIGGER_FILENAME` environment variable before starting
143+
training.
144+
141145
Workers check for the trigger file at each synchronization point in the
142146
training loop (multiple times per step). Once any rank on any node detects
143147
it, all ranks coordinate via `all_reduce` to save a checkpoint and exit.

src/instructlab/training/on_demand_checkpoint.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -89,12 +89,12 @@
8989
# 2. Shared between all containers in the same Kubernetes pod.
9090
# 3. Automatically cleaned up when the pod is destroyed.
9191
_TRIGGER_DIR = Path("/dev/shm")
92-
_TRIGGER_FILENAME = "instructlab_checkpoint_requested"
9392

9493

9594
def _get_trigger_path() -> Path:
9695
"""Return the path to the checkpoint trigger file."""
97-
return _TRIGGER_DIR / _TRIGGER_FILENAME
96+
filename = os.environ.get("CHECKPOINT_TRIGGER_FILENAME", "checkpoint_requested")
97+
return _TRIGGER_DIR / filename
9898

9999

100100
def write_trigger_file() -> Path:

tests/unit/test_on_demand_checkpoint.py

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,13 @@
2929
class TestGetTriggerPath:
3030
def test_returns_correct_name(self):
3131
path = _get_trigger_path()
32-
assert path.name == "instructlab_checkpoint_requested"
32+
assert path.name == "checkpoint_requested"
33+
assert str(path.parent) == "/dev/shm"
34+
35+
def test_respects_env_override(self, monkeypatch):
36+
monkeypatch.setenv("CHECKPOINT_TRIGGER_FILENAME", "my_custom_trigger")
37+
path = _get_trigger_path()
38+
assert path.name == "my_custom_trigger"
3339
assert str(path.parent) == "/dev/shm"
3440

3541

@@ -43,7 +49,7 @@ def test_creates_file(self, tmp_path):
4349
def test_returns_correct_path(self, tmp_path):
4450
with patch("instructlab.training.on_demand_checkpoint._TRIGGER_DIR", tmp_path):
4551
path = write_trigger_file()
46-
assert path == tmp_path / "instructlab_checkpoint_requested"
52+
assert path == tmp_path / "checkpoint_requested"
4753

4854

4955
class TestTriggerFileExists:

0 commit comments

Comments
 (0)