fix: Rename variable to control PG timeout to not refer to NCCL

The variable is not backend specific. In the future, if/when we support
other backends, this will become more evidently a problem.

Also clarified docs about the variable a bit.

Signed-off-by: Ihar Hrachyshka <ihar.hrachyshka@gmail.com>

Author: booxter

README.md

@@ -375,4 +375,6 @@ run_training(
 
 Below is a list of custom environment variables users can set in the training library.
 
-1. `INSTRUCTLAB_NCCL_TIMEOUT_MS`, this environment variable controls the NCCL timeout in milliseconds. Consider increasing if seeing FSDP related NCCL errors.
+1. `INSTRUCTLAB_PROCESS_GROUP_TIMEOUT_MS`, this environment variable controls
+   the process group timeout in milliseconds. Consider increasing if seeing
+   FSDP related collective timeout errors.

src/instructlab/training/main_ds.py

@@ -568,7 +568,7 @@ def main(args):
 
     # solution discovered from torchtune https://github.com/pytorch/torchtune/issues/2093
     # gets converted to a timedelta of 1:40:00 if the default is kept
-    nccl_timeout = int(os.getenv("INSTRUCTLAB_NCCL_TIMEOUT_MS", "6000000"))
+    nccl_timeout = int(os.getenv("INSTRUCTLAB_PROCESS_GROUP_TIMEOUT_NS", "6000000"))
     #### distributed init #####
     torch.cuda.set_device(int(os.environ["LOCAL_RANK"]))
     args.local_rank = int(os.environ["LOCAL_RANK"])

tox.ini

@@ -18,7 +18,7 @@ basepython = python3.11
 description = run unit tests with pytest
 passenv =
 	HF_HOME
-        INSTRUCTLAB_NCCL_TIMEOUT_MS
+        INSTRUCTLAB_PROCESS_GROUP_TIMEOUT_MS
 deps = 
     pytest
     -r requirements-dev.txt
@@ -32,7 +32,7 @@ commands = {envpython} -m pytest tests/unit {posargs}
 description = run accelerated smoke tests with pytest
 passenv =
 	HF_HOME
-        INSTRUCTLAB_NCCL_TIMEOUT_MS
+        INSTRUCTLAB_PROCESS_GROUP_TIMEOUT_MS
 deps = 
     pytest
     -r requirements-dev.txt