SaveConfigCallback does not work as expected and raises an error under specific settings (DDPStrategy start_method="spawn") #21277

Activity

1. lengmo1996

   added

   bugSomething isn't workingSomething isn't working

   needs triageWaiting to be triaged by maintainersWaiting to be triaged by maintainers

   on Oct 9, 2025
2. github-actions

   added

   ver: 2.5.x

   on Oct 9, 2025
3. lengmo1996 commented on Oct 9, 2025

   

   lengmo1996

   on Oct 9, 2025

   Author

   More actions

   🔄 Additional findings and diagnostics

   1. start_method="fork" also fails

   After further testing, the same FileNotFoundError occurs even when switching to

   DDPStrategy(start_method="fork")
   

   So the issue is not specific to "spawn" — both "spawn" and "fork" modes trigger the same error:

   FileNotFoundError: [Errno 2] No such file or directory: '/workspace/auto'
   

   This suggests that the root cause is related to how LightningCLI handles "auto" default values during argument resolution, not merely a multiprocessing problem.

   2. Effect of jsonargparse / omegaconf loader registration

   When I comment out the following block:

   # import jsonargparse
   # from omegaconf import OmegaConf
   # from jsonargparse import set_loader, set_dumper
   # if "omegaconf" not in jsonargparse._loaders_dumpers.loaders:
   
   #     set_loader("omegaconf", OmegaConf.load)
   
   # if "omegaconf" not in jsonargparse._loaders_dumpers.dumpers:
   
   #     set_dumper("omegaconf", OmegaConf.save)
   
   

   the behavior changes depending on the multiprocessing method:

   DDPStrategy(start_method=...) | Behavior -- | -- "fork" | ✅ Works fine — no FileNotFoundError "spawn" | ❌ Raises KeyError: 'Validation failed: omegaconf'

   This indicates that:

   • The global loader/dumper registry (jsonargparse._loaders_dumpers) is not shared across processes when using "spawn".

   • LightningCLI likely attempts to validate or reload configs in the new process before the omegaconf loader is re-registered, leading to KeyError or misinterpreting "auto" as a literal path.

   • Under "fork", the registry state is inherited, which explains the behavioral difference.

   💡 Hypothesis

   The bug seems to stem from how LightningCLI manages or reuses jsonargparse’s omegaconf loader/dumper registration across subprocesses.
   When using "spawn", that registration doesn’t propagate, causing either:

   • Missing loader errors (KeyError: 'Validation failed: omegaconf'), or

   • Misinterpretation of config defaults like "auto" as file paths (FileNotFoundError: '/workspace/auto').

   A safer approach might be to re-register the omegaconf loader/dumper explicitly within each spawned process, or to avoid performing config validation in workers.
4. SkafteNicki

   added

   lightningclipl.cli.LightningCLIpl.cli.LightningCLI

   strategy: ddpDistributedDataParallelDistributedDataParallel

   and removed

   needs triageWaiting to be triaged by maintainersWaiting to be triaged by maintainers

   on Oct 10, 2025
5. mauvilsa commented on Oct 10, 2025

   

   mauvilsa

   on Oct 10, 2025

   Contributor

   More actions

   if "omegaconf" not in jsonargparse._loaders_dumpers.loaders:
       set_loader("omegaconf", OmegaConf.load)
   if "omegaconf" not in jsonargparse._loaders_dumpers.dumpers:
       set_dumper("omegaconf", OmegaConf.save)

   @lengmo1996 this is incorrect, why are you doing this? set_loader expects a function that receives a string to parse. OmegaConf.load expects a path, so it is not compatible. I mean OmegaConf.load('{"x":1}') would need to work, but it fails with FileNotFoundError: [Errno 2] No such file or directory: '/tmp/{"x":1}'. And this is the error you are getting.

   Similarly the use of OmegaConf.save is not valid. The jsonargparse docs could be improved, but still with this, even the most basic parser would fail. Also the _loaders_dumpers model starts with _ to indicate that is private, so users should not access it.

   However, what I wonder most about is why are you using set_loader and set_dumper for OmegaConf. As explained in the docs, the support for OmegaConf comes out of the box. The only thing to do is to set the parser mode to "omegaconf". No need to set a loader or dumper.

   👍React with 👍1Reacted by Guoqing Zhu
6. lengmo1996 commented on Oct 11, 2025

   

   lengmo1996

   on Oct 11, 2025

   Author

   More actions

   Thank you for your prompt reply and detailed explanation! @mauvilsa

   In fact, the initial problem I encountered occurred when debugging in VSCode using DDPStrategy with the default start_method, which is 'popen'.
   At that time, when num_workers > 0 and the execution reached val_run(), I encountered a rather tricky error:

   "/usr/local/lib/python3.12/dist-packages/torch/utils/data/_utils/signal_handling.py", line 73, in handler
   [rank0]: _error_if_any_worker_fails()
   [rank0]: RuntimeError: DataLoader worker (pid 3081641) is killed by signal: Aborted.
   

   Of course, setting num_workers=0 can avoid this issue, but after checking some references, I found that the 'popen' method may have certain compatibility problems.

   Therefore, I tried switching to the 'spawn' method — which then led to the issue discussed here.
   However, when using 'spawn', the first error I got was:

   W1011 06:29:49.602000 13409 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 13824 via signal SIGTERM
   Traceback (most recent call last):
     File "/opt/conda/lib/python3.11/runpy.py", line 198, in _run_module_as_main
       return _run_code(code, main_globals, None,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/opt/conda/lib/python3.11/runpy.py", line 88, in _run_code
       exec(code, run_globals)
     File "/root/.vscode-server/extensions/ms-python.debugpy-2025.10.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 71, in <module>
       cli.main()
     File "/root/.vscode-server/extensions/ms-python.debugpy-2025.10.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 501, in main
       run()
     File "/root/.vscode-server/extensions/ms-python.debugpy-2025.10.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 351, in run_file
       runpy.run_path(target, run_name="__main__")
     File "/root/.vscode-server/extensions/ms-python.debugpy-2025.10.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 310, in run_path
       return _run_module_code(code, init_globals, run_name, pkg_name=pkg_name, script_name=fname)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/root/.vscode-server/extensions/ms-python.debugpy-2025.10.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 127, in _run_module_code
       _run_code(code, mod_globals, init_globals, mod_name, mod_spec, pkg_name, script_name)
     File "/root/.vscode-server/extensions/ms-python.debugpy-2025.10.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 118, in _run_code
       exec(code, run_globals)
     File "main.py", line 17, in <module>
       LightningCLI(
     File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/cli.py", line 414, in __init__
       self._run_subcommand(self.subcommand)
     File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/cli.py", line 747, in _run_subcommand
       fn(**fn_kwargs)
     File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/trainer/trainer.py", line 560, in fit
       call._call_and_handle_interrupt(
     File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/trainer/call.py", line 48, in _call_and_handle_interrupt
       return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/strategies/launchers/multiprocessing.py", line 144, in launch
       while not process_context.join():
                 ^^^^^^^^^^^^^^^^^^^^^^
     File "/opt/conda/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 215, in join
       raise ProcessRaisedException(msg, error_index, failed_process.pid)
   torch.multiprocessing.spawn.ProcessRaisedException: 
   
   -- Process 0 terminated with the following error:
   Traceback (most recent call last):
     File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_typehints.py", line 558, in _check_type
       val, config_path = parse_value_or_config(val, enable_path=enable_path)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_util.py", line 145, in parse_value_or_config
       parsed_val = load_value(value, simple_types=simple_types)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_loaders_dumpers.py", line 199, in load_value
       if loaded_value is not_loaded and not loader_json_superset[mode]:
                                             ~~~~~~~~~~~~~~~~~~~~^^^^^^
   KeyError: 'omegaconf'
   
   During handling of the above exception, another exception occurred:
   
   Traceback (most recent call last):
     File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_core.py", line 1162, in validate
       check_values(cfg)
     File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_core.py", line 1144, in check_values
       self._check_value_key(action, val, key, ccfg)
     File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_core.py", line 1464, in _check_value_key
       value = action._check_type_(value, cfg=cfg, append=append, mode=self.parser_mode)  # type: ignore[attr-defined]
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_common.py", line 391, in _check_type_
       return self._check_type(value, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_typehints.py", line 559, in _check_type
       except get_loader_exceptions():
              ^^^^^^^^^^^^^^^^^^^^^^^
     File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_loaders_dumpers.py", line 168, in get_loader_exceptions
       return loader_exceptions[mode]
              ~~~~~~~~~~~~~~~~~^^^^^^
   KeyError: 'omegaconf'
   
   The above exception was the direct cause of the following exception:
   
   Traceback (most recent call last):
     File "/opt/conda/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 90, in _wrap
       fn(i, *args)
     File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/strategies/launchers/multiprocessing.py", line 173, in _wrapping_function
       results = function(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/trainer/trainer.py", line 598, in _fit_impl
       self._run(model, ckpt_path=ckpt_path)
     File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/trainer/trainer.py", line 973, in _run
       call._call_setup_hook(self)  # allow user to set up LightningModule in accelerator environment
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/trainer/call.py", line 109, in _call_setup_hook
       _call_callback_hooks(trainer, "setup", stage=fn)
     File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/trainer/call.py", line 228, in _call_callback_hooks
       fn(trainer, trainer.lightning_module, *args, **kwargs)
     File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/cli.py", line 283, in setup
       self.parser.save(
     File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_deprecated.py", line 168, in patched_save
       return self._unpatched_save(cfg, *args, multifile=multifile, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_core.py", line 922, in save
       f.write(self.dump(cfg, **dump_kwargs))  # type: ignore[arg-type]
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_deprecated.py", line 159, in patched_dump
       return self._unpatched_dump(cfg, *args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_core.py", line 800, in dump
       self.validate(cfg)
     File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_core.py", line 1170, in validate
       raise type(ex)(message) from ex
   KeyError: 'Validation failed: omegaconf'
   

   The commonly suggested fix online was to add the following code:

   if "omegaconf" not in jsonargparse._loaders_dumpers.loaders:
       set_loader("omegaconf", OmegaConf.load)
   if "omegaconf" not in jsonargparse._loaders_dumpers.dumpers:
       set_dumper("omegaconf", OmegaConf.save)
   

   After adding this, I encountered another error:

   FileNotFoundError: [Errno 2] No such file or directory: '/workspace/auto'
   

   That’s the whole process I went through.
   However, I think the key point here is that when running the code mentioned in this issue — that is, setting start_method='spawn' — it always throws

   KeyError: 'Validation failed: omegaconf'
   

   Is this behavior expected, or does it indicate that I’m doing something wrong in my setup?
7. mauvilsa commented on Oct 11, 2025

   

   mauvilsa

   on Oct 11, 2025

   Contributor

   More actions

   The commonly suggested fix online was to add the following code

   Please share the location where this was suggested. It should be commented there that it is wrong, to avoid misleading people.

   However, I think the key point here is that when running the code mentioned in this issue — that is, setting start_method='spawn' — it always throws

   KeyError: 'Validation failed: omegaconf'
   

   This does look like an actual bug. I will look into it.
8. mauvilsa

   added 2 commits that reference this issue on Oct 13, 2025

   Fix omegaconf parser mode failing on spawned processes (Lightning-AI/…

   ...

   ba1ae84

   Fix omegaconf parser mode failing on spawned processes (Lightning-AI/…

   ...

   8fe3d86
9. mauvilsa commented on Oct 13, 2025

   

   mauvilsa

   on Oct 13, 2025

   Contributor

   More actions

   @lengmo1996 the issue should be fixed with mauvilsa/jsonargparse#784. You can install the current jsonargparse in develop and test it out.

   The commonly suggested fix online was to add the following code

   I tried to find such suggestions but with no luck. If you do have a link, please share it. Or maybe by "online" did you mean that an LLM suggested this to you?
10. lengmo1996 commented on Oct 13, 2025

    

    lengmo1996

    on Oct 13, 2025

    Author

    More actions

    @mauvilsa Thanks a lot for the quick fix! Really appreciate your prompt response and hard work. 🙏
    You’re right — that came from a suggestion by ChatGPT. I also checked around but couldn’t find any related sources online. My apologies for not confirming it more carefully — I hope it didn’t cause you too much inconvenience.

    👍React with 👍1Reacted by Mauricio Villegas
11. lengmo1996

    closed this as completedon Oct 13, 2025