Skip to content

Conversation

@tautomer
Copy link
Collaborator

@tautomer tautomer commented Aug 14, 2025

This PR addresses the random state management issues in #80 to improve reproducibility and eliminate repeated noise patterns or initial optimizer conditions during a campaign.

Implementation details:

  • Introduced a centralized RNG initialization that occurs once per campaign, controlled by a single seed.
  • Updated the code to pass RNG generator instances explicitly whenever possible to components requiring randomness, avoiding global RNG state resets.
  • Use a context manager for temporary global seed override if explicitly passing a generator is not possible, for example, model.fit and optimizer.suggest.
  • Added fallback behavior to maintain reproducibility even when no seed is provided by initializing RNG states with system entropy but tracking them internally.
  • Added a package level toggle obsidian.USE_OLD_RNG_CONTROL for backward compatibility.

Other changes

Some other improvements were made while this new RNG control was implemented.

  1. Introduced direct optimizer options passthrough to model fitting routine in campaign.fit. You can now do

    campaign.fit({"max_attempts": 10, "multi_starts": True})

    to fit with model with 10 different restarts.

  2. Introduced direct optimizer options passthrough to acquisition function optimization in optimizer.suggest

    campaign.optimizer.suggest(
        ...,
        optim_options={"method": "Powell", "maxiter": 100}
    )

    This will tell scipy.optimize.minimize to use Powell and maximum iteration of 100.

Current status

The following tests have been performed, both on single objective (modified from Simple single objective.ipynb)

  1. The new code, with obsidian.USE_OLD_RNG_CONTROL=True produces the same results as the current main branch.
  2. With new RNG control, two runs with the same initial random states achieve identical results with categorical parameters.
  3. With new RNG control, two runs with the same initial random states achieve identical results without categorical parameters. (fixed in 1064d6f).

TODO:

  1. save_state and load_state have been implemented and functions as they should, but they are not part of larger scope (i.e., campaign level) yet.
  2. More extensive tests needed. (Help appreciated.)

Main changes in RNG:
1. Introduced GlobalRNG class for centralized RNG control
2. Replaced all instances of torch random number generation now with a generator
3. Replaced all instances of numpy.random with a generator
4. Introduced a with_tmp_seed context manager for all hidden/implicit random number generation
5. Introduced a package level toggle USE_OLD_RNG_CONTROL for backward compatibility

Misc fixes/improvements
1. Introduced direct optimizer options passthrough to model fitting routine in campaign.fit
2. Introduced direct optimizer options passthrough to acquisition function optimization in optimizer.suggest
3. Vectorized the noise generation in simulator
4. Some code style cleanups and type hint fixes around RNG-related blocks
@tautomer tautomer added bug Something isn't working enhancement New feature or request labels Aug 14, 2025
@tautomer tautomer linked an issue Aug 14, 2025 that may be closed by this pull request
@tautomer tautomer marked this pull request as draft August 14, 2025 22:23
@tautomer tautomer requested review from kstone40 and xuyuting August 14, 2025 22:23
1. Fix a regression from merging
2. Fix a bug where optimize_acqf get an input of fixed_features_list
@tautomer tautomer requested a review from spencermcminn August 25, 2025 21:36
1. Allow externally defined noise functions (can be related to X)
2. Make the Optithon benchmark function compatible with the Simulator
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RNG leads to repeated random numbers or non-reproducible results

1 participant