Skip to content

Conversation

@jmartin-tech
Copy link
Collaborator

#1414 exposed an underlying race condition when using multi-processing capable generators, that are stored in a probe, on the primary execution thread. Serialization leaves the generator in a state that requires client connections to be rebuilt and usage of locks to guard _call_model would introduce further issue in serialization for multiprocessing actions.

By storing local references to the client objects during inference calls the serialization side-effects are limited to a much smaller runtime window

Further work is needed to remove the race condition fully, this is simply a short term mitigation until a more complete refactor can be designed.

Verification

List the steps needed to make sure this thing works

python -m garak -t nim -n <target model name> --parallel_attempts 32 -p <iterative probe class>

@jmartin-tech
Copy link
Collaborator Author

Future iteration will provide unit testing of this type of race condition, this was deferred at this time in the interest of delivering iterative probe patterns more quickly.

There are likely a number of edge cases related to multi-processing that will pop up as the community starts to exercise more probes techniques that require an LLM during the target inference as may multi-turn techniques often rely on this pattern.

@leondz leondz added the bug Something isn't working label Nov 7, 2025
@jmartin-tech jmartin-tech changed the title reduce window for multiprocesing race fix: reduce window for multiprocessing race Nov 7, 2025
By storing local references to the client objects during inference
calls the serialization side-effects are limited to a much smaller
runtime window

Signed-off-by: Jeffrey Martin <[email protected]>
@jmartin-tech jmartin-tech force-pushed the fix/improve-openai-mulitporcessing-support branch from f7c9cd2 to 2906211 Compare November 7, 2025 19:03
@jmartin-tech jmartin-tech merged commit e515dbe into NVIDIA:main Nov 7, 2025
15 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Nov 7, 2025
@jmartin-tech jmartin-tech deleted the fix/improve-openai-mulitporcessing-support branch November 10, 2025 15:16
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants