Skip to content

Conversation

rtibbles
Copy link
Member

@rtibbles rtibbles commented Sep 2, 2025

Summary

  • Adds better handling for multiprocessing task workers for the threading manager used in resource import
  • Adds required fixes for full worker multiprocessing
  • Restricts multiprocessing to Python 3.7 plus because of initializer kwargs support for pool executors
  • Increases the potential parallelism for resource download tasks by increasing the number of max workers to 50, and increasing the requests connection pool size to match

References

Potential future improvement for #13680

Reviewer guidance

This mostly works, but I have observed issues with tasks stalling, and when that happens Kolibri also taking a long time to shut down. Until we can isolate what is causing this (which presumably have the same root cause), multiprocessing workers cannot be the default.

@github-actions github-actions bot added DEV: backend Python, databases, networking, filesystem... SIZE: small labels Sep 2, 2025
Copy link
Contributor

github-actions bot commented Sep 2, 2025

@rtibbles rtibbles force-pushed the streamline_download_multiprocessing_more_parallelism branch from e9f0ae4 to a16095e Compare September 2, 2025 22:39
@rtibbles rtibbles changed the base branch from release-v0.18.x to develop September 3, 2025 18:08
@rtibbles rtibbles marked this pull request as draft September 15, 2025 15:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
DEV: backend Python, databases, networking, filesystem... SIZE: small
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant