Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Assign tasks to workers #525

Merged
merged 9 commits into from
Apr 19, 2023
Merged

Assign tasks to workers #525

merged 9 commits into from
Apr 19, 2023

Conversation

gabegma
Copy link
Contributor

@gabegma gabegma commented Mar 31, 2023 •
edited
Loading

Part of #521
Fixes #411

Description:

  • The tests used to be flacky on the CI, but I after rebasing on all the changes, I'm not sure if that is still the case. It seems it's no longer the case given the amount of back and forth. Yay!!
  • This allows the user to manually assign the GPU to worker 0, because this is where the pipeline and encoder will be loaded.

Here is the memory before and after the change.
6 before
7 assign workers

Checklist:

You should check all boxes before the PR is ready. If a box does not apply, check it to acknowledge it.

  • ISSUE NUMBER. You linked the issue number (Ex: Resolve #XXX).
  • PRE-COMMIT. You ran pre-commit on all commits, or else, you
    ran pre-commit run --all-files at the end.
  • USER CHANGES. The changes are added to CHANGELOG.md and the documentation, if they impact
    our users.
  • DEV CHANGES.
    • Update the documentation if this PR changes how to develop/launch on the app.
    • Update the README files and our wiki for any big design decisions, if relevant.
    • Add unit tests, docstrings, typing and comments for complex sections.

@gabegma gabegma marked this pull request as draft March 31, 2023 19:31
@gabegma gabegma force-pushed the ggm/improve-stability-start-up branch from a7018d4 to 825ef39 Compare March 31, 2023 20:08
@gabegma gabegma changed the title Improve stability start up Assign tasks to workers Mar 31, 2023
@gabegma gabegma changed the title Assign tasks to workers Assign tasks to workers and restart client Mar 31, 2023
@gabegma gabegma force-pushed the ggm/improve-stability-start-up branch from 36b53d8 to 825ef39 Compare April 3, 2023 15:10
@gabegma gabegma changed the title Assign tasks to workers and restart client Assign tasks to workers Apr 3, 2023
@gabegma gabegma marked this pull request as ready for review April 3, 2023 15:35
@gabegma gabegma mentioned this pull request Apr 3, 2023
Merged
4 tasks
@gabegma gabegma force-pushed the ggm/improve-stability-start-up branch from 825ef39 to bfa4299 Compare April 3, 2023 18:51
@gabegma gabegma self-assigned this Apr 3, 2023
@gabegma gabegma force-pushed the ggm/improve-stability-start-up branch 2 times, most recently from 2dd8ce8 to a02a5bb Compare April 5, 2023 18:00
@gabegma gabegma force-pushed the ggm/improve-stability-start-up branch 4 times, most recently from c28c839 to cdaa737 Compare April 9, 2023 02:00
@gabegma gabegma marked this pull request as draft April 9, 2023 02:25
@gabegma gabegma force-pushed the ggm/improve-stability-start-up branch from cdaa737 to e43830a Compare April 13, 2023 21:23
@gabegma gabegma changed the base branch from main to ggm/refactor-artifact-manager April 13, 2023 21:23
@gabegma gabegma force-pushed the ggm/improve-stability-start-up branch from e43830a to 2b6dd36 Compare April 13, 2023 21:47
@gabegma gabegma force-pushed the ggm/refactor-artifact-manager branch from 3ecfa72 to c5fbc3c Compare April 18, 2023 03:06
@gabegma gabegma force-pushed the ggm/improve-stability-start-up branch from 2b6dd36 to 400c873 Compare April 18, 2023 19:04
@gabegma gabegma marked this pull request as ready for review April 18, 2023 19:19
@gabegma gabegma requested a review from JosephMarinier April 18, 2023 19:19
Base automatically changed from ggm/refactor-artifact-manager to main April 18, 2023 21:13
@gabegma gabegma force-pushed the ggm/improve-stability-start-up branch from 7308b74 to 009f85e Compare April 19, 2023 18:16
@gabegma gabegma force-pushed the ggm/improve-stability-start-up branch from 009f85e to 853dae9 Compare April 19, 2023 18:26
@gabegma gabegma force-pushed the ggm/improve-stability-start-up branch from cdcd11b to e7045a4 Compare April 19, 2023 18:45
@gabegma gabegma force-pushed the ggm/improve-stability-start-up branch from e7045a4 to 1657e0e Compare April 19, 2023 18:52
@gabegma
Copy link
Contributor Author

gabegma commented Apr 19, 2023

BTW - While investigating memory usage - I’m under the impression that we have some memory leakage issues with the saliency tasks. I added issue #549.

JosephMarinier
JosephMarinier approved these changes Apr 19, 2023
View reviewed changes
Copy link
Contributor

@JosephMarinier JosephMarinier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! Good job on the graphs and all the profiling. 👍

@gabegma gabegma merged commit add5b58 into main Apr 19, 2023
@gabegma gabegma deleted the ggm/improve-stability-start-up branch April 19, 2023 20:35
gabegma added a commit that referenced this pull request Apr 19, 2023
@gabegma
Assign tasks to workers (#525)
c956dbf 
* Send tasks that require the pipelines to same worker

* Move encoder to ArtifactManager

* Add memory-profiler to profile the memory usage

* Move the encoder out of the ArtifactManager

* Adapt based on comments

* Assign encoder and model tasks to same worker

* Fix similarity test

* Add worker to custom tasks

* Add TODO
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Dref360 Dref360 Dref360 left review comments

@JosephMarinier JosephMarinier JosephMarinier approved these changes

Assignees

@gabegma gabegma

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Keep GPU workload to one worker locally.
3 participants
@gabegma @JosephMarinier @Dref360