Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Autoscaling - Dynamic Services #657

Closed
mguidon opened this issue Aug 4, 2022 · 14 comments · Fixed by ITISFoundation/osparc-simcore#3970, ITISFoundation/osparc-simcore#3914, ITISFoundation/osparc-simcore#3902 or ITISFoundation/osparc-simcore#3876
Closed

Autoscaling - Dynamic Services #657

mguidon opened this issue Aug 4, 2022 · 14 comments · Fixed by ITISFoundation/osparc-simcore#3970, ITISFoundation/osparc-simcore#3914, ITISFoundation/osparc-simcore#3902 or ITISFoundation/osparc-simcore#3876
Assignees
@sanderegg
Labels
PO issue Created by Product owners
Milestone
Mithril

Comments

@mguidon
Copy link
Member

mguidon commented Aug 4, 2022 •
edited
Loading

Auto-scale the docker swarm nodes for the dynamic services.

  • Reuse the tooling created in Autoscaling - Computational Backend #656 for scaling of the cluster
  • The metrics must come from the docker swarm and has to include GPU
  • This will require to assign/consume proper numbers for the available VRAM/AIRAM
  • This will result in users having to wait for machines coming available and they need to be notified about that.
  • Use case will be running tons of sim4life:web instances
@esraneufeld esraneufeld added the PO issue Created by Product owners label Aug 5, 2022
This was referenced Aug 8, 2022
Closed
Closed
@mrnicegyu11
Copy link
Member

mrnicegyu11 commented Sep 7, 2022 •
edited
Loading

a

@sanderegg sanderegg added the Epic label Sep 9, 2022
@pcrespov pcrespov mentioned this issue Sep 18, 2022
Merged
8 tasks
@mrnicegyu11 mrnicegyu11 mentioned this issue Oct 13, 2022
Closed
12 tasks
@sanderegg
Copy link
Member

sanderegg commented Oct 17, 2022 •
edited by Surfict
Loading

Goal for sprint Katherine Switzer

  • show scaling of nodes in the frontend
  • run autoscaling app background task
  • Gracefully handle: No more machines on AWS available
  • --> start a service and wait for a new node to start

@sanderegg
Copy link
Member

Update on sprint Katherine Switzer

  • New Autoscaling application in osparc-simcore stack
    • unit tested
    • allows backend devs to contribute
    • migration still in progress
  • Frontend notifications in progress (#3502)

@colinRawlings colinRawlings assigned mrnicegyu11 and mguidon and unassigned GitHK Nov 14, 2022
@sanderegg
Copy link
Member

sanderegg commented Nov 14, 2022 •
edited
Loading

Goal for sprint Athena

  • Migrate autoscaling script into osparc-simcore autoscaling app osparc-simcore#3558
  • cleanup remaining nodes in the docker swarm, once they are gone
  • Gracefully handle: No more machines on AWS available in case AWS has none available (take the next big one) --> add an ENV SAN
  • all sim4life-light-dy shall start on auto-scaled nodes
  • add VRAM, Generic resources handling
  • test on AWS staging by deploying the full stack (reconfigure deployer) SAN/ALL
  • apply to AWS to get more vCPUs [ALL]
  • metrics (prometheus)
  • Robustness to bugs in dy-sidecar (restarts, manual killing,...)

@sanderegg sanderegg mentioned this issue Nov 23, 2022
Merged
@elisabettai
Copy link
Contributor

@mrnicegyu11 please add new devops stuff

@mrnicegyu11
Copy link
Member

The following tasks also have to be tackled, these need to be done on the DevOps side:

  • Determine how to provide the AMI image for autoscaled nodes (talk to @mguidon )
  • Increase machine linux password security
  • Create aws functional user for autoscaling task with limited capabilities

@sanderegg
Copy link
Member

sanderegg commented Dec 1, 2022 •
edited
Loading

Update on sprint Athena

Done

Ongoing

Next steps

@sanderegg
Copy link
Member

Goal for sprint Zefram Cochrane

@mguidon mguidon mentioned this issue Dec 12, 2022
Closed
13 tasks
@sanderegg
Copy link
Member

sanderegg commented Jan 10, 2023 •
edited
Loading

Update on sprint Zefram Cochrane

Done

Ongoing

Next steps

@mguidon
Copy link
Member Author

mguidon commented Jan 20, 2023

Think about adding 1 machine as a buffer.

@sanderegg
Copy link
Member

sanderegg commented Feb 17, 2023 •
edited
Loading

Update on sprint Resistance is Futile

Done

Ongoing

Next steps

@sanderegg sanderegg added this to the Mithril milestone Feb 28, 2023
@sanderegg
Copy link
Member

sanderegg commented Feb 28, 2023 •
edited
Loading

Goal for sprint Mithril

@sanderegg
Copy link
Member

sanderegg commented Mar 1, 2023 •
edited
Loading

@drniiken for info these are my latest tests with different combinations as of yesterday in staging-AWS (staging.osparc.io) using the latest code. (not yet in production). We can check together tomorrow if you're interested.

testing s4l-lite 2.0.106 startup times

  • times [min:sec] coming from the osparc logs
  • fresh means the machine is created on demand (no buffer)
  • buffer means the machine was pre-created and is available
  • no pre-pull means nothing is pulled on machine start
  • osp pre-pull means only monitoring docker images are pulled on start
  • ops + 2.0.106 means monitoring images and the largest image of s4l-lite is pulled on start
type service-started new machine ready sidecar first log S4l-lite ready
fresh no pre-pull 00:00 02:18 04:30 07:09
fresh ops pre-pull 00:00 02:21 03:53 06:37
fresh full pre-pull 00:00 02:15 03:55 06:31
buffer no pre-pull 00:00 00:09 01:42 04:18
buffer full pre-pull 00:00 00:03 01:37 02:08
hot drained node (standard) 00:00 00:05 00:40 ~01:00

This was referenced Mar 16, 2023
Merged
Merged
Merged
Merged
@sanderegg
Copy link
Member

sanderegg commented Mar 27, 2023 •
edited
Loading

Update on sprint Mithril

Done

MVP completed and deployed to production.
Other steps (features/bugs) shall be tackled in the maintenance case and/or as separate PO cases.

@sanderegg sanderegg mentioned this issue Sep 21, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sanderegg sanderegg

Labels
PO issue Created by Product owners
Projects
None yet
Milestone
Mithril
7 participants
@Surfict @GitHK @mrnicegyu11 @elisabettai @mguidon @esraneufeld @sanderegg