Skip to content

Added stress test#1015

Open
pmahindrakar-oss wants to merge 7 commits intomainfrom
stress-test
Open

Added stress test#1015
pmahindrakar-oss wants to merge 7 commits intomainfrom
stress-test

Conversation

@pmahindrakar-oss
Copy link
Copy Markdown
Collaborator

Example run

examples/stress/sleep_fanout_harness_wrapper.sh \                                                                           flyte
    --config ~/.flyte/config-dogfood.yaml \
    --total-runs 20 \
    --submit-concurrency 20 \
    --n-children 5000 \
    --sleep-duration 900 \
    --poll-interval 2 \
    --run-env _F_MAX_QPS=100 \
    --run-env _F_CTRL_WORKERS=10 \
    --run-env _F_P_CNC=5000
Launching local multi-run harness
  config: /Users/praful/.flyte/config-dogfood.yaml
  total_runs: 20
  submit_concurrency: 20
  n_children_per_run: 5000
  total_children_expected: 100000
  sleep_duration: 900
  poll_interval: 2s
  image target: 376129846803.dkr.ecr.us-east-2.amazonaws.com/union/dogfood
  image builder: remote
  image platforms: linux/amd64
  sdk source: installed flyte via /Users/praful/flyte-sdk/.venv/bin/flyte
  sdk wheel: /Users/praful/flyte-sdk/dist/flyte-2.0.0b18.dev742+g55744f779.d20260427-py3-none-any.whl
  warning: src/flyte is newer than the dist wheel; remote image will not include recent SDK src changes until you rebuild the wheel
  fanout parent resources: cpu 1/2, memory 2Gi/4Gi
  use_actions: 1
  run env overrides: _F_MAX_QPS=100 _F_CTRL_WORKERS=10 _F_P_CNC=5000

time     runs         p_live   p_run    seen_children      not_created    d_seen   create_rps rps/p    eta_fill   running  active  
12:14:39 0/20         0        0        0/100000           100000         0        0          0        0          0        0       
12:14:41 0/20         0        0        0/100000           100000         0        0          0        0          0        0       
12:14:43 0/20         0        0        0/100000           100000         0        0          0        0          0        0       
launch: [12] submit failed rc=1 output='  > Launching remote execution...\n  > Building 2 images...\n    > Building image dogfood for environment sleep_fanout\n    > Building image dogfood for environment sleep_fanout_leaf\n    i Image dogfood:414bee6cdf4315469657ed44360c8d54 was not found or has\nexpired\n    > Image\n376129846803.dkr.ecr.us-east-2.amazonaws.com/union/dogfood:414bee6cdf4315469657e\nd44360c8d54 not found, building...\n\x1b[2;36m╭─\x1b[0m\x1b[2;36m────\x1b[0m\x1b[2;36m \x1b[0m\x1b[1;2;38;5;220mException\x1b[0m\x1b[2;36m \x1b[0m\x1b[2;36m─────\x1b[0m\x1b[2;36m─╮\x1b[0m\n\x1b[2;36m│\x1b[0m \x1b[31m✕ Execution failed:\x1b[0m  \x1b[2;36m│\x1b[0m\n\x1b[2;36m╰──────────────────────╯\x1b[0m'
12:14:45 0/20         0        0        0/100000           100000         0        0          0        0          0        0       
12:14:59 15/20        15       15       0/100000           100000         0        n/a        n/a      n/a        0        0       
12:15:19 19/20        19       19       1354/100000        98646          1354     67.7       3.6      00:24:18   647      647     
12:15:47 19/20        19       19       5950/100000        94050          4596     164.1      8.6      00:09:33   3462     3462    
12:16:23 19/20        19       19       12029/100000       87971          6079     168.9      8.9      00:08:41   8114     8114    
12:17:16 19/20        19       19       19718/100000       80282          7689     145.1      7.6      00:09:14   18121    18121   
12:18:31 19/20        18       18       30988/100000       69012          11270    150.3      8.3      00:07:40   28970    28970   
12:20:30 19/20        18       18       55537/100000       44463          24549    206.3      11.5     00:03:36   37370    37370   
12:23:13 19/20        18       18       80892/100000       19108          25355    155.6      8.6      00:02:03   62501    62501   
12:46:32 19/20        18       18       90000/100000       10000          9108     6.5        0.4      00:25:37   74220    74220   
13:00:25 19/20        18       18       90000/100000       10000          0        0.0        0.0      n/a        78895    78895   
13:03:21 19/20        18       18       90000/100000       10000          0        0.0        0.0      n/a        79592    79592   
13:06:21 19/20        18       18       90000/100000       10000          0        0.0        0.0      n/a        79778    79778   
13:09:22 19/20        18       18       90000/100000       10000          0        0.0        0.0      n/a        80167    80167   
13:12:23 19/20        18       18       90000/100000       10000          0        0.0        0.0      n/a        80398    80398   
13:15:30 19/20        18       18       90000/100000       10000          0        0.0        0.0      n/a        80421    80421   
13:18:30 19/20        18       18       90000/100000       10000          0        0.0        0.0      n/a        80495    80495   
13:21:31 19/20        18       18       90000/100000       10000          0        0.0        0.0      n/a        80782    80782   
13:24:33 19/20        18       18       90000/100000       10000          0        0.0        0.0      n/a        77706    77706   
13:27:34 19/20        17       17       90000/100000       10000          0        0.0        0.0      n/a        68591    68591   
13:30:45 19/20        12       12       90000/100000       10000          0        0.0        0.0      n/a        56168    56168   
13:33:49 19/20        1        1        90000/100000       10000          0        0.0        0.0      n/a        22738    22738   
13:36:50 19/20        1        1        90000/100000       10000          0        0.0        0.0      n/a        11840    11840   
13:39:51 19/20        1        1        90000/100000       10000          0        0.0        0.0      n/a        11840    11840   
13:42:51 19/20        1        1        90000/100000       10000          0        0.0        0.0      n/a        11840    11840   
13:45:52 19/20        1        1        90000/100000       10000          0        0.0        0.0      n/a        11840    11840   
13:48:49 19/20        1        1        90000/100000       10000          0        0.0        0.0      n/a        7841     7841    
^C
Received INT, stopping local submissions and aborting discovered runs.
13:49:48                                                                  0        0.0        n/a      n/a                         

Aggregate Summary
  runs_discovered: /20
  total_expected_children: 100000
  child_run_roots_terminal: /
  peak_parent_live: 19
  peak_parent_running: 19
  children_seen: /100000
  succeeded: 
  failed: 
  aborted: 
  timed_out: 
  peak_seen: 90000/100000
  peak_running: 80782
  peak_active: 80782
  peak_create_rps: 206
  first_run_discovered: 00h:00m:08s
  aggregate_first_running: 00h:00m:40s
  aggregate_all_visible: n/a
  aggregate_terminal: 01h:35m:09s
  total_elapsed: 01h:35m:09s

Signed-off-by: pmahindrakar-oss <[email protected]>
Signed-off-by: pmahindrakar-oss <[email protected]>
Signed-off-by: pmahindrakar-oss <[email protected]>
Signed-off-by: pmahindrakar-oss <[email protected]>
Signed-off-by: pmahindrakar-oss <[email protected]>
Signed-off-by: pmahindrakar-oss <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant