Component
Setup/Standup
Desired use case or feature
A major value aspect of WVA is in multi-variants scenarios (there the model-scope optimization is a differentiator compare to KEDA/HPA based auto-scaling). We need to have the ability to run the existing scenarios (as prefill_heavy) and more in multi-variants setup (can start with two-variants). Will need a simple way of defining the variants (hardware, parallelism, cost etc.) run the benchmarks and canalize the results (additional per-variant graphs etc,).
Proposed solution
An experimental benchmark setup and run is described here
https://github.com/biranofer/workload-variant-autoscaler/blob/feat/two-variant-wva-benchmark/docs/developer-guide/two-variant-wva-benchmark.md, here is a resulted graph from a cost-differentiated two variants (primary and v2) run, using the prefill_heavy scenario:
The example implementation matching the design above is in the two-variant-wva-benchmark branch at my fork
Alternatives
No response
Additional context or screenshots
No response
Component
Setup/Standup
Desired use case or feature
A major value aspect of WVA is in multi-variants scenarios (there the model-scope optimization is a differentiator compare to KEDA/HPA based auto-scaling). We need to have the ability to run the existing scenarios (as prefill_heavy) and more in multi-variants setup (can start with two-variants). Will need a simple way of defining the variants (hardware, parallelism, cost etc.) run the benchmarks and canalize the results (additional per-variant graphs etc,).
Proposed solution
An experimental benchmark setup and run is described here
https://github.com/biranofer/workload-variant-autoscaler/blob/feat/two-variant-wva-benchmark/docs/developer-guide/two-variant-wva-benchmark.md, here is a resulted graph from a cost-differentiated two variants (primary and v2) run, using the prefill_heavy scenario:
The example implementation matching the design above is in the two-variant-wva-benchmark branch at my fork
Alternatives
No response
Additional context or screenshots
No response