Auto-configure prefix-cache-scorer parameters from engine metrics #1629

learner0810 · 2025-09-22T07:44:55Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

Auto-configure prefix-cache-scorer parameters from engine metrics

Which issue(s) this PR fixes:

Fixes # partial fix: #1512

Does this PR introduce a user-facing change?:

netlify · 2025-09-22T07:45:01Z

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	`dc8c13c`
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/68edc603edf97b0008b32014
😎 Deploy Preview	https://deploy-preview-1629--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

k8s-ci-robot · 2025-09-22T07:45:06Z

Hi @learner0810. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ahg-g · 2025-09-22T22:02:30Z

/assign @liu-cong
/ok-to-test

ahg-g · 2025-09-22T22:02:48Z

@learner0810 do we have a similar metric in sglang?

learner0810 · 2025-09-23T02:53:31Z

@learner0810 do we have a similar metric in sglang?

I looked up the sglang documentation and didn't find it, so it probably isn't there yet.

learner0810 · 2025-09-25T02:25:44Z

cc @liu-cong PTAL

liu-cong · 2025-09-25T02:51:51Z

pkg/epp/datalayer/metrics/extractor.go

 	LoraInfoWaitingAdaptersMetricName = "waiting_lora_adapters"
 	LoraInfoMaxAdaptersMetricName     = "max_lora"
+
+	CacheConfigBlockSizeInfoMetricName = "block_size"


do you plan to add the cache size metric as well? maybe in a follow up PR? That's a much more useful metric than the block size. In fact, the block size doesn't necessarily need to be the same with vllm

Yes, If IGW requires cache size metric, I'd be happy to add that in a subsequent pull request.

liu-cong · 2025-09-25T02:54:38Z

cc @elevran

What's the status of the pluggable data layer? Can we use that to retrieve this new metric(s)?

elevran · 2025-10-10T06:14:20Z

@liu-cong I was on PTO the past 2w.
The new data layer it is still in experimental state and I'll be working to test and enable it over the next few weeks.
The metrics Collector provides access to the entire info on the /metrics endpoint. One would need to write an Extractor to retrieve the configuration parameters and store them as an Attribute on the endpoint and then register it with the Collector.

learner0810 · 2025-10-10T06:24:56Z

@liu-cong I was on PTO the past 2w. The new data layer it is still in experimental state and I'll be working to test and enable it over the next few weeks. The metrics Collector provides access to the entire info on the /metrics endpoint. One would need to write an Extractor to retrieve the configuration parameters and store them as an Attribute on the endpoint and then register it with the Collector.

Excuse me, could you please advise me? What else should I change in my PR?

pkg/epp/scheduling/framework/plugins/multi/prefix/plugin.go

zetxqx · 2025-10-13T21:12:33Z

@liu-cong I was on PTO the past 2w. The new data layer it is still in experimental state and I'll be working to test and enable it over the next few weeks. The metrics Collector provides access to the entire info on the /metrics endpoint. One would need to write an Extractor to retrieve the configuration parameters and store them as an Attribute on the endpoint and then register it with the Collector.

Excuse me, could you please advise me? What else should I change in my PR?

The other approach for data layer extracting metrics should start from here. ref. But since it's not enabled, do you think we can merge this using the v1 approach first? @liu-cong @elevran And we can do the migration in another PR.

liu-cong

/lgtm

The implementation adds to the metric code paths of the legacy and new data layer, which is fine until we fully migrate to the new data layer.

pkg/epp/scheduling/framework/plugins/multi/prefix/plugin.go

learner0810 · 2025-10-14T04:06:42Z

cc @liu-cong All tests are passing, PATL

liu-cong

/lgtm

kfswain · 2025-10-14T21:21:41Z

/approve

k8s-ci-robot · 2025-10-14T21:21:49Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kfswain, learner0810

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [kfswain]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added the kind/feature Categorizes issue or PR as related to a new feature. label Sep 22, 2025

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Sep 22, 2025

k8s-ci-robot requested review from ahg-g and kfswain September 22, 2025 07:45

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Sep 22, 2025

k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Sep 22, 2025

k8s-ci-robot assigned liu-cong Sep 22, 2025

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Sep 22, 2025

learner0810 force-pushed the auto-configure-prefix-cache-scorer-parameters branch from 76971e1 to d50488a Compare September 23, 2025 02:10

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Sep 23, 2025

liu-cong reviewed Sep 25, 2025

View reviewed changes

zetxqx reviewed Oct 13, 2025

View reviewed changes

pkg/epp/scheduling/framework/plugins/multi/prefix/plugin.go Outdated Show resolved Hide resolved

pkg/epp/scheduling/framework/plugins/multi/prefix/plugin.go Outdated Show resolved Hide resolved

liu-cong reviewed Oct 13, 2025

View reviewed changes

pkg/epp/scheduling/framework/plugins/multi/prefix/plugin.go Outdated Show resolved Hide resolved

pkg/epp/scheduling/framework/plugins/multi/prefix/plugin.go Outdated Show resolved Hide resolved

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 13, 2025

learner0810 force-pushed the auto-configure-prefix-cache-scorer-parameters branch from d50488a to 93882e5 Compare October 14, 2025 02:33

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 14, 2025

learner0810 force-pushed the auto-configure-prefix-cache-scorer-parameters branch 2 times, most recently from 72e001e to 20ec337 Compare October 14, 2025 03:05

liu-cong reviewed Oct 14, 2025

View reviewed changes

pkg/epp/scheduling/framework/plugins/multi/prefix/plugin.go Outdated Show resolved Hide resolved

Auto-configure prefix-cache-scorer parameters from engine metrics

dc8c13c

learner0810 force-pushed the auto-configure-prefix-cache-scorer-parameters branch from 20ec337 to dc8c13c Compare October 14, 2025 03:39

liu-cong reviewed Oct 14, 2025

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 14, 2025

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 14, 2025

k8s-ci-robot merged commit e441c89 into kubernetes-sigs:main Oct 14, 2025
11 checks passed

Auto-configure prefix-cache-scorer parameters from engine metrics #1629

Auto-configure prefix-cache-scorer parameters from engine metrics #1629

Uh oh!

Conversation

learner0810 commented Sep 22, 2025

Uh oh!

netlify bot commented Sep 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for gateway-api-inference-extension ready!

Uh oh!

k8s-ci-robot commented Sep 22, 2025

Uh oh!

ahg-g commented Sep 22, 2025

Uh oh!

ahg-g commented Sep 22, 2025

Uh oh!

learner0810 commented Sep 23, 2025

Uh oh!

learner0810 commented Sep 25, 2025

Uh oh!

liu-cong Sep 25, 2025

Choose a reason for hiding this comment

Uh oh!

learner0810 Sep 25, 2025

Choose a reason for hiding this comment

Uh oh!

liu-cong commented Sep 25, 2025

Uh oh!

elevran commented Oct 10, 2025

Uh oh!

learner0810 commented Oct 10, 2025

Uh oh!

Uh oh!

Uh oh!

zetxqx commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

liu-cong left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

learner0810 commented Oct 14, 2025

Uh oh!

liu-cong left a comment

Choose a reason for hiding this comment

Uh oh!

kfswain commented Oct 14, 2025

Uh oh!

k8s-ci-robot commented Oct 14, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

netlify bot commented Sep 22, 2025 •

edited

Loading

zetxqx commented Oct 13, 2025 •

edited

Loading