model-serving: build the plugin chain once instead of per role replica

[Abhayanthk]

What would you like to be added:

Build the ModelServing plugin chain once and reuse it for every pod created in the same pass, as the TODO in CreatePodsByRole suggests:

kthena/pkg/model-serving-controller/controller/model_serving_controller.go

Lines 2723 to 2730 in f9b54e9

	func (c *ModelServingController) CreatePodsByRole(ctx context.Context, role workloadv1alpha1.Role, ms *workloadv1alpha1.ModelServing, roleIndex int, servingGroupOrdinal int, revision string, roleTemplateHash string) error {
	servingGroupName := utils.GenerateServingGroupName(ms.Name, servingGroupOrdinal)
	// TODO(hzxuzhonghu): build the plugin chain only once per ModelServing
	// This is not critical now, so we leave it for future optimization.
	chain, err := c.buildPluginChain(ms)
	if err != nil {
	return fmt.Errorf("build plugin chain: %w", err)
	}

Today CreatePodsByRole calls buildPluginChain on every call. Its callers invoke it inside loops, once per role replica:

• CreatePodsForServingGroup
• scaleUpRoles
• manageRoleReplicasPerGroup

Proposed change: build the chain once in each caller and pass it into CreatePodsByRole as a parameter. Behaviour stays the same, and unit tests would check that the chain is built once per pass and still applied to entry and worker pods.

Why is this needed:

When spec.plugins is set, plugins.NewChain runs every plugin factory again for each role replica, so a ServingGroup with N role replicas builds the same chain N times in one reconcile. When no plugins are configured, buildPluginChain returns early, so the cost only applies to ModelServings that use plugins.

@hzxuzhonghu, is this still wanted? If so, I'd like to work on it. One question first: is a chain safe to reuse across pods within one reconcile, or do some plugins keep per-pod state?

[Abhayanthk]

/assign

[Abhayanthk]

The approach I propose is to build the chain once in the callers and pass it to CreatePodsByRole. We only need to build it in CreatePodsForServingGroup and manageRoleReplicasPerGroup (scaleUpRoles is covered by manageRoleReplicasPerGroup).

This is only safe while our plugins stay stateless. Both current plugins (LWSLabelsPlugin, DemoPlugin) are, so future plugins would need to follow the same rule or keep per-pod data in HookRequest.

I also considered caching the chain on the controller and updating it every time the plugins change, but it makes things unnecessarily complex.

handleReadyPod also builds the chain per ready pod, but it handles one pod per call, so only a cache would help there. I'd leave it as is.

Does this sound good? If so, I'll open a PR.