feat: Deployment for EPP as a static library (#3314)

Signed-off-by: Anna Tchernych <[email protected]>

Author: atchernych

deploy/inference-gateway/README.md

@@ -5,7 +5,7 @@ This guide demonstrates two setups.
 - The basic setup treats each Dynamo deployment as a black box and routes traffic randomly among the deployments.
 - The EPP-aware setup uses a custom Dynamo plugin `dyn-kv` to pick the best worker.
 
-EPP’s default approach is token-aware only `by approximation` because it relies on the non-tokenized text in the prompt. But the Dynamo plugin uses a token-aware KV algorithm. It employs the dynamo router which implements kv routing by running your model’s tokenizer inline. The EPP plugin configuration lives in [`helm/dynamo-gaie/epp-config-dynamo.yaml`](helm/dynamo-gaie/epp-config-dynamo.yaml) per EPP [convention](https://gateway-api-inference-extension.sigs.k8s.io/guides/epp-configuration/config-text/).
+EPP’s default kv-routing approach is token-aware only `by approximation` because the prompt is tokenized with a generic tokenizer unaware of the model deployed. But the Dynamo plugin uses a token-aware KV algorithm. It employs the dynamo router which implements kv routing by running your model’s tokenizer inline. The EPP plugin configuration lives in [`helm/dynamo-gaie/epp-config-dynamo.yaml`](helm/dynamo-gaie/epp-config-dynamo.yaml) per EPP [convention](https://gateway-api-inference-extension.sigs.k8s.io/guides/epp-configuration/config-text/).
 
 Currently, these setups are only supported with the kGateway based Inference Gateway.
 
@@ -87,6 +87,28 @@ kubectl apply -f agg.yaml -n my-model
 ```
 Take a note of or change the DYNAMO_IMAGE in the model deployment file.
 
+
+Do not forget docker registry secret if needed.
+```bash
+kubectl create secret docker-registry docker-imagepullsecret \
+  --docker-server=$DOCKER_SERVER \
+  --docker-username=$DOCKER_USERNAME \
+  --docker-password=$DOCKER_PASSWORD \
+  --namespace=$NAMESPACE
+```
+
+Do not forget to include the the HuggingFace token if required.
+```bash
+export HF_TOKEN=your_hf_token
+kubectl create secret generic hf-token-secret \
+  --from-literal=HF_TOKEN=${HF_TOKEN} \
+  -n ${NAMESPACE}
+```
+
+Create a model configuration file similar to the vllm_agg_qwen.yaml for you model.
+This file demonstrates the values needed for the Vllm Agg setup in [agg.yaml](../../components/backends/vllm/deploy/agg.yaml)
+Take a note of the model's block size provided in the model card.
+
 ### 4. Install Dynamo GAIE helm chart ###
 
 The Inference Gateway is configured through the `inference-gateway-resources.yaml` file.
@@ -95,7 +117,7 @@ Deploy the Inference Gateway resources to your Kubernetes cluster by running one
 
 #### Basic Black Box Integration ####
 
-For the basic black box integration run:
+The basic black box integration uses a standard EPP image`us-central1-docker.pkg.dev/k8s-staging-images/gateway-api-inference-extension/epp:v0.4.0`. For the basic black box integration run:
 
 ```bash
 cd deploy/inference-gateway
@@ -104,9 +126,13 @@ helm install dynamo-gaie ./helm/dynamo-gaie -n my-model -f ./vllm_agg_qwen.yaml
 
 #### EPP-aware Integration with the custom Dynamo Plugin ####
 
+Dynamo provides a custom routing plugin `pkg/epp/scheduling/plugins/dynamo_kv_scorer/plugin.go` to perform efficient kv routing.
+The Dynamo router is built as a static library, the EPP router will call to provide fast inference.
+You can either use the image `nvcr.io/nvstaging/ai-dynamo/epp-inference-extension-dynamo:v0.6.0-1` for the EPP_IMAGE in the Helm deployment command and proceed to the step 2 or you can build the image yourself following the steps below.
+
 ##### 1. Build the custom EPP image #####
 
-We provide git patches for you to use.
+If you choose to build your own image use the steps below. Proceed to step 2 otherwise to deploy with Helm.
 
 ##### 1.1 Clone the official GAIE repo in a separate folder #####
 
@@ -116,44 +142,74 @@ cd gateway-api-inference-extension
 git checkout v0.5.1
 ```
 
-##### 1.2 Apply patch(es) #####
+##### 1.2 Build the Dynamo Custom EPP #####
+
+
+
+###### 1.2.1 Clone the official EPP repo ######
+
+```bash
+# Clone the official GAIE repo in a separate folder
+cd path/to/gateway-api-inference-extension
+git clone [email protected]:kubernetes-sigs/gateway-api-inference-extension.git
+git checkout v0.5.1
+```
+
+###### 1.2.2 Run the script to build the EPP image ######
+
+The script will apply a custom patch to the code with your GAIE repo and build the image for you to use.
 
 ```bash
-git apply <dynamo-folder>/deploy/inference-gateway/epp-patches/v0.5.1-1/epp-v0.5.1-dyn1.patch
+# Use your custom paths
+export DYNAMO_DIR=/path/to/dynamo
+export EPP_DIR=/path/to/gateway-api-inference-extension
+
+# Run the script
+cd deploy/inference-gateway
+./build-epp-dynamo.sh
 ```
 
-##### 1.3 Build the custom EPP image #####
+Under the hood the script applies the Dynamo Patch to the EPP code base; creates a Dynamo Router static library and builds a custom EPP image with it.
+Re-tag the freshly built image and push it to your registry.
 
 ```bash
-# Build the image <your-docker-registry/dynamo-custom-epp:<your-tag> and then manually push
-make image-local-load \
-  IMAGE_REGISTRY=<your-docker-registry> \
-  IMAGE_NAME=dynamo-custom-epp \
-  EXTRA_TAG=<your-tag>
-
-# Or run the command below to build push to your registry
-make image-local-push \
-  IMAGE_REGISTRY=<your-docker-registry> \
-  IMAGE_NAME=dynamo-custom-epp \
-  EXTRA_TAG=<your-tag>
+docker images
+docker tag <your-new-id> <your-image-tag>
+docker push  <your-image-tag>
 ```
 
-##### 2. Install through helm #####
+##### 2. Deploy through helm #####
 
 ```bash
 cd deploy/inference-gateway
 
 # Export the Dynamo image you have used when deploying your model in Step 3.
 export DYNAMO_IMAGE=<the-dynamo-image-you-have-used-when-deploying-the-model>
-export EPP_IMAGE=<the-epp-image-you-built>  # i.e. docker.io/lambda108/epp-inference-extension-dynamo:v0.5.1-1
+# Export the image tag you have used when building the EPP i.e. docker.io/lambda108/epp-inference-extension-dynamo:v0.5.1-2
+export EPP_IMAGE=<the-epp-image-you-built>
+```
 
+**Configuration**
+You can configure the plugin by setting environment vars in your [values-epp-aware.yaml].
+- Overwrite the `DYNAMO_NAMESPACE` env var if needed to match your model's dynamo namespace.
+- Set `DYNAMO_BUSY_THRESHOLD` to configure the upper bound on how “full” a worker can be (often derived from kv_active_blocks or other load metrics) before the router skips it. If the selected worker exceeds this value, routing falls back to the next best candidate. By default the value is negative meaning this is not enabled.
+- Set `DYNAMO_ROUTER_REPLICA_SYNC=true` to enable a background watcher to keep multiple router instances in sync (important if you run more than one KV router per component).
+- By default the Dynamo plugin uses KV routing. You can expose `DYNAMO_USE_KV_ROUTING=false`  in your [values-epp-aware.yaml] if you prefer to route in the round-robin fashion.
+- If using kv-routing:
+  - Overwrite the `DYNAMO_KV_BLOCK_SIZE` in your [values-epp-aware.yaml](./values-epp-aware.yaml) to match your model's block size.The `DYNAMO_KV_BLOCK_SIZE` env var is ***MANDATORY*** to prevent silent KV routing failures.
+  - Set `DYNAMO_OVERLAP_SCORE_WEIGHT` to weigh how heavily the score uses token overlap (predicted KV cache hits) versus other factors (load, historical hit rate). Higher weight biases toward reusing workers with similar cached prefixes.
+  - Set `DYNAMO_ROUTER_TEMPERATURE` to soften or sharpen the selection curve when combining scores. Low temperature makes the router pick the top candidate deterministically; higher temperature lets lower-scoring workers through more often (exploration).
+  - Set `DYNAMO_USE_KV_EVENTS=false` if you want to disable KV event tracking while using kv-routing
+  - See the [KV cache routing design](../../docs/architecture/kv_cache_routing.md) for details.
+
+
+```bash
 helm upgrade --install dynamo-gaie ./helm/dynamo-gaie \
   -n my-model \
   -f ./vllm_agg_qwen.yaml \
   -f ./values-epp-aware.yaml \
   --set eppAware.enabled=true \
-  --set-string eppAware.eppImage=$EPP_IMAGE \
-  --set-string eppAware.sidecar.image=$DYNAMO_IMAGE
+  --set-string eppAware.eppImage=$EPP_IMAGE
 ```
 
 
@@ -162,6 +218,7 @@ Key configurations include:
 - A service for the inference gateway
 - Required RBAC roles and bindings
 - RBAC permissions
+- values-epp-aware.yaml sets eppAware.dynamoNamespace=vllm-agg for the bundled example. Point it at your actual Dynamo namespace by editing that file or adding --set eppAware.dynamoNamespace=<namespace> (and likewise for dynamoComponent, dynamoKvBlockSize if they differ).
 
 ### 5. Verify Installation ###
 

deploy/inference-gateway/build-epp-dynamo.sh

@@ -0,0 +1,95 @@
+#!/usr/bin/env bash
+
+# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+set -e  # Exit on any error
+
+# Configuration - Set these environment variables before running
+if [[ -z "${DYNAMO_DIR}" ]]; then
+    echo "DYNAMO_DIR environment variable must be set"
+    echo "   Example: export DYNAMO_DIR=/path/to/dynamo"
+    exit 1
+fi
+
+if [[ -z "${EPP_DIR}" ]]; then
+    echo "EPP_DIR environment variable must be set"
+    echo "   Example: export EPP_DIR=/path/to/gateway-api-inference-extension-dynamo"
+    exit 1
+fi
+DYNAMO_LIB_DIR="${EPP_DIR}/pkg/epp/scheduling/plugins/dynamo_kv_scorer/lib"
+DYNAMO_INCLUDE_DIR="${EPP_DIR}/pkg/epp/scheduling/plugins/dynamo_kv_scorer/include"
+
+echo "🏗️  Building Dynamo KV Router C Library..."
+
+# Step 1: Build the static library
+echo "📦 Building static library..."
+cd "${DYNAMO_DIR}"
+cargo build --release -p libdynamo_llm
+
+# Step 2: Generate header file (with fallback)
+echo "📝 Generating C header..."
+HEADER_OUTPUT="${DYNAMO_DIR}/lib/bindings/c/include/nvidia/dynamo_llm/llm_engine.h"
+
+if ! cbindgen --config lib/bindings/c/cbindgen.toml --crate libdynamo_llm --output "${HEADER_OUTPUT}"; then
+    echo "cbindgen failed, using fallback header..."
+    cp "${DYNAMO_DIR}/lib/bindings/c/src/fallback_header.h" "${HEADER_OUTPUT}"
+fi
+
+# Step 3: Ensure EPP directories exist
+echo "Preparing EPP directories..."
+mkdir -p "${DYNAMO_LIB_DIR}"
+mkdir -p "${DYNAMO_INCLUDE_DIR}"
+
+# Step 4: Copy files to EPP
+echo "Copying files to EPP..."
+cp "${HEADER_OUTPUT}" "${DYNAMO_INCLUDE_DIR}/"
+cp "${DYNAMO_DIR}/target/release/libdynamo_llm_capi.a" "${DYNAMO_LIB_DIR}/"
+
+# Verify files were copied
+if [[ ! -f "${DYNAMO_INCLUDE_DIR}/llm_engine.h" ]]; then
+    echo "Header file copy failed!"
+    exit 1
+fi
+
+if [[ ! -f "${DYNAMO_LIB_DIR}/libdynamo_llm_capi.a" ]]; then
+    echo "Library file copy failed!"
+    exit 1
+fi
+
+echo "Files copied successfully:"
+echo "   Header: ${DYNAMO_INCLUDE_DIR}/llm_engine.h"
+echo "   Library: ${DYNAMO_LIB_DIR}/libdynamo_llm_capi.a"
+
+# Step 5: Apply Dynamo patch (if it exists)
+echo "🔧 Applying Dynamo patch..."
+cd "${EPP_DIR}"
+
+PATCH_FILE="${DYNAMO_DIR}/deploy/inference-gateway/epp-patches/v0.5.1-2/epp-v0.5.1-dyn2.patch"
+if [[ -f "${PATCH_FILE}" ]]; then
+    if git apply --check "${PATCH_FILE}" 2>/dev/null; then
+        git apply "${PATCH_FILE}"
+        echo "Patch applied successfully"
+    else
+        echo "Patch doesn't apply cleanly - may already be applied or need manual resolution"
+    fi
+else
+    echo "No patch file found at ${PATCH_FILE}"
+fi
+
+# Step 6: Build the EPP image
+echo "Building the EPP image..."
+make dynamo-image-local-load
+
+echo "EPP with Dynamo KV routing built"