Version: Latest

Deploy with Envoy AI Gateway

Use this topology when Envoy AI Gateway already owns north-south traffic and provider integration. Semantic Router can either choose a model from the request's meaning or run only as an ExtProc service for OpenAI Responses state and protocol conversion. Envoy AI Gateway remains responsible for Gateway API resources, provider credentials, rate limits, and traffic policy.

For large request bodies or streamed immediate responses from Semantic Router, also see Streamed ExtProc and immediate responses. That guide shows how to switch the ExtProc filter from BUFFERED to STREAMED request bodies and how streamed Chat Completions clients receive looper or fast_response immediate responses.

Responsibility split

The deployment consists of:

Semantic Router evaluates the selected recipe and chooses the logical model or provider alias.
Envoy Gateway provides the Kubernetes Gateway API data plane.
Envoy AI Gateway translates provider APIs and applies gateway-owned authentication, rate limiting, and traffic policy.
Model providers serve the selected model. This guide uses a demo backend; it does not install production inference capacity.

Provider support changes independently of Semantic Router. Use the Envoy AI Gateway provider documentation to choose an AIServiceBackend and credential policy, then bind the provider names to the aliases used by your Semantic Router configuration.

Responses state without model selection

Use the responses-state.yaml profile when the external gateway already selects the route and backend, but clients need Semantic Router's OpenAI Responses implementation. The profile enables POST /v1/responses, GET /v1/responses/{id}, and previous_response_id conversation chaining. It converts Responses requests to the declared upstream wire format and converts upstream responses back to Responses objects.

The ownership boundary is:

Concern	Owner
Public listener, Gateway/route, provider backend, credentials, rate limits, and traffic policy	External gateway
Responses object storage, `previous_response_id` expansion, and request/response protocol conversion	Semantic Router ExtProc

This profile deliberately has no Semantic Router listener, backend reference, credential, routing signal, decision, or optional plugin. Model entries are protocol metadata only. Keep their names and api_format values aligned with the models accepted by the external gateway. Semantic Router rewrites the upstream API path as part of protocol conversion without clearing Envoy's route cache, so the route selected by the gateway remains authoritative.

Install the chart with the state-only profile in place of the model-selection values used in Step 2:

helm install semantic-router oci://ghcr.io/vllm-project/charts/semantic-router \
  --version 0.0.0-latest \
  --namespace vllm-semantic-router-system \
  --create-namespace \
  -f https://raw.githubusercontent.com/vllm-project/semantic-router/refs/heads/main/deploy/kubernetes/ai-gateway/semantic-router-values/responses-state.yaml

The included in-memory response store is intended for a single-replica demo or local validation; state is lost on restart. Configure the Response API Redis backend for durable or replicated production deployments. The external gateway's ExtProc filter must send buffered request and response headers and bodies to Semantic Router, and its route for /v1/responses must select the intended provider backend before request-body conversion runs.

Prerequisites

You need:

Kubernetes 1.32 or later for the pinned Envoy AI Gateway v1.0.x and Envoy Gateway v1.8.x compatibility set; kind is sufficient for the demo;
Gateway API v1.5.x CRDs. The default Envoy Gateway Helm installation below installs a compatible set; if your platform owns those CRDs, verify its versions before installing the chart;
kubectl;
Helm; and
credentials and network access for every provider used by your config.

Step 1: Create Kind Cluster (Optional)

Create a local Kubernetes cluster optimized for the semantic router workload:

kind create cluster --name semantic-router-cluster

# Verify cluster is ready
kubectl wait --for=condition=Ready nodes --all --timeout=300s

Step 2: Deploy vLLM Semantic Router

Deploy Semantic Router with the integration values:

# Install with custom values from GHCR OCI registry
# (Optional) If you use a registry mirror/proxy, append: --set global.imageRegistry=<your-registry>
helm install semantic-router oci://ghcr.io/vllm-project/charts/semantic-router \
  --version 0.0.0-latest \
  --namespace vllm-semantic-router-system \
  --create-namespace \
  -f https://raw.githubusercontent.com/vllm-project/semantic-router/refs/heads/main/deploy/kubernetes/ai-gateway/semantic-router-values/values.yaml

# Wait for deployment to be ready (this may take several minutes for model downloads)
kubectl wait --for=condition=Available deployment/semantic-router -n vllm-semantic-router-system --timeout=600s

# Verify deployment status
kubectl get pods -n vllm-semantic-router-system

Note: The values file contains the Semantic Router model bindings, signals, decisions, and routing rules. Download and review values.yaml before adapting it to a real provider pool.

Step 3: Install Envoy Gateway

Install the Envoy Gateway release supported by Envoy AI Gateway v1.0.0:

export AIGW_VERSION=v1.0.0
export ENVOY_GATEWAY_VERSION=v1.8.1

helm upgrade -i eg oci://docker.io/envoyproxy/gateway-helm \
  --version "${ENVOY_GATEWAY_VERSION}" \
  --namespace envoy-gateway-system \
  --create-namespace \
  -f "https://raw.githubusercontent.com/envoyproxy/ai-gateway/${AIGW_VERSION}/manifests/envoy-gateway-values.yaml"

kubectl wait --timeout=2m -n envoy-gateway-system deployment/envoy-gateway --for=condition=Available

Step 4: Install Envoy AI Gateway

Install the AI Gateway CRDs before the controller. These versions follow the upstream v1.0.x compatibility matrix.

# Install Envoy AI Gateway CRDs
helm upgrade -i aieg-crd oci://docker.io/envoyproxy/ai-gateway-crds-helm \
  --version "${AIGW_VERSION}" \
  --namespace envoy-ai-gateway-system \
  --create-namespace

# Install the controller
helm upgrade -i aieg oci://docker.io/envoyproxy/ai-gateway-helm \
  --version "${AIGW_VERSION}" \
  --namespace envoy-ai-gateway-system

# Wait for AI Gateway Controller to be ready
kubectl wait --timeout=300s -n envoy-ai-gateway-system deployment/ai-gateway-controller --for=condition=Available

Step 5: Deploy Demo LLM

Create a demo LLM to serve as the backend for the semantic router:

# Deploy demo LLM
kubectl apply -f https://raw.githubusercontent.com/vllm-project/semantic-router/refs/heads/main/deploy/kubernetes/ai-gateway/aigw-resources/base-model.yaml

Step 6: Create Gateway API Resources

Create the necessary Gateway API resources for the AI gateway:

kubectl apply -f https://raw.githubusercontent.com/vllm-project/semantic-router/refs/heads/main/deploy/kubernetes/ai-gateway/aigw-resources/gwapi-resources.yaml

Testing the Deployment

Method 1: Port Forwarding (Recommended for Local Testing)

Set up port forwarding to access the gateway locally:

# Get the Envoy service name
export ENVOY_SERVICE=$(kubectl get svc -n envoy-gateway-system \
  --selector=gateway.envoyproxy.io/owning-gateway-namespace=default,gateway.envoyproxy.io/owning-gateway-name=semantic-router \
  -o jsonpath='{.items[0].metadata.name}')

kubectl port-forward -n envoy-gateway-system svc/$ENVOY_SERVICE 8080:80

Send Test Requests

Once the gateway is accessible, test the inference endpoint:

# Test math domain chat completions endpoint
curl -i -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "vllm-sr/auto",
    "messages": [
      {"role": "user", "content": "What is the derivative of f(x) = x^3?"}
    ]
  }'

Troubleshooting

Common Issues

Gateway not accessible:

# Check gateway status
kubectl get gateway semantic-router -n default

# Check Envoy service
kubectl get svc -n envoy-gateway-system

AI Gateway controller not ready:

# Check AI gateway controller logs
kubectl logs -n envoy-ai-gateway-system deployment/ai-gateway-controller

# Check controller status
kubectl get deployment -n envoy-ai-gateway-system

Semantic router not responding:

# Check semantic router pod status
kubectl get pods -n vllm-semantic-router-system

# Check semantic router logs
kubectl logs -n vllm-semantic-router-system deployment/semantic-router

Cleanup

To remove the entire deployment:

# Remove Gateway API resources and Demo LLM
kubectl delete -f https://raw.githubusercontent.com/vllm-project/semantic-router/refs/heads/main/deploy/kubernetes/ai-gateway/aigw-resources/gwapi-resources.yaml
kubectl delete -f https://raw.githubusercontent.com/vllm-project/semantic-router/refs/heads/main/deploy/kubernetes/ai-gateway/aigw-resources/base-model.yaml

# Remove semantic router
helm uninstall semantic-router -n vllm-semantic-router-system

# Remove AI gateway
helm uninstall aieg -n envoy-ai-gateway-system
helm uninstall aieg-crd -n envoy-ai-gateway-system

# Remove Envoy gateway
helm uninstall eg -n envoy-gateway-system

# Delete kind cluster (optional)
kind delete cluster --name semantic-router-cluster

Next Steps

Replace the demo backend with the provider resources and credentials owned by your gateway team.
Keep the model aliases in AIGatewayRoute aligned with the names emitted by Semantic Router.
Add authentication, rate limits, observability, and capacity policy at their owning layers before exposing the gateway.

Responsibility split​

Responses state without model selection​

Prerequisites​

Step 1: Create Kind Cluster (Optional)​

Step 2: Deploy vLLM Semantic Router​

Step 3: Install Envoy Gateway​

Step 4: Install Envoy AI Gateway​

Step 5: Deploy Demo LLM​

Step 6: Create Gateway API Resources​

Testing the Deployment​

Method 1: Port Forwarding (Recommended for Local Testing)​

Send Test Requests​

Troubleshooting​

Common Issues​

Cleanup​

Next Steps​