Envoy AI Gateway

Tetrate and Bloomberg engineers initiated Envoy AI Gateway under the Cloud Native Computing Foundation to bring standard Envoy Gateway traffic management to generative AI workloads. Designed to eliminate disparate vendor SDKs across microservices, it translates OpenAI API calls to backends like Anthropic, Bedrock, and Gemini while enforcing network policies locally. Running on Envoy Proxy with an external processing filter, it lets infrastructure teams control prompt traffic without routing credentials through third-party SaaS vendors.

Why Envoy AI Gateway instead of Kong AI Gateway

Kong AI Gateway handles LLM traffic through plugins on its OpenResty architecture, but enterprise governance often requires proprietary commercial tiers or heavy supporting services. Envoy AI Gateway integrates directly into Kubernetes as an Envoy Gateway extension, using an external processor (ExtProc) to inspect prompts, enforce token budgets, and route requests inside your private VPC. For teams with strict security mandates, this topology ensures that raw prompts and context embeddings never traverse external networks before reaching the designated model provider.

Control also extends to credential isolation and operations. Kong manages API keys across its control plane, whereas Envoy AI Gateway maps internal client tokens to upstream provider credentials stored in native Kubernetes secrets. Client pods query a single local endpoint using generic tokens, isolated from real upstream credentials. Because routing policies use standard Kubernetes Gateway API resources, DevOps teams audit and version traffic rules through existing GitOps workflows without vendor lock-in.

What it does well

• Translates unified OpenAI-compatible requests to Anthropic, AWS Bedrock, Google Vertex, Mistral, and local inference engines.
• Isolates API credentials by mapping internal tokens to upstream keys stored in Kubernetes secrets, preventing secret exposure in application pods.
• Enforces token-aware rate limits alongside request quotas at the proxy layer, preventing unexpected provider invoice spikes.
• Supports two-tier gateway topologies, separating edge authentication from localized backend inference routing and endpoint picking.
• Exports OpenTelemetry traces and metrics adhering to OpenInference standards, tracking request latency, token consumption, and error rates.