LLM Gateway

LLM Gateway is built on a TypeScript and Hono foundation that routes requests across more than thirty-five machine learning providers behind a single OpenAI-compatible HTTP interface. The proxy runs as a self-hosted Linux container bundling PostgreSQL and Redis, allowing engineering teams to manage model keys, fallback rules, and cost tracking on their own infrastructure. It handles chat completions, streaming responses, and document OCR while keeping raw prompt payloads off external servers.

Why LLM Gateway instead of Vercel AI Gateway

Vercel AI Gateway forwards prompts, completions, and metadata through a multi-tenant edge network. For teams subject to GDPR, healthcare privacy mandates, or internal security policies, routing sensitive payloads through a hosted intermediary creates compliance roadblocks. LLM Gateway terminates client traffic inside a private VPC or data center, keeping prompts, completions, and API keys confined to internal networks.

Vercel provides automated edge scaling across regions. LLM Gateway trades managed simplicity for infrastructure control: your team owns the underlying PostgreSQL database, Redis cache, and audit trails. You can configure custom rate limits, enforce internal retention rules, and query local models without paying per-seat subscription fees or exposing tokens to third-party proxies.

What it does well

• Unified protocol translation: Maps OpenAI chat completions and Anthropic Messages requests across dozens of providers without altering client SDK code.
• Local response caching: Stores repeated prompt completions in Redis to eliminate duplicate upstream API charges and lower latency.
• Provider failover: Automatically reroutes failed or rate-limited requests to pre-configured fallback models across different cloud vendors.
• Granular spend analytics: Tracks token consumption, latency metrics, and financial expenditure across internal projects and API keys.
• Multi-modal endpoints: Exposes dedicated routes for document OCR extraction and WebSocket audio streaming alongside standard text completion APIs.
• Unified container packaging: Bundles the Hono gateway, web dashboard, Redis, and PostgreSQL into a single Docker image for rapid local deployment.

Limitations

• Schema conversion drops request parameters: GitHub issues #4148 and #4149 show that gateway validation schemas discard tool strict mode, parallel tool call flags, and JSON Schema keywords like const and pattern when routing to Gemini and Anthropic.