hub / github.com/envoyproxy/ai-gateway
github.com/envoyproxy/ai-gateway @v1.0.0
Chat with this repo
5,510 symbols
23,086 edges
551 files
3,094 documented · 56%
228 cross-repo links
● updated 3d agov1.0.0 · 2026-06-23★ 1,813116 open issues
What it actually does AI analysis from the code graph — generated when you open this
loading…
README
Envoy AI Gateway
Envoy AI Gateway is an open source project for using Envoy Gateway to handle request traffic from application clients to Generative AI services.
Usage
When using Envoy AI Gateway, we refer to a two-tier gateway pattern. The Tier One Gateway functions as a centralized entry point, and the Tier Two Gateway handles ingress traffic to a self-hosted model serving cluster.
- The Tier One Gateway handles authentication, top-level routing, and global rate limiting
- The Tier Two Gateway provides fine-grained control over self-hosted model access, with endpoint picker support for LLM inference optimization.
Supported AI Providers
Envoy AI Gateway supports a wide range of AI providers, making it easy to integrate with your preferred LLM services:
OpenAI
|
Azure OpenAI
|
Google Gemini
|
Vertex AI
|
AWS Bedrock
|
Mistral
|
Cohere
|
Groq
|
Together AI
|
DeepInfra
|
DeepSeek
|
Hunyuan
|
SambaNova
|
Grok
|
Tetrate Agent Router Service
|
Anthropic
|
Documentation
- Blog introducing Envoy AI Gateway.
- Documentation for Envoy AI Gateway.
- Quickstart to use Envoy AI Gateway in a few simple steps.
- Concepts to understand the architecture and resources of Envoy AI Gateway.
- Talks and Presentations about Envoy AI Gateway.
Contact
- Slack: Join the Envoy Slack workspace if you're not already a member. Otherwise, use the Envoy AI Gateway channel to start collaborating with the community.
Get Involved
We adhere to the CNCF Code of conduct
The Envoy AI Gateway team and community members meet every Monday. Please register for the meeting, add agenda points, and get involved. The meeting details are available in the public document.
To contribute to the project via pull requests, please read the CONTRIBUTING.md file which includes information on how to build and test the project.
Background
The proposal of using Envoy Gateway as a Cloud Native LLM Gateway inspired the initiation of this project.
Extension points exported contracts — how you extend this code
Spec (Interface)
Spec defines methods for parsing request bodies and selecting translators for different API endpoints. Type Parameters: [10 …
internal/endpointspec/endpointspec.go
BackendAuthHandler (Interface)
BackendAuthHandler is the interface that deals with the backend auth for a specific backend. [8 implementers]
internal/filterapi/runtime.go
Translator (Interface)
Translator translates the request and response messages between the client and the backend API schemas. ReqT represents [19 …
internal/translator/translator.go
Span (Interface)
Span standardizes span interfaces, supporting both streaming and non-streaming endpoints. [9 implementers]
internal/tracing/tracingapi/api.go
BackendSecurityPoliciesGetter (Interface)
BackendSecurityPoliciesGetter has a method to return a BackendSecurityPolicyInterface. A group's client should implement [8 …
api/v1beta1/client/clientset/versioned/typed/api/v1beta1/backendsecuritypolicy.go
BackendSecurityPoliciesGetter (Interface)
BackendSecurityPoliciesGetter has a method to return a BackendSecurityPolicyInterface. A group's client should implement [8 …
api/v1alpha1/client/clientset/versioned/typed/api/v1alpha1/backendsecuritypolicy.go
TokenProvider (Interface)
TokenProvider is an interface for retrieving tokens. [5 implementers]
internal/controller/tokenprovider/token_provider.go
Metrics (Interface)
Metrics is the interface for the base AI Gateway metrics. [2 implementers]
internal/metrics/metrics.go
Core symbols most depended-on inside this repo
Equal
called by 2918
internal/apischema/openai/openai.go
String
called by 2040
internal/testing/ollama.go
Error
called by 444
internal/mcpproxy/handlers.go
InputMessageAttribute
called by 379
internal/tracing/openinference/openinference.go
Get
called by 375
api/v1alpha1/client/listers/api/v1alpha1/mcproute.go
String
called by 354
internal/mcpproxy/session.go
Set
called by 296
internal/tracing/mcp.go
Create
called by 287
api/v1alpha1/client/clientset/versioned/typed/api/v1alpha1/mcproute.go
Shape
Languages
Go99%
TypeScript1%
Python1%
Modules by API surface
internal/apischema/openai/openai.go494 symbols
api/v1alpha1/zz_generated.deepcopy.go148 symbols
api/v1beta1/zz_generated.deepcopy.go128 symbols
internal/apischema/openai/openai_test.go121 symbols
internal/apischema/anthropic/anthropic.go120 symbols
internal/endpointspec/endpointspec.go66 symbols
internal/tracing/tracer_test.go59 symbols
internal/mcpproxy/handlers_test.go55 symbols
internal/translator/openai_helper.go53 symbols
internal/mcpproxy/handlers.go52 symbols
internal/apischema/anthropic/anthropic_test.go52 symbols
tests/internal/e2elib/e2elib.go51 symbols
Dependencies from manifests, versioned
4d63.com/gocheckcompilerdirectivesv1.3.0 · 1×
4d63.com/gochecknoglobalsv0.2.2 · 1×
al.essio.dev/pkg/shellescapev1.5.1 · 1×
cel.dev/exprv0.25.1 · 1×
cloud.google.com/gov0.121.2 · 1×
cloud.google.com/go/authv0.20.0 · 1×
cloud.google.com/go/auth/oauth2adaptv0.2.8 · 1×
cloud.google.com/go/compute/metadatav0.9.0 · 1×
codeberg.org/chavacava/garifv0.2.0 · 1×
codeberg.org/polyfloyd/go-errorlintv1.9.0 · 1×
dario.cat/mergov1.0.2 · 1×
dev.gaijin.team/go/exhaustruct/v4v4.0.0 · 1×
For agents
$ claude mcp add ai-gateway \
-- python -m otcore.mcp_server <graph>