ai-gateway

ai-gateway

Manages Unified Access to Generative AI Services built on Envoy Gateway

Stars: 1064

Visit
 screenshot

Envoy AI Gateway is an open source project that utilizes Envoy Gateway to manage request traffic from application clients to Generative AI services. The project aims to provide a seamless and efficient solution for handling communication between clients and AI services. It is designed to enhance the performance and scalability of AI applications by leveraging the capabilities of Envoy Gateway. The project welcomes contributions from the community and encourages collaboration to further develop and improve the functionality of the AI Gateway.

README:

Envoy AI Gateway

Envoy AI Gateway is an open source project for using Envoy Gateway to handle request traffic from application clients to Generative AI services.

Usage

When using Envoy AI Gateway, we refer to a two-tier gateway pattern. The Tier One Gateway functions as a centralized entry point, and the Tier Two Gateway handles ingress traffic to a self-hosted model serving cluster.

  • The Tier One Gateway handles authentication, top-level routing, and global rate limiting
  • The Tier Two Gateway provides fine-grained control over self-hosted model access, with endpoint picker support for LLM inference optimization.

Supported AI Providers

Envoy AI Gateway supports a wide range of AI providers, making it easy to integrate with your preferred LLM services:

OpenAI
OpenAI
Azure OpenAI
Azure OpenAI
Google Gemini
Google Gemini
Vertex AI
Vertex AI
AWS Bedrock
AWS Bedrock
Mistral
Mistral
Cohere
Cohere
Groq
Groq
Together AI
Together AI
DeepInfra
DeepInfra
DeepSeek
DeepSeek
Hunyuan
Hunyuan
SambaNova
SambaNova
Grok
Grok

Documentation

  • Blog introducing Envoy AI Gateway.
  • Documentation for Envoy AI Gateway.
  • Quickstart to use Envoy AI Gateway in a few simple steps.
  • Concepts to understand the architecture and resources of Envoy AI Gateway.

Contact

Get Involved

We adhere to the CNCF Code of conduct

The Envoy AI Gateway team and community members meet every Thursday. Please register for the meeting, add agenda points, and get involved. The meeting details are available in the public document.

To contribute to the project via pull requests, please read the CONTRIBUTING.md file which includes information on how to build and test the project.

Background

The proposal of using Envoy Gateway as a Cloud Native LLM Gateway inspired the initiation of this project.

For Tasks:

Click tags to check more tools for each tasks

For Jobs:

Alternative AI tools for ai-gateway

Similar Open Source Tools

For similar tasks

For similar jobs