Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding initial draft of overview document and community links #110

Merged
merged 3 commits into from
Jan 16, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 58 additions & 0 deletions site/docs/overview.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
---
id: overview
title: Overview
---

# Envoy AI Gateway Overview

Welcome to the **Envoy AI Gateway** documentation! This open-source project, built on **Envoy Proxy**, aims to simplify how application clients interact with **Generative AI (GenAI)** services. It provides a secure, scalable, and efficient way to manage LLM/AI traffic, with backend rate limiting and policy control.

## **Project Overview**

The **Envoy AI Gateway** was created to address the complexity of connecting applications to GenAI services by leveraging Envoy's flexibility and Kubernetes-native features. The project has evolved through contributions from the Envoy community, fostering a collaborative approach to solving real-world challenges.

### **Key Objectives**
- Provide a unified layer for routing and managing LLM/AI traffic.
- Support automatic failover mechanisms to ensure service reliability.
- Ensure end-to-end security, including upstream authorization for LLM/AI traffic.
- Implement a policy framework to support usage limiting use cases.
- Foster an open-source community to address GenAI-specific routing and quality of service needs.


## **Release Goals**

The initial release focuses on key foundational features to provide LLM/AI traffic management:

- **Request Routing and Failover**: Directs API requests to appropriate GenAI services and switches providers based on availability and performance.
- **Load Balancing**: Distributes incoming requests to maintain optimal system performance.
- **Authentication and Authorization**: Implement API key validation to secure communication.
- **Backend Security Policy**: Introduces fine-grained access control for backend services.
This also controls LLM/AI backend usage using token-per-second (TPS) policies to prevent overuse.
- **Multi-Upstream Provider Support for LLM/AI Services**: The ability to receive requests in the format of one LLM provider and route them to different upstream providers, ensuring compatibility with their expected formats. This is made possible through built-in transformation capabilities that adapt requests and responses accordingly.
- **AWS Request Signing**: Supports external processing for secure communication with AWS-hosted LLM/AI services.

Documentation for installation, setup, and contribution guidelines is included to help new users and contributors get started easily.


## **Community Collaboration**

Weekly community meetings are held every Thursday to discuss updates, address issues, and review contributions.

## **Architecture Overview**


## **Get Involved**

We welcome community contributions! Here's how you can participate:
- Attend the weekly community meetings to stay updated and share ideas.
- Submit feature requests and pull requests via the GitHub repository.
- Join discussions in the #envoy-ai-gateway Slack channel.

Refer to the contribution guide in the GitHub repository for detailed instructions on setting up your environment and contributing.

---

The **Envoy AI Gateway** addresses the growing demand for secure, scalable, and efficient AI/LLM traffic management. Your contributions and feedback are key to its success and to advancing the future of AI service integration.



31 changes: 29 additions & 2 deletions site/docusaurus.config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -74,12 +74,39 @@ const config: Config = {
// position: 'left',
// label: 'Tutorial',
// },
{to: '/blog', label: 'Blog', position: 'left'},
{
label: 'Overview',
to: '/docs/overview', // Path to your Overview page
position: 'left',
},
{
label: 'Community',
position: 'left',
items: [
{
label: 'Join us on Slack',
href: 'https://envoyproxy.slack.com/archives/C07Q4N24VAA',
},
{
label: 'Weekly Meeting Notes (Thursdays)',
href: 'https://docs.google.com/document/d/10e1sfsF-3G3Du5nBHGmLjXw5GVMqqCvFDqp_O65B0_w/edit?tab=t.0',
},
{
label: 'GitHub Discussions',
href: 'https://github.com/envoyproxy/ai-gateway/issues?q=is%3Aissue+label%3Adiscussion',
},
],
},
{
label: 'Blog',
to: '/blog',
position: 'left',
},
{
href: 'https://github.com/envoyproxy/ai-gateway',
label: 'GitHub',
position: 'right',
},
}
],
},
footer: {
Expand Down
Loading