|
| 1 | +# Flyte 2 Backend |
| 2 | + |
| 3 | +This repository contains the backend infrastructure for deploying a distributed, multi-node version of Flyte 2. The backend is **Kubernetes-native** — it orchestrates workflow execution using Kubernetes primitives, scheduling tasks as pods across clusters with built-in support for multi-cluster routing, service account-based identity, and pod-level log tracking. The core architecture consists of gRPC services (QueueService, RunService, StateService) backed by PostgreSQL, using async processing and real-time streaming via PostgreSQL LISTEN/NOTIFY. See the full [Implementation Spec](https://github.com/flyteorg/flyte/blob/v2/docs/IMPLEMENTATION_SPEC.md) for details. |
| 4 | + |
| 5 | +This repo also defines the protocol buffer schemas for Flyte's APIs and generates client libraries for Go, TypeScript, Python, and Rust. Deploy this when you need Flyte running as a scalable, distributed service across your organization. |
| 6 | + |
| 7 | +**Want to contribute?** Join us on [slack.flyte.org](https://slack.flyte.org) to get involved. |
| 8 | + |
| 9 | +## Repository Structure |
| 10 | + |
| 11 | +``` |
| 12 | +flyte/ |
| 13 | +├── flyteidl2/ # Protocol buffer definitions |
| 14 | +│ ├── common/ # Common types and utilities |
| 15 | +│ ├── core/ # Core Flyte types (tasks, workflows, literals) |
| 16 | +│ ├── imagebuilder/ # Image builder service definitions |
| 17 | +│ ├── logs/ # Logging types |
| 18 | +│ ├── secret/ # Secret management types |
| 19 | +│ ├── task/ # Task execution types |
| 20 | +│ ├── trigger/ # Trigger service definitions |
| 21 | +│ ├── workflow/ # Workflow types |
| 22 | +│ └── gen_utils/ # Language-specific generation utilities |
| 23 | +├── gen/ # Generated code (not checked into version control) |
| 24 | +│ ├── go/ # Generated Go code |
| 25 | +│ ├── ts/ # Generated TypeScript code |
| 26 | +│ ├── python/ # Generated Python code |
| 27 | +│ └── rust/ # Generated Rust code |
| 28 | +├── buf.yaml # Buf configuration |
| 29 | +├── buf.gen.*.yaml # Language-specific generation configs |
| 30 | +└── Makefile # Build automation |
| 31 | +``` |
| 32 | + |
| 33 | +## Prerequisites |
| 34 | + |
| 35 | +- [Buf CLI](https://buf.build/docs/installation) - Protocol buffer tooling |
| 36 | +- Go 1.24.6 or later |
| 37 | +- Node.js/npm (for TypeScript generation) |
| 38 | +- Python 3.9+ with `uv` package manager (for Python generation) |
| 39 | +- Rust toolchain (for Rust generation) |
| 40 | + |
| 41 | +## Quick Start |
| 42 | + |
| 43 | +### Generate All Code |
| 44 | + |
| 45 | +To generate code for all supported languages: |
| 46 | + |
| 47 | +```bash |
| 48 | +make gen |
| 49 | +``` |
| 50 | + |
| 51 | +This will: |
| 52 | +1. Update buf dependencies |
| 53 | +2. Format and lint proto files |
| 54 | +3. Generate code for Go, TypeScript, Python, and Rust |
| 55 | +4. Generate mocks for Go |
| 56 | +5. Run `go mod tidy` |
| 57 | + |
| 58 | +### Generate for Specific Languages Locally |
| 59 | + |
| 60 | +```bash |
| 61 | +make buf-go # Generate Go code only |
| 62 | +make buf-ts # Generate TypeScript code only |
| 63 | +make buf-python # Generate Python code only |
| 64 | +make buf-rust # Generate Rust code only |
| 65 | +``` |
| 66 | + |
| 67 | +## Making Changes |
| 68 | + |
| 69 | +### 1. Modify Protocol Buffers |
| 70 | + |
| 71 | +Edit `.proto` files in the `flyteidl2/` directory following these guidelines: |
| 72 | +- Follow the existing naming conventions |
| 73 | +- Use proper protobuf style (snake_case for fields, PascalCase for messages) |
| 74 | +- Add appropriate comments and documentation |
| 75 | +- Ensure backward compatibility when modifying existing messages |
| 76 | + |
| 77 | +### 2. Generate Code |
| 78 | + |
| 79 | +After modifying proto files: |
| 80 | + |
| 81 | +```bash |
| 82 | +make docker-pull # Pull the docker image for generation |
| 83 | +make gen |
| 84 | +``` |
| 85 | + |
| 86 | +### 3. Verify Your Changes |
| 87 | + |
| 88 | +Run the following to ensure everything builds correctly: |
| 89 | + |
| 90 | +```bash |
| 91 | +# For Go |
| 92 | +make go-tidy |
| 93 | +go build ./... |
| 94 | + |
| 95 | +# For Rust |
| 96 | +make build-crate |
| 97 | + |
| 98 | +# For Python |
| 99 | +cd gen/python && uv lock |
| 100 | + |
| 101 | +# For TypeScript |
| 102 | +cd gen/ts && npm install |
| 103 | +``` |
| 104 | + |
| 105 | +### 4. Generate Mocks (Go only) |
| 106 | + |
| 107 | +If you've added or modified Go interfaces: |
| 108 | + |
| 109 | +```bash |
| 110 | +make gen |
| 111 | +``` |
| 112 | + |
| 113 | +## Development Workflow |
| 114 | + |
| 115 | +1. **Format proto files**: `make buf-format` |
| 116 | +2. **Lint proto files**: `make buf-lint` |
| 117 | +3. **Generate code**: `make buf` or `make gen` |
| 118 | +4. **Verify builds**: Build generated code in your target language |
| 119 | +5. **Commit changes**: Commit both proto files and generated code |
| 120 | + |
| 121 | +## Common Tasks |
| 122 | + |
| 123 | +### Update Buf Dependencies |
| 124 | + |
| 125 | +```bash |
| 126 | +make gen |
| 127 | +``` |
| 128 | + |
| 129 | +### View Available Commands |
| 130 | + |
| 131 | +```bash |
| 132 | +make help |
| 133 | +``` |
| 134 | + |
| 135 | +## Versioning and Releases |
| 136 | + |
| 137 | +See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed release instructions. |
| 138 | + |
| 139 | +## Generated Code |
| 140 | + |
| 141 | +The `gen/` directory contains auto-generated code and should not be manually edited. Changes to generated code should be made by: |
| 142 | +1. Modifying the source `.proto` files in `flyteidl2/` |
| 143 | +2. Updating generation utilities in `flyteidl2/gen_utils/` if needed |
| 144 | +3. Running `make gen` to regenerate all code |
| 145 | + |
| 146 | +## Troubleshooting |
| 147 | + |
| 148 | +### Buf Errors |
| 149 | +- Ensure you have the latest version of Buf: `buf --version` |
| 150 | +- Update dependencies: `make buf-dep` |
| 151 | +- Check `buf.lock` for dependency conflicts |
| 152 | + |
| 153 | +### Go Module Issues |
| 154 | +- Run `make go-tidy` to clean up dependencies |
| 155 | +- Ensure you're using Go 1.24.6 or later |
| 156 | + |
| 157 | +### Python Generation Issues |
| 158 | +- Ensure `uv` is installed: `pip install uv` |
| 159 | +- Set the environment variable: `export SETUPTOOLS_SCM_PRETEND_VERSION=0.0.0` |
| 160 | + |
| 161 | +### Rust Build Issues |
| 162 | +- Update Rust toolchain: `rustup update` |
| 163 | +- Navigate to `gen/rust` and run `cargo update` |
| 164 | + |
| 165 | +## Contributing |
| 166 | + |
| 167 | +We welcome contributions to Flyte 2! Please follow the guide [here](CONTRIBUTING.md). |
0 commit comments