Flyte 2: Cut over to Flyte 2 new backend implementation

[EngHabu]

Summary

This PR merges the entire Flyte 2 backend from the v2 branch into main. It represents a ground-up rewrite of the Flyte control plane: Kubernetes-native, gRPC/buf-connect-based, PostgreSQL-backed, and structured as a set of independently runnable services that can also be composed into a single binary.

This is a cutover, not an incremental change. After merge, main becomes the home of Flyte 2; Flyte 1 continues to live on master.

What's in v2

New protobuf surface — flyteidl2/

A clean v2 schema replacing flyteidl. Domains include core, workflow, task, actions, event, auth, cluster, org, project, notification, secret, imagebuilder, trigger, cacheservice, dataproxy, connector, plugins, logs, settings, and common. Code generation supports Go, TypeScript, Python, and Rust via buf.

New gRPC services (each with its own cmd/, config/, service/, repository/, migrations/)

• runs/ — RunService: workflow run lifecycle, with a built-in scheduler.
• queue/ — QueueService: action enqueue/abort.
• events/ — EventService: real-time event streaming via PostgreSQL LISTEN/NOTIFY.
• cache_service/ — Cache lookup/upload for task outputs.
• dataproxy/ — Signed-URL data proxy + presigned log streaming.
• secret/ — SecretService.
• app/ — AppLogsService (tail/stream pod logs).
• manager/ — Unified host process composing the services for single-binary deployments.

All services share a common pattern: pgx/gorm for Postgres, flytestdlib for config/logging/database/health, buf-connect handlers over net/http, graceful shutdown, and standardized health endpoints. See IMPLEMENTATION_SPEC.md for the architectural contract.

Executor — executor/

Kubernetes-native task execution. Schedules tasks as pods, supports multi-cluster routing, service-account-based identity, and per-pod log tracking. Includes its own Dockerfile, Makefile, controller scaffolding, and dev workflow.

Triggers

Triggers API + validation annotations end-to-end (proto, server, scheduler).

Generated clients — gen/

Build artifacts for Go, TypeScript, Python, and Rust generated from flyteidl2. The Python SDK lives in its own repo (flyteorg/flyte-sdk).

Deployment

• New Helm charts: charts/flyte-binary, charts/flyte-devbox, charts/flyte-sandbox.
• New top-level Dockerfile for the unified binary; DOCKER_QUICK_START.md for the local Docker workflow.
• Reworked docker/ and CI image-build workflows.

Foundational packages

• flytestdlib/ — config, database, logging, promutils, etc., updated for v2 patterns.
• flyteplugins/, flytecopilot/ — carried forward and adapted.

Repository & developer experience

• Module path: github.com/flyteorg/flyte/v2.
• Top-level Makefile, buf.gen.{go,ts,python,rust}.yaml, .mockery.yaml.
• Contributor docs (BACKEND_README.md, CONTRIBUTING.md, per-service README.md / DEVELOPMENT.md).
• New CODEOWNERS for v2 modules.

Why now

main exists specifically as the v2 line — it forked from master at the "Clear flyte to prep for v2" commit and is not being kept in sync with Flyte 1 work. Flyte 1 maintenance continues on master. This PR makes main reflect the actual state of v2 development that has been happening on the v2 branch.

DCO

The v2 → main history contains 133 commits across ~20 contributors that fail the strict default DCO check (mostly GitHub noreply-vs-personal-email mismatches on commits that were originally merged through DCO-clean PRs to master). Resolved via:

• ci: add DCO config to allow remediation commits #7297 — adds .github/dco.yml to master enabling third-party remediation commits.
• ci: DCO remediation commit for v2 -> main integration #7298 — adds the remediation commit to v2 covering all 133 SHAs.

Merge strategy

Squash

• Flyte 2: Cut over to Flyte 2 new backend implementation 👈
  • try to remove buf validate deps #6646
  • [Feat] Add GetLatestRun service request and response proto #6690
  • first pass copying over most of flyteplugins #6715
  • Common Filter to take in int values as well #6742
  • Fix lint issues #6748
  • Add design docs for flyte v2 #6766
  • [Feat][V2] Implement simple plugin in executor #6806
  • [Refactor]: CreateRun add initial service and response structure #6853
  • [Fix] Prevent uv.lock changes (#6845) #6879
  • fix: resolve flytestdlib test failures #6880
  • Add ExecutedBy EnrichedIdentity to RunSpec #6906
  • Add listAndWatch response proto #6909
  • Adam/domains flyte2 #6933
  • Move to endpoint slices #6938
  • bump spark operator #6944
  • Add proto files for domain service #6963
  • [V2] Example script for run service #6983
  • [V2] Add AuthMetadataService + wire external OAuth2 auth into runs service #6998
  • proto definitions for settings service #7023
  • fix: Track ResourceVersion in TaskAction watcher to prevent duplicate ADDED events #7037
  • feat(executor): support immediate GC cleanup when MaxTTL <= 0 #7052
  • [Flyte 6978] Ensure WatchRunDetails works expected #7066
  • Add codespell config to v2 branch and fix detected typos #7121
  • Fix/remove state min len validation 2 #7138
  • [V2] App Service design doc #7154
  • Add retry execution error #7162
  • [Flyte2 7005]  #7171
  • Yaroslav/scheduler queue #7222
  • use Helm to init rustfs in sandbox #7233
  • [V2] feat: Dockerfile for each components #7238
  • Allow routing secrets endpoints via SelectCluster #7245
  • Add system architecture documentation #7249
  • [Flyte 7198] Enabling flyte plugins #7259
  • perf(devbox): parallelize image preload in devbox-bundled build #7263
  • chore: upgrade Go version to 1.26 #7267
  • Delegate RunLogsService.TailLogs to DataProxyService to remove duplicate implementation #7252 #7269
  • [V2] feat: Create Helm charts for flyte-core #7285
  • [WIP][V2]Add devbox integration test #7316

[flyte-bot]

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at [email protected].

[flyte-bot]

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at [email protected].

[flyte-bot]

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at [email protected].

[flyte-bot]

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at [email protected].

[flyte-bot]

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at [email protected].

[bito-code-review]

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at [email protected].

[bito-code-review]

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at [email protected].

[bito-code-review]

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at [email protected].

[bito-code-review]

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at [email protected].

[bito-code-review]

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at [email protected].

[bito-code-review]

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at [email protected].

[bito-code-review]

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at [email protected].

[bito-code-review]

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at [email protected].

[bito-code-review]

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at [email protected].

[bito-code-review]

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at [email protected].

[bito-code-review]

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at [email protected].

[bito-code-review]

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at [email protected].

[flyte-bot]

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at [email protected].

[github-actions]

🐳 Docker CI Image Built

The CI Docker image has been built and pushed for this PR!

Image: ghcr.io/flyteorg/flyte/ci:pr-6583

This image will be automatically used by CI workflows in this PR.

To test locally:

make gen DOCKER_CI_IMAGE=ghcr.io/flyteorg/flyte/ci:pr-6583

[flyte-bot]

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at [email protected].

[flyte-bot]

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at [email protected].

[flyte-bot]

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at [email protected].

[flyte-bot]

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at [email protected].

[flyte-bot]

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at [email protected].