docs(where-data-lives): expand what goes in the control plane database by LeonKolyang · Pull Request #1676 · unionai/unionai-docs

LeonKolyang · 2026-10-02T10:07:12Z

Summary

Expands What goes in the database in Where your data lives with records that weren't listed, based on the control-plane schema.

Both variants

Task registrations store the full task template: image reference, command and args, resource requests, environment variables (as literal key/value pairs), and secret names.
Runs store their run-level configuration: labels, annotations, run-context environment variables, custom blob storage prefixes, cache settings, and notification rules.
A note that environment variables are stored as plain text, so credentials belong in secrets. Secrets can be mounted as environment variables, and their values never enter the database.

Union only (in a {{< variant union >}} block)

Run placement: queue and cluster pool.
Apps: full spec and status, including public, CNAME, and VPC URLs, assigned cluster, creator, and last start time.
Reusable environments: spec, scaling state, and per-cluster worker status.

Existing statements (URI-only I/O pointers, the default-input and condition exceptions, the cache mapping) are unchanged.

Verification

Claims were checked against unionai/cloud and flyteorg/flyte origin/main (migrations, models, and the stored protos). Open-source Flyte keeps no app records in the database and has no reusable environments, which is why those are Union-only. make dist builds, and the Flyte variant renders without the Union-only block. check-links, markdownlint, and cspell pass.

🤖 Generated with Claude Code

Copilot

Copilot review overview

🟡 Changes recommended

The secret-name guidance contradicts the documented persistence of secret reference names.

Review effort: Balanced
Findings: 1

Open (1)

Clarify which secret names are persisted in specifications · New

What changed in this PR

Expands documentation of control-plane database contents and data-plane storage boundaries.

Changes:

Documents additional registration, execution, app, workspace, and environment records.
Clarifies sensitive-data, logs, and Deck storage guidance.

File	Description
`content/user-guide/get-started/core-concepts/where-data-lives.md`	Expands database and bucket residency details.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

ppiegaze · 2026-10-05T17:18:20Z


+Two things never enter the database at all:
+
+- **Secret values and secret names.** Secrets are created and stored in your data plane (Kubernetes Secrets or your cloud secret manager). The control plane keeps no record of them; `flyte get secret` lists them live from the data plane. Only the *references* a task, run, app, or workspace declares are stored, and a reference is just a key name.


Resolved: the 'never enters the database' block was removed earlier, and the current text says secret values are never stored, while specs record secret names.

github-actions · 2026-10-02T10:11:13Z

GHA build & deploy preview

Built by .github/workflows/build-pr.yml and deployed to the docs CF Pages project by .github/workflows/deploy-pr-preview.yml.


Branch alias	https://pr-1676-docs-where-data-live.docs-dog.pages.dev
This commit	https://d33700bd.docs-dog.pages.dev
Commit SHA	`f0e029efbc7956a96f0729406d62a270b4bc8a18`

Updated automatically on every push.

EngHabu

A couple of nits! thank you for the update

EngHabu · 2026-10-02T17:20:45Z


 The values your tasks actually pass at runtime, even a bare `int`, do **not** live in the database. They are written to `inputs.pb` / `outputs.pb` in the bucket, and the database keeps only the pointer. See the next section.

+Because environment variables in task, run, and app specs are stored as plain text in the database, put credentials and other sensitive configuration in secrets rather than in `env` maps.


Maybe add that secrets can be mounted as env vars if code expects to read env vars.

Done in f0e029e: 'A secret can be mounted as an environment variable, and because it is mounted at runtime, its value is never stored in the control plane database.' Also links to the Secrets page.

EngHabu · 2026-10-02T17:22:17Z

+- **Execution records**: every run, every action (task / trace / condition) inside that run, attempts, phases, timing, error messages, parent/child relationships. Each run also stores the run-level configuration it was launched with: labels, annotations, environment variables set through the run context, queue / cluster pool, `raw_data_path`, cache settings, notification rules, and the names of secrets requested at run level.
 - **Schedules and triggers**: `Cron`, event triggers, and their revision history.
+- **Apps**: every deployed app's full spec (container image, command and arguments, environment variables, ports, resources, autoscaling and ingress settings, secret *names*, and any string inputs you pass to the app) plus its current status, including the public / CNAME / VPC URLs it is reachable at, the assigned cluster, the creator, and the last start time.
+- **Reusable environments**: only a `TaskEnvironment` with a `ReusePolicy` gets a record of its own; a plain environment is flattened into the registrations of its tasks and has no separate row. For a reusable environment the database holds the environment spec (image, resources, environment variables, secret names, pod template, parallelism), its scaling state (replicas, idle and scale-down TTLs), and the per-cluster status of its worker pool, including worker states and recent worker error messages.


"gets a record of its own... a plain...." feels like an implementation detail we may decide to change later and is immaterial to the where data lives question IMHO

Done in f0e029e: removed 'Plain environment specs are not persisted' and the record-of-its-own phrasing.

… env vars and secret handling in the database section Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: LeonKolyang <leon.menkreo@googlemail.com> Signed-off-by: Peeter Piegaze <1153481+ppiegaze@users.noreply.github.com>

…their own record Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: LeonKolyang <leon.menkreo@googlemail.com> Signed-off-by: Peeter Piegaze <1153481+ppiegaze@users.noreply.github.com>

Signed-off-by: LeonKolyang <leon.menkreo@googlemail.com> Signed-off-by: Peeter Piegaze <1153481+ppiegaze@users.noreply.github.com>

Co-authored-by: Daniel Sola <40698988+dansola@users.noreply.github.com> Signed-off-by: Leon Menkreo <50681264+LeonKolyang@users.noreply.github.com> Signed-off-by: Peeter Piegaze <1153481+ppiegaze@users.noreply.github.com>

Signed-off-by: Peeter Piegaze <1153481+ppiegaze@users.noreply.github.com>

Checked against unionai/cloud and flyteorg/flyte origin/main: - OSS Flyte keeps no app records in the database (state lives in the Knative service) and has no reusable environments; queue and cluster pool are stored but unused, and only the public app URL exists. Moved apps, reusable environments, and run placement into a Union-only block, and gated the matching plain-text env var note. - Dropped "secrets requested at run level": RunSpec can hold them, but the SDK only sets a service account there. - App inputs can be strings, app IDs, or artifact references, not only strings; app image is a reference, as for tasks. - Removed "Plain environment specs are not persisted" (implementation detail, and tasks still record their environment's name). - Linked secrets, and split the env var sentence. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Peeter Piegaze <1153481+ppiegaze@users.noreply.github.com>

Copilot AI balanced review requested due to automatic review settings October 2, 2026 10:07

LeonKolyang requested review from EngHabu, cosmicbboy, kumare3, ppiegaze and samhita-alla as code owners October 2, 2026 10:07

Copilot started reviewing on behalf of LeonKolyang October 2, 2026 10:07 View session

Copilot AI reviewed Oct 2, 2026

View reviewed changes

LeonKolyang force-pushed the docs/where-data-lives-db-contents branch from c6fe1ff to c8de224 Compare October 2, 2026 10:39

dansola reviewed Oct 2, 2026

View reviewed changes

Comment thread content/user-guide/get-started/core-concepts/where-data-lives.md Outdated

dansola reviewed Oct 2, 2026

View reviewed changes

Comment thread content/user-guide/get-started/core-concepts/where-data-lives.md Outdated

EngHabu approved these changes Oct 2, 2026

View reviewed changes

LeonKolyang and others added 7 commits October 5, 2026 19:13

docs(where-data-lives): clarify that only reusable environments have …

9858511

…their own record Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: LeonKolyang <leon.menkreo@googlemail.com> Signed-off-by: Peeter Piegaze <1153481+ppiegaze@users.noreply.github.com>

docs(where-data-lives): cleanup

1d9e8bf

Signed-off-by: LeonKolyang <leon.menkreo@googlemail.com> Signed-off-by: Peeter Piegaze <1153481+ppiegaze@users.noreply.github.com>

docs(where-data-lives): remove secrets and logs

b40b334

Signed-off-by: LeonKolyang <leon.menkreo@googlemail.com> Signed-off-by: Peeter Piegaze <1153481+ppiegaze@users.noreply.github.com>

docs(where-data-lives): remove implementation specifics

f3382dc

Signed-off-by: Peeter Piegaze <1153481+ppiegaze@users.noreply.github.com>

ppiegaze force-pushed the docs/where-data-lives-db-contents branch from acddbc7 to f0e029e Compare October 5, 2026 17:18

ppiegaze merged commit b223444 into main Oct 5, 2026
17 checks passed

ppiegaze deleted the docs/where-data-lives-db-contents branch October 5, 2026 17:34

Navigation Menu

Repository navigation

docs(where-data-lives): expand what goes in the control plane database - #1676

docs(where-data-lives): expand what goes in the control plane database#1676
ppiegaze merged 8 commits into
mainfrom
docs/where-data-lives-db-contents

LeonKolyang commented Oct 2, 2026 •

edited by ppiegaze

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

ppiegaze Oct 5, 2026

Uh oh!

github-actions Bot commented Oct 2, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

EngHabu left a comment

Uh oh!

EngHabu Oct 2, 2026

Uh oh!

ppiegaze Oct 5, 2026

Uh oh!

EngHabu Oct 2, 2026

Uh oh!

ppiegaze Oct 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants


		Two things never enter the database at all:

		- Secret values and secret names. Secrets are created and stored in your data plane (Kubernetes Secrets or your cloud secret manager). The control plane keeps no record of them; `flyte get secret` lists them live from the data plane. Only the references a task, run, app, or workspace declares are stored, and a reference is just a key name.


		The values your tasks actually pass at runtime, even a bare `int`, do not live in the database. They are written to `inputs.pb` / `outputs.pb` in the bucket, and the database keeps only the pointer. See the next section.

		Because environment variables in task, run, and app specs are stored as plain text in the database, put credentials and other sensitive configuration in secrets rather than in `env` maps.

Conversation

LeonKolyang commented Oct 2, 2026 • edited by ppiegaze Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Verification

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

ppiegaze Oct 5, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

GHA build & deploy preview

Uh oh!

Uh oh!

Uh oh!

EngHabu left a comment

Choose a reason for hiding this comment

Uh oh!

EngHabu Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

ppiegaze Oct 5, 2026

Choose a reason for hiding this comment

Uh oh!

EngHabu Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

ppiegaze Oct 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

LeonKolyang commented Oct 2, 2026 •

edited by ppiegaze

Loading

github-actions Bot commented Oct 2, 2026 •

edited

Loading