Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 35 additions & 10 deletions docs/developer_guide/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,8 +52,13 @@ nixidy/
├── pkgs/ # Nix packages and generators
│ └── generators/ # CRD and K8s schema generators
│ ├── crd2jsonschema.py # CRD to JSON schema converter
│ ├── default.nix # Generator entry point
│ ├── generator.nix # Nix options generator
│ ├── default.nix # Generator entry point (accessors)
│ ├── walk.nix # Shared schema traversal (backend-parameterized)
│ ├── backend-text.nix # Text backend — emits Nix source
│ ├── backend-value.nix # Value backend — live module values
│ ├── generator.nix # File assembler (text backend → .nix file)
│ ├── module.nix # Module assembler (value backend → module value)
│ ├── runtime.nix # Runtime helpers as values (for module.nix)
│ └── versions.nix # Kubernetes versions config
├── tests/ # Module unit tests
│ ├── helm/ # Helm-specific tests
Expand Down Expand Up @@ -355,38 +360,58 @@ Nixidy generates typed Nix options from:

Output: `modules/generated/k8s/v1.XX.nix`

### Shared schema walk and backends

The schema-to-options logic lives in one place, `walk.nix`, parameterized over a *backend* so the same traversal can produce either Nix **source text** or live module **values**:

- `walk.nix` — the single traversal of the (JSON) schema. All type/coercion branch logic (int-or-string, `additionalProperties`→`attrsOf`, coerce-by-name lists, patch-merge-key, `skipCoerceToList`, `specialMapKeys`, nested submodules, etc.) lives here, expressed against an abstract backend `b`.
- `backend-text.nix` — emits Nix **source**. Used by the file generators; it also inlines the runtime helpers (see below) as source so committed standalone files stay self-contained.
- `backend-value.nix` — produces live module **values** using `runtime.nix`.
- `generator.nix` — thin assembler over the text backend (renders + `nixfmt` + writes a `.nix` file).
- `module.nix` — thin assembler over the value backend (returns a module function).
- `runtime.nix` — the per-generated-module runtime helpers (`coercedTo`, `mergeValuesByKey`, `submoduleForDefinition`, …) as **values**, imported by the value backend. The text backend inlines source-form equivalents of these same helpers. Only the *walk* is single-sourced; the helpers must exist in both forms because live values can't be serialized back to source and committed files can't reference nixidy internals.

### CRD Generation

Two entry points:
CRD accessors form a matrix over output shape × source, all built on the shared walk:

| | source files (`src`) | Helm chart |
| --- | --- | --- |
| → generated file | `fromCRD` | `fromChartCRD` |
| → module value | `fromCRDModule` | `fromChartCRDModule` |
| → raw objects | `crdObjects` | `crdObjectsFromChart` |

#### `fromCRD`
```nix
fromCRD {
name = "cilium";
src = pkgs.fetchFromGitHub { ... };
crds = [ "path/to/crd.yaml" ];
crdFiles = [ "path/to/crd.yaml" ]; # list of YAML files under `src`
kindFilter = [ ]; # Optional: only these CRD kinds (default: all)
namePrefix = ""; # Optional prefix for attribute names
attrNameOverrides = { }; # Manual name overrides
skipCoerceToList = { }; # Skip list coercion for specific fields
}
```

#### `fromChartCRD`
```nix
fromChartCRD {
name = "cert-manager";
chartAttrs = { repo = "..."; chart = "..."; version = "..."; };
crds = [ "Certificate" ]; # Filter by kind
kindFilter = [ "Certificate" ]; # Optional: only these CRD kinds (default: all)
kubeVersion = "v1.31.0"; # Optional: helm template --kube-version
}
```

The `*Module` variants take the same arguments and return a module value (no file); `crdObjects`/`crdObjectsFromChart` return the raw `CustomResourceDefinition` manifests as values.

!!! note "Renamed arguments"
The overloaded `crds` argument was split: source-based accessors now take `crdFiles` (the list of YAML files) and chart-based accessors take `kindFilter` (the kind filter). `crds` remains as a deprecated alias on every accessor (resolved by the shared `renamedArg` helper, which warns and points at the new name).

### CRD Processing (`crd2jsonschema.py`)

Python script that:
1. Reads CRD YAML files
2. Extracts OpenAPI v3 schemas
3. Flattens `$ref` references
4. Outputs JSON schema for `generator.nix`
4. Outputs JSON schema for the shared walk (`walk.nix`)

## Testing Framework

Expand Down
2 changes: 1 addition & 1 deletion docs/llms.txt
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ Nixidy brings the power of Nix to Kubernetes cluster management, enabling declar
## User Guide

- [Using Helm Charts](https://nixidy.dev/user_guide/helm_charts/): Integrate existing Helm charts with patch support
- [Typed Resource Options](https://nixidy.dev/user_guide/typed_resources/): Generate typed Nix options from CRDs using fromCRD and fromChartCRD
- [Typed Resource Options](https://nixidy.dev/user_guide/typed_resources/): Generate typed Nix options from CRDs fromCRD/fromChartCRD (generated file), fromCRDModule/fromChartCRDModule (module value, no file), and crdObjects/crdObjectsFromChart (raw CRD manifests)
- [Templates](https://nixidy.dev/user_guide/templates/): Create reusable application patterns with configurable options
- [Git Strategies](https://nixidy.dev/user_guide/git_strategies/): Monorepo, environment branches, or separate repositories
- [GitHub Actions](https://nixidy.dev/user_guide/github_actions/): CI/CD integration with nixidy/build and nixidy/switch actions
Expand Down
90 changes: 83 additions & 7 deletions docs/user_guide/typed_resources.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,22 @@ Each file's basename becomes the output filename. Basenames must be unique withi

Thankfully a code generator for generating resource options from CRDs is provided by nixidy (this is based heavily on kubenix's code generator).

nixidy offers a small matrix of CRD accessors. Pick one by **what you want back** (a generated `.nix` file, a live module value, or the raw CRD manifests) and **where the CRDs come from** (local source files, or a Helm chart):

| | source files (`src`) | Helm chart |
| ------------------ | -------------------- | --------------------- |
| **→ generated file** | [`fromCRD`](#from-source-files-fromcrd) | [`fromChartCRD`](#from-helm-chart-crds-fromchartcrd) |
| **→ module value** | [`fromCRDModule`](#as-a-module-value-fromcrdmodule-fromchartcrdmodule) | [`fromChartCRDModule`](#as-a-module-value-fromcrdmodule-fromchartcrdmodule) |
| **→ raw objects** | [`crdObjects`](#raw-crd-objects-crdobjects-crdobjectsfromchart) | [`crdObjectsFromChart`](#raw-crd-objects-crdobjects-crdobjectsfromchart) |

- **→ generated file** renders the resource options to Nix source, formats it, and writes a `.nix` file you commit and `import`. Good when you want the generated types checked into your repository and reviewable in diffs.
- **→ module value** returns the resource options as a module value directly — no generated file, no `import`. Drop it straight into `nixidy.applicationImports`. Good for flake/programmatic consumers that don't need a committed file.
- **→ raw objects** returns the `CustomResourceDefinition` manifests themselves as Nix values (not resource options), e.g. so you can deploy the CRDs alongside the resources that use them.

All variants share the same `name`, `namePrefix`, `attrNameOverrides` and `skipCoerceToList` arguments; the source-based ones take `crdFiles` and the chart-based ones take `chartAttrs`/`chart`/`values`/`kubeVersion`. Both families take an optional `kindFilter` to narrow which CRD kinds are processed.

### From source files (`fromCRD`)

As an example, to generate resource options for Cilium's `CiliumNetworkPolicy` and `CiliumClusterwideNetworkPolicy` the following can be defined in `flake.nix`.

```nix title="flake.nix"
Expand Down Expand Up @@ -64,7 +80,7 @@ As an example, to generate resource options for Cilium's `CiliumNetworkPolicy` a
rev = "v1.15.6";
hash = "sha256-oC6pjtiS8HvqzzRQsE+2bm6JP7Y3cbupXxCKSvP6/kU=";
};
crds = [
crdFiles = [
"pkg/k8s/apis/cilium.io/client/crds/v2/ciliumnetworkpolicies.yaml"
"pkg/k8s/apis/cilium.io/client/crds/v2/ciliumclusterwidenetworkpolicies.yaml"
];
Expand Down Expand Up @@ -113,7 +129,7 @@ Then running `nix build .#generators.cilium` will produce a nix file that can be
rev = "v1.15.6";
hash = "sha256-oC6pjtiS8HvqzzRQsE+2bm6JP7Y3cbupXxCKSvP6/kU=";
};
crds = [
crdFiles = [
"pkg/k8s/apis/cilium.io/client/crds/v2/ciliumnetworkpolicies.yaml"
"pkg/k8s/apis/cilium.io/client/crds/v2/ciliumclusterwidenetworkpolicies.yaml"
];
Expand All @@ -123,7 +139,7 @@ Then running `nix build .#generators.cilium` will produce a nix file that can be

Then running `nix-build generate.nix -A cilium` will produce a nix file that can be copied into place in your repository and imported using `nixidy.applicationImports` in a nixidy module.

### Generating resource options from Helm Chart CRDs
### From Helm Chart CRDs (`fromChartCRD`)

In some cases, CRDs are only available through Helm charts or it's beneficial to keep them in sync with the chart version you're deploying. The `fromChartCRD` function provides a solution by templating the Helm chart and extracting CRDs from the output, generating nixidy resource options from them.

Expand Down Expand Up @@ -166,7 +182,7 @@ As an example, to generate resource options for cert-manager's `Certificate` CRD
};
# Or from nixhelm
# chart = nixhelm.chartsDerivations.${system}.jetstack.cert-manager;
crds = [ "Certificate" ]; # Optional: filter by specific CRD kinds
kindFilter = [ "Certificate" ]; # Optional: filter by specific CRD kinds
extraOpts = [ "--set", "crds.enabled=true"]; # Optional: pass extra options to helm template generation
};
};
Expand Down Expand Up @@ -205,7 +221,7 @@ Then running `nix build .#generators.certManager` will produce a nix file that c
version = "v1.19.1";
chartHash = "sha256-fs14wuKK+blC0l+pRfa//oBV2X+Dr3nNX+Z94nrQVrA=";
};
crds = [ "Certificate" ]; # Optional: filter by specific CRD kinds
kindFilter = [ "Certificate" ]; # Optional: filter by specific CRD kinds
extraOpts = [ "--set", "crds.enabled=true"]; # Optional: pass extra options to helm template generation
};
}
Expand All @@ -218,12 +234,72 @@ The `fromChartCRD` function accepts the same optional arguments as `fromCRD` (`n
- `chartAttrs`: Chart repository, name, version and chartHash configuration
- `chart`: Alternative to `chartAttrs`, can use a pre-downloaded chart
- `values`: Values to pass to the Helm chart templating
- `crds`: List of CRD kinds to extract (empty list extracts all CRDs)
- `kindFilter`: List of CRD kinds to extract (empty list extracts all CRDs)
- `kubeVersion`: Kubernetes version to template the chart against (`helm template --kube-version`). Defaults to the version in nixidy's nixpkgs; set it to match the cluster the CRDs are destined for.
- `extraOpts`: List of extra arguments to pass through to helm template

!!! note "Renamed argument"
`kindFilter` was previously called `crds`. The old name still works as a deprecated alias (it emits a warning pointing at `kindFilter`); prefer `kindFilter` in new code. Note that on `fromCRD` the `crds` argument instead became `crdFiles` — the two used to share a name despite meaning different things (a kind filter vs. a list of YAML files).

This approach ensures your CRD definitions stay synchronized with the Helm chart version you're actually deploying in your applications.

### As a module value (`fromCRDModule` / `fromChartCRDModule`)

`fromCRD` and `fromChartCRD` build a `.nix` file that you commit and `import`. For flake or other programmatic consumers that don't need a committed file, `fromCRDModule` (and its chart counterpart `fromChartCRDModule`) return the resource options as a **module value** instead — skipping the render-to-source, format, write-file and `import` round-trip. The result is a module function, which is exactly what `nixidy.applicationImports` accepts, so it drops straight in.

```nix title="flake.nix"
nixidyEnvs = nixidy.lib.mkEnvs {
inherit pkgs;

envs.dev.modules = [
./env/dev.nix
{
nixidy.applicationImports = [
(nixidy.packages.${system}.generators.fromCRDModule {
name = "cilium";
src = pkgs.fetchFromGitHub {
owner = "cilium";
repo = "cilium";
rev = "v1.15.6";
hash = "sha256-oC6pjtiS8HvqzzRQsE+2bm6JP7Y3cbupXxCKSvP6/kU=";
};
crdFiles = [
"pkg/k8s/apis/cilium.io/client/crds/v2/ciliumnetworkpolicies.yaml"
"pkg/k8s/apis/cilium.io/client/crds/v2/ciliumclusterwidenetworkpolicies.yaml"
];
})
];
}
];
};
```

`fromChartCRDModule` takes the same chart arguments as `fromChartCRD` (`chartAttrs`/`chart`/`values`/`kindFilter`/`kubeVersion`/`extraOpts`). Because no file is generated there is nothing to `nix build` or commit — the types are produced during evaluation.

### Raw CRD objects (`crdObjects` / `crdObjectsFromChart`)

Sometimes you don't want resource *options* at all — you want the `CustomResourceDefinition` manifests themselves, for example to deploy the CRDs into the cluster alongside the resources that use them. `crdObjects` (source files) and `crdObjectsFromChart` (Helm chart) return the CRD manifests as a list of plain Nix attribute sets. They are deployment-agnostic — what you do with them is up to you.

One way to deploy them is to feed them into an application's `yamls`, which parses YAML/JSON strings into the application's output (`builtins.toJSON` output is valid YAML):

```nix
# inside an env module
applications.crds = {
namespace = "kube-system";
yamls = map builtins.toJSON (
nixidy.packages.${system}.generators.crdObjects {
src = ciliumSrc; # the pkgs.fetchFromGitHub from the fromCRD example
crdFiles = [
"pkg/k8s/apis/cilium.io/client/crds/v2/ciliumnetworkpolicies.yaml"
"pkg/k8s/apis/cilium.io/client/crds/v2/ciliumclusterwidenetworkpolicies.yaml"
];
}
);
};
```

`crdObjectsFromChart` takes the same chart arguments as `fromChartCRD`. Both accept an optional `kindFilter` to keep only specific CRD kinds (an empty or unset list keeps all of them).

### Resolving Naming Conflicts

Sometimes, multiple Custom Resource Definitions from different sources might define the same resource `kind`. This can lead to conflicts in the generated attribute names. For instance, if two different operators both define a CRD with the kind `Database`, they would both try to generate options for `resources.databases`.
Expand Down Expand Up @@ -265,7 +341,7 @@ This argument is a mapping from the CRD's name (`<plural>.<group>`) to the desir
repo = "provider-keycloak";
...
};
crds = [
crdFiles = [
# This CRD conflicts with Kubernetes builtin Binding
"package/crds/authenticationflow.keycloak.crossplane.io_bindings.yaml"
# These CRDs have identical plural references
Expand Down
18 changes: 18 additions & 0 deletions flake.nix
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,24 @@
'').outPath;
};

# Assert the CRD value accessors (fromCRDModule, crdObjects, and their
# chart variants) agree with the file generator and each other, and
# honor kubeVersion. The check derivation builds iff all pass.
crdAccessorTest = {
type = "app";
program =
let
check = import ./pkgs/generators/equiv-test.nix {
inherit pkgs;
mkEnv = self.lib.mkEnv;
generators = self.packages.${system}.generators;
};
in
(pkgs.writeShellScript "crd-accessor-test" ''
echo "CRD accessor checks passed (${check})"
'').outPath;
};

# Serve docs
docsServe = {
type = "app";
Expand Down
80 changes: 80 additions & 0 deletions pkgs/generators/backend-text.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# Text backend for the shared schema walk (./walk.nix).
#
# Every combinator emits a fragment of Nix *source*. ./generator.nix assembles
# these into a standalone, committable `.nix` module file (the names referenced
# here — `submoduleOf`, `mkOption`, `types`, ... — are resolved by the runtime
# `let` block that generator.nix inlines into that file).
{ lib }:
with lib;
let
# Escape ${...} sequences so they appear as literal text in generated
# Nix "..." strings instead of being parsed as string interpolations.
escapeNixStr = str: builtins.replaceStrings [ "\${" ] [ "\\$\{" ] str;

toNixString =
value:
if isAttrs value || isList value then
escapeNixStr (builtins.toJSON value)
else if isString value then
''"${escapeNixStr value}"''
else if value == null then
"null"
else
builtins.toString value;

removeEmptyLines =
str:
concatStringsSep "\n" (filter (l: builtins.match "[[:space:]]*" l != [ ]) (splitString "\n" str));
in
{
types = {
unspecified = "types.unspecified";
str = "types.str";
int = "types.int";
float = "types.float";
bool = "types.bool";
attrs = "types.attrs";
nullOr = val: "(types.nullOr ${val})";
attrsOf = val: "(types.attrsOf ${val})";
listOf = val: "(types.listOf ${val})";
either = val1: val2: "(types.either ${val1} ${val2})";
loaOf = type: "(types.loaOf ${type})";
# `walk` pre-maps the alternatives, so this receives rendered type strings.
oneOf = ts: "(types.oneOf [${concatStringsSep " " ts}])";
};

mkOption =
{
description ? null,
type ? null,
default ? null,
apply ? null,
}:
removeEmptyLines ''
mkOption {
${optionalString (
description != null
) "description = ${escapeNixStr (builtins.toJSON description)};"}
${optionalString (type != null) "type = ${type};"}
${optionalString (default != null) "default = ${toNixString default};"}
${optionalString (apply != null) "apply = ${apply};"}
}'';

mkOverrideNull = "mkOverride 1002 null";

submoduleOf = ref: ''(submoduleOf "${ref}")'';

globalSubmoduleOf = ref: ''(globalSubmoduleOf "${ref}")'';

submoduleForDefinition =
ref: name: kind: group: version:
''(submoduleForDefinition "${ref}" "${name}" "${kind}" "${group}" "${version}")'';

coerceAttrsOfSubmodulesToListByKey =
ref: attrMergeKey: listMergeKeys:
''(coerceAttrsOfSubmodulesToListByKey "${ref}" "${attrMergeKey}" [${
concatStringsSep " " (map (key: "\"${toString key}\"") listMergeKeys)
}])'';

attrsToList = "attrsToList";
}
22 changes: 22 additions & 0 deletions pkgs/generators/backend-value.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Value backend for the shared schema walk (./walk.nix).
#
# Every combinator returns a live value (an option, a type, a submodule). It
# simply adapts the runtime helpers (./runtime.nix, `rt`) and `lib` into the
# backend interface the walk expects. ./module.nix assembles the walk output
# into a module value directly — no source, no file, no import IFD.
{ lib, rt }:
{
# rt.types is `lib.types` with the custom `str`/`coercedTo` overrides the
# generated modules rely on; it already provides int/listOf/oneOf/loaOf/etc.
inherit (rt)
types
submoduleOf
globalSubmoduleOf
coerceAttrsOfSubmodulesToListByKey
submoduleForDefinition
attrsToList
;

mkOption = lib.mkOption;
mkOverrideNull = lib.mkOverride 1002 null;
}
Loading
Loading