Feature spec: compute extensibility user experience #96
Conversation
Signed-off-by: Will Tsai <[email protected]>
Signed-off-by: Will Tsai <[email protected]>
Signed-off-by: Will Tsai <[email protected]>
Signed-off-by: Will Tsai <[email protected]>
… 2.5 Pro assisted in VSCode) Signed-off-by: Will Tsai <[email protected]>
Signed-off-by: Will Tsai <[email protected]>
|
|
||
| ### Non-goals (out of scope) | ||
| <!-- What are we explicitly not trying to accomplish? --> | ||
| - Running the Radius control plane on a non-Kubernetes platform. |
There was a problem hiding this comment.
This is out of scope for now or forever?
There was a problem hiding this comment.
out of scope for now, added a link to the roadmap item here.
| } | ||
| } | ||
| // This container requests confidential compute | ||
| extensions: [ |
There was a problem hiding this comment.
Do we support extensions on UDTs? Wouldn't these be additional optional properties consumed as recipe inputs instead?
| 1. **Configure Environment-Specific Recipe Parameters (Optional)**: | ||
| * If recipes have parameters that need to be set globally for an environment, configure them using `rad recipe update` or during registration. | ||
|
|
||
| #### Packaging and Registering a "Recipe Pack" for an Environment: |
There was a problem hiding this comment.
This is a fantastic implementation, I like that version is already included.
Signed-off-by: Will Tsai <[email protected]>
|
|
||
| ### Non-goals (out of scope) | ||
| <!-- What are we explicitly not trying to accomplish? --> | ||
| - Running the Radius control plane on a non-Kubernetes platform. |
There was a problem hiding this comment.
Does this mean that with extensibility and core types as UDTs we shouldn't expect to be able to host UCP in a different platform?
|
|
||
| ### Positive user outcome | ||
| <!-- What is the positive outcome for the user if we deliver this, i.e. what is the value proposition for the user? Remember, this is user-centric. --> | ||
| As a platform engineer, I can confidently adopt Radius across my organization, knowing I can extend its capabilities to support any compute platform, secret store, or gateway my teams require, without waiting for built-in support or modifying Radius core. I can register, customize, and share recipes that define how Radius provisions these resources, ensuring consistency with our application definitions while maintaining flexibility in our infrastructure choices and adhering to organizational standards. This empowers my development teams to leverage Radius benefits regardless of the underlying infrastructure. |
There was a problem hiding this comment.
We are talking about containers, secret stores, and gateways a lot. Are these the main resource types that we are focusing on? Why are we not talking about other core resource types?
There was a problem hiding this comment.
Good point, we should include volumes, Will T said extenders are out of scope which I agree as I believe they'd just be replaced by new types/recipes.
There was a problem hiding this comment.
yup, added the inclustion of volume resources too now.
| } | ||
| } | ||
| - providers: { | ||
| - // provider configurations no longer hard-coded here |
There was a problem hiding this comment.
Are we saying we will add info such as subscription and respurce grpup to the bicep recipe itself?
There was a problem hiding this comment.
We already have a requirement from some users that we will need to move those to the environment
There was a problem hiding this comment.
On second thought, this shouldn't be encoded in the recipe. We can keep providers as is for now.
Signed-off-by: Will Tsai <[email protected]>
| * Once the Recipe Pack is registered, the environment is configured with all the specified recipes for the core UDTs. | ||
| * When applications are deployed to this environment, Radius automatically uses the corresponding recipes from the pack to provision `Applications.Core/containers@2025-05-01-preview`, `Applications.Core/gateways@2025-05-01-preview`, and `Applications.Core/secretStores@2025-05-01-preview` resources. | ||
| 1. **Manage and Update Recipe Packs**: | ||
| * Platform engineers can update the Recipe Pack manifest (e.g., point to new recipe versions, change default parameters) and re-register it. The CLI could offer options to overwrite existing registrations or manage versions of the pack within the environment. |
There was a problem hiding this comment.
We need to design/review/implement recipe versioning. Will that be a prerequisite before we work on recipe packs?
There was a problem hiding this comment.
I think Recipes versioning will be a separate feature that we don't need to implement as a part of Recipe Packs. However, we should include versioning in scope for the implementation of Recipe Packs from the get go, even if the underlying Recipes packaged within the Pack don't support versioning yet.
Clarified this in the goals/non-goals
| - } | ||
| ] | ||
| // This container requests confidential compute | ||
| + runtimes: { |
There was a problem hiding this comment.
Should the confidential compute property be part of the ACI-specific recipe? Or is this an interim case while we look at Radius supporting conf containers?
Signed-off-by: Will Tsai <[email protected]>
| 2. **Discover/Create Recipes**: | ||
| * Find community-provided recipes for desired platforms (e.g., ACI, AWS Fargate) from an OCI registry or Radius documentation. | ||
| * Or, create custom Bicep/Terraform recipes for `Applications.Core/containers@2025-05-01-preview`, `Applications.Core/gateways@2025-05-01-preview`, and `Applications.Core/secretStores@2025-05-01-preview` to target a specific platform or customize existing behavior. | ||
| 3. **Register Recipes for Core Types**: |
There was a problem hiding this comment.
Is there a way to constraint certain app files to be used against only certain recipe? Example: App with confidential container property set to true, should be allowed only on confidential recipe.
There was a problem hiding this comment.
Not currently, we discussed this as well as the ability to only allow certain apps to use certain options/properties in a recipe.
|
|
||
| ## Topic Summary | ||
| <!-- A paragraph or two to summarize the topic area. Just define it in summary form so we all know what it is. --> | ||
| Radius will be enhanced to support multiple compute platforms, secret stores, and gateway resources through a recipe-based extensibility model. This approach decouples Radius's core logic from platform-specific provisioning code. Core resource types (`containers`, `gateways`, and `secretStores`) will be implemented as User-Defined Types (UDTs) and will allow platform engineers to register Bicep or Terraform recipes for them. Radius will provide default recipes for Kubernetes and Azure Container Instances (ACI), but platform engineers can use, modify, or replace these to customize how Radius provisions resources to different environments, or to add support for entirely new platforms without requiring changes to Radius core. |
There was a problem hiding this comment.
Shouldn't core resource types include volumes?
There was a problem hiding this comment.
yes, it should. i'll add it.
| } | ||
| } | ||
| } | ||
| - providers: { |
There was a problem hiding this comment.
The provider stores the AWS and Azure details that get past to the recipe context right? https://docs.radapp.io/reference/context-schema/#environment So we'll need the AWS account and region, and Azure subscription and resource group in the environment somewhere.
There was a problem hiding this comment.
Yes, good point, this shouldn't be encoded in the recipe. We can keep providers in the environment as is for now.
| * The platform engineer configures `std-env` with recipes for `Applications.Core/containers@2025-05-01-preview` that deploy to standard compute (e.g., regular ACI or Kubernetes pods). These recipes might ignore or log a warning for confidential container requests if they don't support them. | ||
| * Example recipe registration (conceptual): | ||
| ```bash | ||
| rad recipe register std-container-recipe --environment std-env \ |
There was a problem hiding this comment.
I strongly believe we should not have any recipe names. This leaks the implementation abstraction into the developers' world. If the developer needs a different recipe, that should be a different resource type.
| // This container requests confidential compute | ||
| extensions: [ | ||
| { | ||
| kind: 'confidentialCompute' // Example extension kind |
There was a problem hiding this comment.
If there are additional properties needed to be set by the developer, that should be a different resource type. I do not think extension is the right thing here, nor do I understand how one would create an extension.
| 1. **Configure Environment-Specific Recipe Parameters (Optional)**: | ||
| * If recipes have parameters that need to be set globally for an environment, configure them using `rad recipe update` or during registration. | ||
|
|
||
| #### Packaging and Registering a "Recipe Pack" for an Environment: |
There was a problem hiding this comment.
A "resource pack" would be more useful. It would be a manifest of resource types and associated recipes. I would specify the resource pack during rad init.
There was a problem hiding this comment.
Could we define a set of resources and a set of recipes within a single environment resource, and deploy that? That could replace the need for recipe or resource packs.
There was a problem hiding this comment.
@brooke-hamilton see answer in a previous comment: https://github.com/radius-project/design-notes/pull/96/files#r2141274374
@zachcasper - I'm thinking Recipes and Resources would be bundled into the same pack - it seems pointless to have them separately as one wouldn't be used without the other. Thoughts?
| ``` | ||
| 1. **Package the Recipe Pack (Optional but Recommended)**: | ||
| * The manifest file and any local recipe files (if not using OCI URIs exclusively) could be bundled into an OCI artifact or a simple archive (e.g., .zip, .tar.gz) for easier distribution and versioning. | ||
| 1. **Register the Recipe Pack to an Environment**: |
There was a problem hiding this comment.
Register isn't the right verb here. We're configuring the environment. rad environment create --recipe-pack <location> seems more accurate.
There was a problem hiding this comment.
Good point, I've changed it to leverage the rad environment update my-env --recipe-pack ... command instead.
Signed-off-by: Will Tsai <[email protected]>
Signed-off-by: Will Tsai <[email protected]>
Signed-off-by: Will Tsai <[email protected]>
Signed-off-by: Will Tsai <[email protected]>
Signed-off-by: Will Tsai <[email protected]>
Signed-off-by: Will Tsai <[email protected]>
Co-authored-by: Zach Casper <[email protected]> Signed-off-by: Will <[email protected]>
There was a problem hiding this comment.
High level thoughts:
- What are the proposed schema changes for the core types? I would expect the new schema to be defined in this doc.
- I'm very confused and concerned about
allowAdvancedContainerCapabilitiesbut there are not a lot of details here. I'm not supportive of an unstructured map. We should find the platform-specific properties that we want to expose for ACI and Kubernetes and put them in the extensions block. - What does a vanilla environment look like? Does it have all of the recipes enumerated?
- Where are the recipes stored? How does that work in an air-gapped environment.
- We need a user story for modifying the schema for a core type. I can use rad resource-type create but I will need the original resource definition. Today there is no way to get the current definition.
Co-authored-by: Zach Casper <[email protected]> Signed-off-by: Will <[email protected]>
Signed-off-by: Will Tsai <[email protected]>
…com/willtsai/design-notes-radius into compute-extensibility-feature-specs
Signed-off-by: Will Tsai <[email protected]>
Signed-off-by: Will Tsai <[email protected]>
Signed-off-by: Will Tsai <[email protected]>
Signed-off-by: Will Tsai <[email protected]>
Signed-off-by: Will Tsai <[email protected]>
Signed-off-by: Will Tsai <[email protected]>
Signed-off-by: Will Tsai <[email protected]>
|
|
||
| * The default Recipe packs will have the `allowPlatformOptions` parameter set to `true` for core types like `Applications.Core/containers@2025-05-01-preview`, which allows platform engineers to punch through the Radius abstraction and use platform-specific options (e.g., `containerGroupProfile` for ACI) in their application definitions. | ||
|
|
||
| * By default, `rad init` will register recipes for Kubernetes provisioning for core types like `Applications.Core/containers@2025-05-01-preview` so that Radius may continue providing a local kubernetes experience out of the box. |
There was a problem hiding this comment.
so if a user wants to use a different compute platform, it's expected that the user registers an additional recipe pack for ACI, ECS, etc?
There was a problem hiding this comment.
nvm, just saw the point below. why don't we offer the choice between compute platforms as a user input for rad init?
There was a problem hiding this comment.
correct - initially i had considered expanding rad init experiences to be per platform, e.g. rad init kubernetes and rad init aci but ultimately decided against it as rad init is meant to be for a quickstart rather than setting up production like environments.
There was a problem hiding this comment.
potentially introduce ACI, ECS, etc. as a part of the rad init --full experience
There was a problem hiding this comment.
Great progress. I made several smaller comments, but the big changes I think we need to make are:
- A recipe pack should be a resource. Radius should store its configuration within Radius. Storing configuration outside the system introduces external dependencies and raises the risk of things being out of sync.
- secretStores should be renamed secrets.
- I still don't understand what is going on with volumes.
| | Term | Definition | | ||
| |------|------------| | ||
| | **Compute platform** | An environment where applications can be deployed and run, such as Kubernetes, Azure Container Instances (ACI), etc. | | ||
| | **Core types** | Built-in resource types provided by Radius, including `containers`, `gateways`, `secretStores`, `volumes`, `environments`, and `applications`. | |
There was a problem hiding this comment.
We should take this opportunity to rename secretStores to secrets.
secretStore is a platform engineer resource where the platform engineer has configured an infrastructure resource to store secrets. A secret is something a developer would use in his/her application.
| * Example `recipe-pack.yaml`: | ||
| ```yaml | ||
| name: aci-production-pack | ||
| version: 1.0.0 |
There was a problem hiding this comment.
I see you have decorated the recipe pack schema with several metadata fields like version and description. If we foresee having some functionality around these fields (possible related to rad recipe show) then great. Otherwise, comments in the YAML file should be sufficient. Specifically:
version – I do not expect Radius to have versioning on a recipe pack. The recipe itself could have a version that is specified in the recipeLocation (as you have shown on line 139). But the pack would not be versioned. I would remove this.
description – We don't have a description field today on the environment.recipe. If we thought there was a use case for a developer browsing the environment details and seeing the recipes I would understand the need to capture a description, but my preference is to limit the amount of recipe details exposed to the develop. I would also remove description. The platform engineer can simply use comments in the YAML.
| ``` | ||
| > Note: `templateKind` is changed to `recipeKind` and `templatePath` is changed to `recipeLocation` | ||
|
|
||
| 1. **Add the Recipe Pack to an Environment**: |
There was a problem hiding this comment.
We need to give some thought to what configuration data is stored within the Radius data store and what is referenced externally. I would argue that Radius should store all configuration data internally to minimize external dependencies. However, there is a GitOps school of thought where all configuration is stored in a Git repo. The key is consistency. Today, Radius is inconsistent since we reference recipes external to Radius (this actually causes lots of problems if the recipe is ever changed, we have a high risk of resources and recipes getting out of sync).
I recommend recipe packs being a resource type (so it's is stored within the Radius data store). The recipe pack resource would look like:
resource kubernetesCore 'Applications.Core/recipePack@2025-05-01-preview' = {
name: 'kubernetesCore'
properties: {
recipes: [
Applications.Core/containers@2025-05-01-preview: {
# Note that the recipe does not have a name since it is not referenceable in the application definition
recipeKind: 'terraform'
recipeLocation: 'https://github.com/project-radius/resource-types-contrib.git/recipes/core/containers?ref=v1.2.0'
parameters: {
allowPlatformOptions: true
}
Applications.Core/gateways@2025-05-01-preview: {
recipeKind: 'terraform'
recipeLocation: 'https://github.com/project-radius/resource-types-contrib.git/recipes/core/gateways?ref=v1.2.0'
Applications.Core/secrets@2025-05-01-preview: {
recipeKind: 'terraform'
recipeLocation: 'https://github.com/project-radius/resource-types-contrib.git/recipes/core/secrets?ref=v1.2.0'
}
]
}
}
| description: "Recipe Pack for deploying to ACI in production." | ||
| recipes: | ||
| - resourceType: "Applications.Core/containers@2025-05-01-preview" | ||
| name: "aci-prod-container" # Optional: a friendly name for this recipe registration |
There was a problem hiding this comment.
We should not include a name for the recipe. Resource types intensionally do not have the ability for developers to specify the recipe name.
| > Note: if there are existing recipes registered for the same resource type in the environment, this command will overwrite them with the recipes defined in the pack. | ||
|
|
||
| 1. **List the Registered Recipes**: | ||
| * Use `rad recipe list --environment <env-name>` to see all registered recipes for core resource types in the specified environment. |
There was a problem hiding this comment.
Can we get an example here? I'm imagining:
$ rad recipe list
RECIPE PACK RESOURCE TYPE KIND LOCATION
kubernetesCore Applications.Core/containers Terraform https://github.com/project/radius/resource-type-contrib.git/recipes/core/kubernetes/containers?ref=v1.2.0
# Note the name argument is no longer required
$ rad recipe show --resource-type Applications.Core/containers
RECIPE PACK RESOURCE TYPE KIND LOCATION
kubernetesCore Applications.Core/containers Terraform https://github.com/project/radius/resource-type-contrib.git/recipes/core/kubernetes/containers?ref=v1.2.0
PARAMETER TYPE DEFAULT
allowPlatformOptions BOOL TRUE
| #### User Story 7: As a platform engineer or app developer, I want to migrate existing Radius applications to use the new RRT-based core types and recipes, so that I can take advantage of the new extensibility features without breaking existing applications: | ||
|
|
||
| 1. **Understand the New Model**: | ||
| * Platform engineers and developers review documentation on the RRT-based core types (`Applications.Core/containers@2025-05-01-preview`, `Applications.Core/gateways@2025-05-01-preview`, `Applications.Core/secretStores@2025-05-01-preview`, `Applications.Core/volumes@2025-05-01-preview`) and the recipe-driven approach. |
There was a problem hiding this comment.
I'd like to have a serious conversation about the -preview in the version. Do we honestly anticipate making changes to the schema for these core resources? I feel like they are fairly stable. If we think they will change in the future, we should include the preview, but if not, let's remove.
|
|
||
| > This approach will require deprecation of the current [Azure KeyVault Radius Volumes resource type](https://docs.radapp.io/reference/resource-schema/core-schema/volumes/azure-keyvault/) where the Azure KeyVault will just be a Secret Store resource type provisioned using a recipe for Azure KeyVault that can be mounted to a Radius Container resource as a volume. | ||
|
|
||
| #### User Story 9: As an application developer, I want to create custom gateway resources and configure them to route traffic to my application containers, so that I can manage ingress traffic using my organization's preferred gateway solution: |
There was a problem hiding this comment.
What is a custom gateway? I think what you are saying is:
As a developer, I need to route ingress traffic to one of my containers on a particular port. My platform engineer has configured the platform to handle ingress traffic, so I only need to specify the route.
| Public endpoint http://1.1.1.1.nip.io/ | ||
| ``` | ||
|
|
||
| > Note: Contour is currently installed as a [hard dependency for Radius](https://github.com/radius-project/radius/blob/main/pkg/kubernetes/object.go#L27) to provide http routing for Kubernetes deployments. This change allows users to disable the Contour installation in their Radius environments if they want to use a different gateway solution, such as NGINX Ingress Controller or Traefik, by registering a custom recipe for `Applications.Core/gateways@2025-05-01-preview` that provisions the desired gateway solution. |
There was a problem hiding this comment.
| > Note: Contour is currently installed as a [hard dependency for Radius](https://github.com/radius-project/radius/blob/main/pkg/kubernetes/object.go#L27) to provide http routing for Kubernetes deployments. This change allows users to disable the Contour installation in their Radius environments if they want to use a different gateway solution, such as NGINX Ingress Controller or Traefik, by registering a custom recipe for `Applications.Core/gateways@2025-05-01-preview` that provisions the desired gateway solution. | |
| The developer UX is the same as today. However, several changes are implemented in the default gateway recipe implementation: | |
| 1. Contour is no longer installed and the `--skip-contout-install` option on `rad install` is removed | |
| 2. The gateway recipe, part of the kubernetesCore, implements L4 ingress only using a Kubernetes Service of type `LoadBalancer` | |
| 3. The gateway recipe, part of aciCore`, implements L7 ingress using Azure Application Gateway | |
| In the future, the gateway recipe in ecsCore will use AWS Application Load Balancer. | |
| Platform engineers can modify the gateway recipe to use other ingress options such as NGINX or Cilium on Kubernetes, or Azure Front Door with ACI. |
| Given: my platform engineer has set up a Radius environment with recipes registered for `Applications.Core/Containers@2025-05-01-preview` and `Applications.Core/volumes@2025-05-01-preview` resources. The volume recipe is configured to provision custom volumes based on the specifications determined by my organization. | ||
|
|
||
| 1. **Define a Custom Volume Resource**: | ||
| * The developer defines a custom volume resource in their application Bicep file that will be provisioned by the default recipe. |
There was a problem hiding this comment.
This user story doesn't quite make sense to me. How is this different from the volumes property on the container resource? Maybe if there were properties on the volume resource it would make more sense.
Co-authored-by: Zach Casper <[email protected]> Signed-off-by: Will <[email protected]>
Co-authored-by: Zach Casper <[email protected]> Signed-off-by: Will <[email protected]>
Add feature spec for compute extensibility to define requirements and user experience