Skip to content

{AKS} az aks create/update: add --enable/--disable-control-plane-metrics - #33537

Merged
Julie Zhu (yanzhudd) merged 6 commits into
Azure:devfrom
bragi92:kadubey/aks-control-plane-metrics
Jun 16, 2026
Merged

Julie Zhu (yanzhudd) merged 6 commits into
Azure:devfrom
bragi92:kadubey/aks-control-plane-metrics

Conversation

@bragi92

@bragi92 bragi92 commented Jun 11, 2026 •

Copy link
Copy Markdown
Member

Surface azureMonitorProfile.metrics.controlPlane.enabled so users can opt clusters in/out of Azure Monitor managed Prometheus control-plane metrics (controlplane-apiserver, controlplane-etcd) via the first-class API property — replaces the AFEC-gated preview.

New flags:

  • az aks create: --enable-control-plane-metrics (--enable-cp-metrics)
  • az aks update: --enable-control-plane-metrics (--enable-cp-metrics)
  • az aks update: --disable-control-plane-metrics (--disable-cp-metrics)

Enable requires Azure Monitor metrics to already be on or to be enabled in the same command via --enable-azure-monitor-metrics. Enable + disable in the same command, or enable-CP + --disable-azure-monitor-metrics, are rejected client-side with MutuallyExclusiveArgumentError.

Greenfield race fix:
On aks create, metrics.controlPlane.enabled=true is intentionally NOT set on the initial cluster PUT. Otherwise the RP would schedule the control-plane-metrics collection (CCP) pod before the DCRA is created in postprocessing (link_azure_monitor_profile_artifacts), causing the CCP pod to crash-loop with "DCRA not found" until reconciliation. The flip is deferred to the existing post-DCRA addon_put PUT, so the CCP pod is scheduled only after its DCRA exists. The update path is unchanged — brownfield updates target a cluster whose DCRA already exists, so there is no race.

Related command

  • az aks create
  • az aks update

Description

This PR plumbs the new azureMonitorProfile.metrics.controlPlane.enabled API property end-to-end through the AKS CLI.

What's added:

  • New --enable-control-plane-metrics / --enable-cp-metrics flag on az aks create and az aks update.
  • New --disable-control-plane-metrics / --disable-cp-metrics flag on az aks update.
  • Argument registration in _params.py and help text in _help.py.
  • Client-side validation in _validators.py:
    • Enable requires azureMonitorProfile.metrics.enabled=true on the cluster (or --enable-azure-monitor-metrics in the same command).
    • --enable-control-plane-metrics + --disable-control-plane-metrics → MutuallyExclusiveArgumentError.
    • --enable-control-plane-metrics + --disable-azure-monitor-metrics → MutuallyExclusiveArgumentError.
  • AKSManagedClusterContext getters in managed_cluster_decorator.py and routing in custom.py so create/update both populate the correct subfield of ManagedClusterAzureMonitorProfileMetrics.
  • Greenfield race fix (see above): on aks create the CP-metrics flip is held back from the initial PUT and applied after link_azure_monitor_profile_artifacts provisions the DCRA, then re-PUT via the existing addon_put path. Brownfield aks update is unchanged.

Scenario coverage:

# Command Cluster state Result
1 aks create --enable-azure-monitor-metrics --enable-control-plane-metrics n/a Cluster created, AMW linked, DCRA created, then CP-metrics flipped on
2 aks create --enable-control-plane-metrics (no AMW flag) n/a Rejected: AMW required
3 aks update --enable-control-plane-metrics AMW already on CP-metrics enabled
4 aks update --enable-control-plane-metrics AMW off Rejected: AMW required
5 aks update --enable-azure-monitor-metrics --enable-control-plane-metrics … AMW off AMW enabled + CP-metrics enabled in one call
6 aks update --disable-control-plane-metrics AMW on, CP on CP-metrics disabled, AMW left intact
7 aks update --enable-control-plane-metrics --disable-control-plane-metrics any Rejected: mutually exclusive
8 aks update --enable-control-plane-metrics --disable-azure-monitor-metrics any Rejected: mutually exclusive

Files changed:

  • src/azure-cli/azure/cli/command_modules/acs/_help.py
  • src/azure-cli/azure/cli/command_modules/acs/_params.py
  • src/azure-cli/azure/cli/command_modules/acs/_validators.py
  • src/azure-cli/azure/cli/command_modules/acs/custom.py
  • src/azure-cli/azure/cli/command_modules/acs/managed_cluster_decorator.py
  • src/azure-cli/azure/cli/command_modules/acs/tests/latest/test_managed_cluster_decorator.py
  • src/azure-cli/azure/cli/command_modules/acs/tests/latest/recordings/… (recorded test cassettes)

Pairs with the matching aks-preview change: Azure/azure-cli-extensions#9931.

Testing Guide

Unit tests:

azdev test acs --discover
azdev test acs --series --pytest-args "-k control_plane_metrics"

Live validation against a real AKS cluster + Azure Monitor workspace:

RG=ccp-test-rg
LOC=eastus
AMW=/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Monitor/accounts/<amw>
GRAFANA=/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Dashboard/grafana/<g>

# Greenfield: enable CP-metrics at create time
az aks create -g $RG -n green-cp --location $LOC \
  --enable-azure-monitor-metrics --azure-monitor-workspace-resource-id $AMW \
  --grafana-resource-id $GRAFANA \
  --enable-control-plane-metrics
az aks show -g $RG -n green-cp --query "azureMonitorProfile.metrics" -o jsonc

# Brownfield: enable, then disable, then re-enable
az aks create -g $RG -n brown-cp --location $LOC \
  --enable-azure-monitor-metrics --azure-monitor-workspace-resource-id $AMW \
  --grafana-resource-id $GRAFANA
az aks update -g $RG -n brown-cp --enable-control-plane-metrics
az aks show -g $RG -n brown-cp --query "azureMonitorProfile.metrics" -o jsonc
az aks update -g $RG -n brown-cp --disable-control-plane-metrics
az aks show -g $RG -n brown-cp --query "azureMonitorProfile.metrics" -o jsonc
az aks update -g $RG -n brown-cp --enable-control-plane-metrics
az aks show -g $RG -n brown-cp --query "azureMonitorProfile.metrics" -o jsonc

# Negative cases
az aks update -g $RG -n brown-cp --enable-control-plane-metrics --disable-control-plane-metrics
az aks update -g $RG -n brown-cp --enable-control-plane-metrics --disable-azure-monitor-metrics
az aks create -g $RG -n bad-cp --enable-control-plane-metrics   # no AMW => rejected

Validation in Azure Monitor workspace after each enable: default CCP metric families flow within ~5–10 min (apiserver_request_total, apiserver_request_duration_seconds_*, etcd_server_has_leader, etcd_mvcc_db_total_size_in_bytes, process_start_time_seconds). After disable, allow ~15 min for the previous deployment's metrics to age out before re-asserting.

History Notes

[AKS] az aks create: Add --enable-control-plane-metrics/--enable-cp-metrics to opt new clusters into Azure Monitor managed Prometheus control-plane metrics
[AKS] az aks update: Add --enable-control-plane-metrics/--enable-cp-metrics and --disable-control-plane-metrics/--disable-cp-metrics to toggle Azure Monitor managed Prometheus control-plane metrics on existing clusters


This checklist is used to make sure that common guidelines for a pull request are followed.

Surface `azureMonitorProfile.metrics.controlPlane.enabled` so users can
opt clusters in/out of Azure Monitor managed Prometheus control-plane
metrics (kube-apiserver, etcd, scheduler, controller-manager) via the
first-class API property — replaces the AFEC-gated preview.

New flags:
  az aks create:  --enable-control-plane-metrics  (--enable-cp-metrics)
  az aks update:  --enable-control-plane-metrics  (--enable-cp-metrics)
                  --disable-control-plane-metrics (--disable-cp-metrics)

Enable requires Azure Monitor metrics to already be on or to be enabled
in the same command via --enable-azure-monitor-metrics. Enable + disable
in the same command, or enable-CP + --disable-azure-monitor-metrics,
are rejected client-side with MutuallyExclusiveArgumentError.

Greenfield race fix:
On `aks create`, `metrics.controlPlane.enabled=true` is intentionally
NOT set on the initial cluster PUT. Otherwise the RP would schedule the
control-plane-metrics collection (CCP) pod before the DCRA is created
in postprocessing (link_azure_monitor_profile_artifacts), causing the
CCP pod to crash-loop with "DCRA not found" until reconciliation. The
flip is deferred to the existing post-DCRA addon_put PUT, so the CCP
pod is scheduled only after its DCRA exists. The update path is
unchanged — brownfield updates target a cluster whose DCRA already
exists, so there is no race.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@azure-client-tools-bot-prd

azure-client-tools-bot-prd Bot commented Jun 11, 2026 •

Copy link
Copy Markdown
️✔️AzureCLI-FullTest
️✔️acr
️✔️latest
️✔️3.12
️✔️3.14
️✔️acs
️✔️latest
️✔️3.12
️✔️3.14
️✔️advisor
️✔️latest
️✔️3.12
️✔️3.14
️✔️ams
️✔️latest
️✔️3.12
️✔️3.14
️✔️apim
️✔️latest
️✔️3.12
️✔️3.14
️✔️appconfig
️✔️latest
️✔️3.12
️✔️3.14
️✔️appservice
️✔️latest
️✔️3.12
️✔️3.14
️✔️aro
️✔️latest
️✔️3.12
️✔️3.14
️✔️backup
️✔️latest
️✔️3.12
️✔️3.14
️✔️batch
️✔️latest
️✔️3.12
️✔️3.14
️✔️batchai
️✔️latest
️✔️3.12
️✔️3.14
️✔️billing
️✔️latest
️✔️3.12
️✔️3.14
️✔️botservice
️✔️latest
️✔️3.12
️✔️3.14
️✔️cdn
️✔️latest
️✔️3.12
️✔️3.14
️✔️cloud
️✔️latest
️✔️3.12
️✔️3.14
️✔️cognitiveservices
️✔️latest
️✔️3.12
️✔️3.14
️✔️compute_recommender
️✔️latest
️✔️3.12
️✔️3.14
️✔️computefleet
️✔️latest
️✔️3.12
️✔️3.14
️✔️config
️✔️latest
️✔️3.12
️✔️3.14
️✔️configure
️✔️latest
️✔️3.12
️✔️3.14
️✔️consumption
️✔️latest
️✔️3.12
️✔️3.14
️✔️container
️✔️latest
️✔️3.12
️✔️3.14
️✔️containerapp
️✔️latest
️✔️3.12
️✔️3.14
️✔️core
️✔️latest
️✔️3.12
️✔️3.14
️✔️cosmosdb
️✔️latest
️✔️3.12
️✔️3.14
️✔️databoxedge
️✔️latest
️✔️3.12
️✔️3.14
️✔️dls
️✔️latest
️✔️3.12
️✔️3.14
️✔️dms
️✔️latest
️✔️3.12
️✔️3.14
️✔️eventgrid
️✔️latest
️✔️3.12
️✔️3.14
️✔️eventhubs
️✔️latest
️✔️3.12
️✔️3.14
️✔️feedback
️✔️latest
️✔️3.12
️✔️3.14
️✔️find
️✔️latest
️✔️3.12
️✔️3.14
️✔️hdinsight
️✔️latest
️✔️3.12
️✔️3.14
️✔️identity
️✔️latest
️✔️3.12
️✔️3.14
️✔️iot
️✔️latest
️✔️3.12
️✔️3.14
️✔️keyvault
️✔️latest
️✔️3.12
️✔️3.14
️✔️lab
️✔️latest
️✔️3.12
️✔️3.14
️✔️managedservices
️✔️latest
️✔️3.12
️✔️3.14
️✔️maps
️✔️latest
️✔️3.12
️✔️3.14
️✔️marketplaceordering
️✔️latest
️✔️3.12
️✔️3.14
️✔️monitor
️✔️latest
️✔️3.12
️✔️3.14
️✔️mysql
️✔️latest
️✔️3.12
️✔️3.14
️✔️netappfiles
️✔️latest
️✔️3.12
️✔️3.14
️✔️network
️✔️latest
️✔️3.12
️✔️3.14
️✔️policyinsights
️✔️latest
️✔️3.12
️✔️3.14
️✔️postgresql
️✔️latest
️✔️3.12
️✔️3.14
️✔️privatedns
️✔️latest
️✔️3.12
️✔️3.14
️✔️profile
️✔️latest
️✔️3.12
️✔️3.14
️✔️rdbms
️✔️latest
️✔️3.12
️✔️3.14
️✔️redis
️✔️latest
️✔️3.12
️✔️3.14
️✔️relay
️✔️latest
️✔️3.12
️✔️3.14
️✔️resource
️✔️latest
️✔️3.12
️✔️3.14
️✔️role
️✔️latest
️✔️3.12
️✔️3.14
️✔️search
️✔️latest
️✔️3.12
️✔️3.14
️✔️security
️✔️latest
️✔️3.12
️✔️3.14
️✔️servicebus
️✔️latest
️✔️3.12
️✔️3.14
️✔️serviceconnector
️✔️latest
️✔️3.12
️✔️3.14
️✔️servicefabric
️✔️latest
️✔️3.12
️✔️3.14
️✔️signalr
️✔️latest
️✔️3.12
️✔️3.14
️✔️sql
️✔️latest
️✔️3.12
️✔️3.14
️✔️sqlvm
️✔️latest
️✔️3.12
️✔️3.14
️✔️storage
️✔️latest
️✔️3.12
️✔️3.14
️✔️synapse
️✔️latest
️✔️3.12
️✔️3.14
️✔️telemetry
️✔️latest
️✔️3.12
️✔️3.14
️✔️util
️✔️latest
️✔️3.12
️✔️3.14
️✔️vm
️✔️latest
️✔️3.12
️✔️3.14

@azure-client-tools-bot-prd

azure-client-tools-bot-prd Bot commented Jun 11, 2026 •

Copy link
Copy Markdown
⚠️AzureCLI-BreakingChangeTest
⚠️acs
rule cmd_name rule_message suggest_message
⚠️ 1006 - ParaAdd aks create cmd aks create added parameter enable_control_plane_metrics
⚠️ 1006 - ParaAdd aks update cmd aks update added parameter disable_control_plane_metrics
⚠️ 1006 - ParaAdd aks update cmd aks update added parameter enable_control_plane_metrics

@yonzhan

Copy link
Copy Markdown
Collaborator

AKS

@bragi92
bragi92 marked this pull request as ready for review June 11, 2026 23:41
Copilot AI review requested due to automatic review settings June 11, 2026 23:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR adds support for enabling/disabling Azure Monitor managed Prometheus control plane metrics for AKS clusters, with validation and create-flow handling to avoid a DCRA/CCP race.

Changes:

  • Introduces new CLI flags (--enable-control-plane-metrics/--enable-cp-metrics, --disable-control-plane-metrics/--disable-cp-metrics) and wires them into create/update flows.
  • Adds validation to reject invalid flag combinations and require Azure Monitor metrics as a prerequisite.
  • Updates postprocessing to defer enabling control plane metrics on create until after DCRA creation, and adds unit + live tests.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
src/azure-cli/azure/cli/command_modules/acs/managed_cluster_decorator.py Adds getters/validation for control plane metrics flags; defers control plane enablement on create; allows toggling on update.
src/azure-cli/azure/cli/command_modules/acs/azuremonitormetrics/azuremonitorprofile.py Adds a create-flow addon PUT variant that flips metrics.controlPlane.enabled after DCRA creation.
src/azure-cli/azure/cli/command_modules/acs/custom.py Plumbs new parameters into aks_create / aks_update entrypoints.
src/azure-cli/azure/cli/command_modules/acs/_params.py Registers new CLI arguments and help text.
src/azure-cli/azure/cli/command_modules/acs/_help.py Documents the new flags in command help.
src/azure-cli/azure/cli/command_modules/acs/tests/latest/test_managed_cluster_decorator.py Adds unit tests for create/update validation and payload shaping.
src/azure-cli/azure/cli/command_modules/acs/tests/latest/test_aks_commands.py Adds live tests covering create/update/negative cases for the new flags.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/azure-cli/azure/cli/command_modules/acs/managed_cluster_decorator.py Outdated
Comment thread src/azure-cli/azure/cli/command_modules/acs/managed_cluster_decorator.py Outdated
- Wait on the LRO in _addon_put_with_control_plane via poller.result(). This is the
  only place controlPlane.enabled is set during the greenfield create flow, so the
  CP flip must be durably persisted before the create command returns. Without the
  wait, callers and tests that read the cluster immediately could observe the
  pre-flip state. (The sibling addon_put intentionally remains fire-and-forget
  because metrics.enabled was already persisted on the initial cluster PUT.)
- Replace raise UnknownError(e) with raise UnknownError(str(e)) from e so the
  message is readable and the original traceback is preserved.
- Coerce _get_enable_control_plane_metrics / _get_disable_control_plane_metrics
  return values to bool() to match the declared -> bool return type when the
  parameter dict omits the key.
- Make the live test_aks_create_with_control_plane_metrics assertion robust:
  the controlPlane.enabled check is moved out of the immediate create response
  into an explicit aks show after aks wait, since the flip is intentionally
  deferred to post-DCRA postprocessing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@FumingZhang

Copy link
Copy Markdown
Member

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 3 pipeline(s).

@FumingZhang FumingZhang left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fine with approving the change since the test result was positive, but I have to admit that the monitoring addon configuration has become quite confusing for me. It's difficult to understand all the options and the control flow between the v1 and v2 versions. I believe I've mentioned this before—do you and your team have any plans to refactor this part of the code?

# Trigger control-plane-metrics validation even if the parent metrics flag was
# not specified, so users get a clear error instead of silent ignore when they
# pass --enable-control-plane-metrics on its own.
self.context.get_enable_control_plane_metrics()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps you could separate the validation logic from the getter and call the validator here instead, as it's unusual to retrieve a value only to discard it.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated

mc.azure_monitor_profile.metrics.kube_state_metrics = self.models.ManagedClusterAzureMonitorProfileKubeStateMetrics( # pylint:disable=line-too-long
metric_labels_allowlist=str(ksm_metric_labels_allow_list),
metric_annotations_allow_list=str(ksm_metric_annotations_allow_list))
# NOTE: control_plane.enabled is intentionally NOT set here on the create flow.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This approach is somewhat unconventional from a client perspective. I've encountered some CRIs who reported that the monitoring addon wasn't functioning as expected when deploying with an ARM template or similar methods. Would it be possible to handle these types of tasks on the server side instead? This allows users on different clients to have a consistent experience.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, agreed. We've had this in out backlog for a while. The logs and prometheus addon team has been looking into it this semester and currently Rashmi has been working on the design for that.

Once the logic is moved to the RP, I'll create a PR to remove it from the CLI.

@bragi92

bragi92 commented Jun 12, 2026

Copy link
Copy Markdown
Member Author

I'm fine with approving the change since the test result was positive, but I have to admit that the monitoring addon configuration has become quite confusing for me. It's difficult to understand all the options and the control flow between the v1 and v2 versions. I believe I've mentioned this before—do you and your team have any plans to refactor this part of the code?

Yup, Rashmi has been working on moving the creation logic for most of the dependencies to the AKS-RP. This should alleviate the pain of maintaing it in the CLI, UX and make our ARM, Bicep etc. teamplates simpler.

Per FumingZhang review feedback on PR Azure#33537: calling get_enable_control_plane_metrics() purely to trigger validation and discarding the return value is a confusing pattern. Extract the validation block into a new private _validate_control_plane_metrics_params method, expose a public validate_control_plane_metrics_params, and have the getters delegate to it when enable_validation=True (preserves existing API). The set_up_azure_monitor_profile call site now calls the validator directly instead of discarding a getter result.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
bragi92 added a commit to bragi92/azure-cli-extensions that referenced this pull request Jun 12, 2026
Per FumingZhang review feedback on Azure/azure-cli#33537: calling get_enable_control_plane_metrics() purely to trigger validation and discarding the return value is a confusing pattern. Extract the validation block into a new private _validate_control_plane_metrics_params method, expose a public validate_control_plane_metrics_params, and have the getters delegate to it when enable_validation=True (preserves existing API). The two _setup_azure_monitor_profile call sites now call the validator directly instead of discarding a getter result.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
bragi92 and others added 3 commits June 12, 2026 08:31
Other validators in this file (e.g. validate_byo_hosted_system_subnets) are a single public def validate_xxx(self) -> None — no private companion. Collapse the extra _validate_control_plane_metrics_params indirection so the new validator matches the file's convention. Tests + behavior unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The aka.ms/aks/controlplane-metrics shortlink does not resolve. Drop the trailing reference from the four help strings (create + update, both _help.py and _params.py). The remaining help text already explains the flag and its prerequisite, matching the sibling --enable-azure-monitor-metrics line which has no docs URL.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace 'kube-apiserver, etcd, etc' with the actual default Prometheus scrape job names: controlplane-apiserver and controlplane-etcd. These are the targets users see in AMW and what the AKS docs reference. The 'etc' was also misleading since scheduler / controller-manager / NAP targets are opt-in via MinimalIngestionProfile and are not flipped on by this flag.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@bragi92

bragi92 commented Jun 12, 2026

Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Commenter does not have sufficient privileges for PR 33537 in repo Azure/azure-cli

@FumingZhang

Copy link
Copy Markdown
Member

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 3 pipeline(s).

@FumingZhang FumingZhang changed the title {AKS} az aks create/update: add --enable/--disable-control-plane-metrics {AKS} az aks create/update: add --enable/--disable-control-plane-metrics Jun 15, 2026
@FumingZhang

Copy link
Copy Markdown
Member

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 3 pipeline(s).

@yanzhudd
Julie Zhu (yanzhudd) merged commit 1d68d09 into Azure:dev Jun 16, 2026
50 checks passed
bragi92 added a commit to Azure/prometheus-collector that referenced this pull request Aug 24, 2026
…oarding templates (ARM/Bicep/Terraform/Policy) (#1590)

## What

Plumbs the new AKS API property
**`azureMonitorProfile.metrics.controlPlane.enabled`** through the four
onboarding IaC templates in this repo, so clusters can be opted in/out
of Azure Monitor managed Prometheus **control-plane** metrics (default
targets: `controlplane-apiserver`, `controlplane-etcd`) declaratively.

Pairs with the azure-cli change **Azure/azure-cli#33537**
(`--enable/--disable-control-plane-metrics`).

## Changes

| Template | Change |
|---|---|
| **ARM** (`AddonArmTemplate`) | New `enableControlPlaneMetrics`
parameter (default `false`) + `controlPlane.enabled` under
`azureMonitorProfile.metrics`; bumped the managed-cluster PUT apiVersion
`2023-01-01` → `2026-06-01`. Added to the parameters file. |
| **Bicep** (`AddonBicepTemplate`) | New `enableControlPlaneMetrics`
param threaded into the nested profile module; `controlPlane.enabled`
added; apiVersion bumped. Added to parameters file. |
| **Policy** (`AddonPolicyTemplate`) | New `enableControlPlaneMetrics`
policy parameter, threaded through the deployment into
`azureMonitorProfile.metrics.controlPlane.enabled`; apiVersion bumped.
The `existenceCondition` is intentionally left checking only
`metrics.enabled` (control-plane metrics is opt-in and should not drive
compliance). |
| **Terraform** (`AddonTerraformTemplate`) | New
`enable_control_plane_metrics` variable. The `azurerm` `monitor_metrics`
block can't express this property, so it's set via the **`azapi`**
provider (`azapi_update_resource`), added to `providers.tf`. |
| READMEs | Documented the new opt-in flag for all four. |

Default is **opt-in / `false`** everywhere, matching the CLI's opt-in
flag and preserving current template behavior.

## Greenfield race handling

The CLI PR notes that setting `controlPlane.enabled=true` before the
cluster's DCRA exists makes the RP schedule the control-plane-metrics
(CCP) collection pod prematurely → CrashLoopBackOff until reconcile.
These templates avoid that **by ordering**:

- ARM/Bicep/Policy: the managed-cluster profile PUT already `dependsOn`
the DCRA deployment, so `controlPlane.enabled` is only applied after the
DCRA exists.
- Terraform: the `azapi_update_resource` has `depends_on =
[azurerm_monitor_data_collection_rule_association.dcra]`.

## API version

`azureMonitorProfile.metrics.controlPlane` went **GA in `2026-04-01`** —
it is absent from every stable version through `2026-01-01`, first
appeared in `2026-02-02-preview`, and has been in every stable version
since. These templates use **`2026-06-01`, the latest stable AKS API
version**, which still carries the property. The paired aks-preview CLI
extension
([Azure/azure-cli-extensions#9931](Azure/azure-cli-extensions#9931))
vendors `2026-04-02-preview`; for these customer-facing IaC templates
the latest **GA stable** is preferred.

All four managed-cluster call sites use `2026-06-01` (ARM/Bicep/Policy
resource `apiVersion`, Terraform `azapi` `type`). The Bicep CLI's
bundled type catalog does not yet include `2026-06-01`, so the nested
module suppresses the benign `BCP081` "types not available" warning; the
compiled ARM correctly emits `controlPlane.enabled` and deploys fine.

### Other resource API versions

Audited every resource type in these templates against the registered
provider versions and the published stable swagger:

| Resource type | Version | |
|---|---|---|
| `Microsoft.ContainerService/managedClusters` | `2026-06-01` | bumped —
latest stable |
| `Microsoft.Insights/dataCollectionRuleAssociations` | `2025-05-11` |
bumped — latest stable, now matches the DCR version |
| `Microsoft.Insights/dataCollectionRules` | `2025-05-11` | already
latest stable |
| `Microsoft.Insights/dataCollectionEndpoints` | `2024-03-11` | already
latest stable |
| `Microsoft.AlertsManagement/prometheusRuleGroups` | `2023-03-01` |
already latest stable |
| `Microsoft.Authorization/roleAssignments` | `2022-04-01` | already
latest stable |
| `Microsoft.Dashboard/grafana` | `2025-08-01` | left as-is — see below
|

`Microsoft.Dashboard/grafana` has a newer stable (`2026-09-01`), but
nothing in this PR needs it and Bicep has no types for it yet, so
bumping would trade compile-time validation for no functional gain. Left
at `2025-08-01`.

## Live deployment validation

Deployed the ARM template end to end against a real cluster (`eastus2`,
AKS 1.35, 2 nodes) with
`enableControlPlaneMetrics=true`, alongside a new Azure Monitor
Workspace and Managed Grafana.

**Result: `provisioningState: Succeeded`.** All resources created — DCE,
DCR, DCRA, 6 recording
rule groups, Grafana→AMW integration, and the Monitoring Data Reader
role assignment.

Cluster property, read back at `2026-06-01`:

```json
"azureMonitorProfile": {
  "metrics": {
    "enabled": true,
    "controlPlane": { "enabled": true },
    "kubeStateMetrics": { "metricLabelsAllowlist": "", "metricAnnotationsAllowList": "" }
  }
}
```

Agent pods, all healthy:

```
ama-metrics-797764c94f-6xpvv                    2/2  Running
ama-metrics-797764c94f-q5wmc                    2/2  Running
ama-metrics-ksm-d484f9df8-69g8w                 1/1  Running
ama-metrics-node-2xtgr                          2/2  Running
ama-metrics-node-6kj2z                          2/2  Running
ama-metrics-operator-targets-74f955d776-qh94f   2/2  Running
```

Scrape jobs reporting into the AMW — **both control plane jobs
present**:

| Job | Targets |
|---|---|
| `controlplane-apiserver` | 2 |
| `controlplane-etcd` | 3 |
| cadvisor / kubelet / node | 2 each |
| kube-state-metrics | 1 |
| networkobservability-retina | 2 |

Control plane metric samples queried from the AMW Prometheus endpoint:

| Metric | Value |
|---|---|
| `apiserver_request_total` | 27,753 |
| `apiserver_request_duration_seconds_sum` | 495,383 |
| `apiserver_current_inflight_requests` | 4 |
| `etcd_server_has_leader` | 3 |
| `etcd_mvcc_db_total_size_in_bytes` | 24,604,672 |

Standard metrics also flowing: `kube_pod_info` 33,
`node_cpu_seconds_total` 64 series,
`container_memory_working_set_bytes` ~12.9 GB,
`kube_node_status_condition{Ready}` 2.

Worth noting: the `AzureMonitorMetricsControlPlanePreview` AFEC flag was
**Unregistered** on the
test subscription, so this exercises the GA API-property path rather
than the legacy feature-flag
gating.

## Bug found and fixed during validation

The first deployment attempt failed outright:

```
UnmarshalError: Invalid request body. Converting request body to a managed cluster
encountered error: json: unknown field "mode"
```

and after removing `mode`, the same error for `id`.

The nested managed-cluster resource in the ARM and Policy templates
carried `"mode":
"Incremental"` and `"id"` inside its `properties`. Neither is a
ManagedCluster property — `mode`
belongs on the enclosing deployment, and the cluster is already
addressed by name plus the nested
deployment's subscription and resource group.

**These fields pre-date this PR.** The `2023-01-01` RP silently ignored
them; from `2026-06-01`
the RP rejects unknown fields, so the API version bump turned a latent
template bug into a hard
deployment break. Both are now removed. Bicep and Terraform never
carried them.

## Static checks

- All edited JSON (ARM, ARM params, Policy rules, Policy params, Bicep
params) parses.
- `az bicep build` succeeds (exit 0) for all three Bicep files; both
nested modules build **warning-free**, and the compiled ARM contains
`"apiVersion": "2026-06-01"` and `controlPlane.enabled`.
- `az deployment group validate` passes.
- Terraform: HCL added by hand (no binary in the authoring env);
`terraform init -upgrade && terraform validate` recommended before
merge.

Worth noting that **all of the above passed while the `mode`/`id` bug
was still present.** JSON parsing, `bicep build` and ARM preflight
validation do not catch invalid properties inside a nested
managed-cluster body; only a real deployment surfaced it.

## Unrelated CI fix included

`otelcollector/build/linux/Dockerfile` copied
`/usr/lib/libp11-kit.so.0.3.0` by its full version.
That file is the target of the `p11-kit-proxy.so` and `libp11-kit.so.0`
symlinks so it has to be
copied, but its name changes whenever the base image is rebuilt.

`mcr.microsoft.com/azurelinux/base/core:3.0` is a floating tag and was
rebuilt **2026-08-20** with
p11-kit `0.26.5-1.azl3`, which ships `libp11-kit.so.0.4.11`. Every linux
AMD64/ARM64 image build
has failed since:

```
ERROR: failed to compute cache key: failed to calculate checksum of ref ...:
"/usr/lib/libp11-kit.so.0.3.0": not found
```

Reproduced against the current base image — only the pinned path is
missing:

```
OK      /usr/lib/p11-kit-trust.so
OK      /usr/lib/p11-kit-proxy.so   -> libp11-kit.so.0.4.11
MISSING /usr/lib/libp11-kit.so.0.3.0
OK      /usr/lib/libnssckbi.so
OK      /usr/lib/libp11-kit.so.0    -> libp11-kit.so.0.4.11
```

Fixed by globbing `libp11-kit.so.0*`, which picks up the SONAME symlink
and whichever versioned
file it points to. Verified by building the same base-to-distroless COPY
sequence locally: the
layer builds and `libp11-kit.so.0.4.11` lands in the image with both
symlinks resolving.

This is not related to the template changes, but it blocked CI on this
branch and every other open
PR (an unrelated dependabot build failed identically an hour before this
one), so it is fixed here.

### Same class of bug, also unpinned

While confirming the root cause I checked every other `COPY
--from=builder` path in both the main and CCP Dockerfiles. Three more
were pinned to a full version:

| Library | Was | Now |
|---|---|---|
| `libboost_filesystem.so` | `1.83.0` | `libboost_filesystem.so.*` |
| `libboost_atomic.so` | `1.83.0` | `libboost_atomic.so.*` |
| `libcpprest.so` | `2.10` | `libcpprest.so.*` |

Unlike most libraries copied here, these have **no stable SONAME
symlink** to point at instead: the fully qualified version *is* the
SONAME, so there is no `libboost_filesystem.so.1` or `libcpprest.so.2`
to reference. Verified by reproducing the builder stage locally, where
only the versioned regular files exist and no bare `.so` symlinks are
present.

They resolve today because azurelinux still ships `boost 1.83.0-2.azl3`,
so nothing is broken right now, but it is the identical failure mode
waiting on the next package refresh, in both Dockerfiles. Globbed for
the same reason.

Every remaining `COPY --from=builder` path uses a stable major-version
SONAME (`.so.6`, `.so.1`, `.so.0`, and `libsymcrypt.so.103` which is
itself a symlink), so nothing else here is version-fragile.

## Remaining items for reviewer

1. **Terraform `azapi` version.** Pinned `~> 1.15` and used a
`jsonencode` body (1.x style). If you prefer azapi 2.x, the `body`
becomes an HCL object (drop `jsonencode`).
2. **Default value.** Defaulted to `false` (opt-in). Confirm that's the
desired default for these onboarding templates vs. on-by-default.

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 09003a28-14fb-4340-83a6-14f284f9cfda
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants