introduce crd-schema-checker #1055

dgn · 2025-07-23T15:47:41Z

This is a script that uses openshift/crd-schema-checker to verify that we're not breaking any CRD API guarantees across releases.

We can't merge this until we fix the errors, but it's pretty neat.

codecov · 2025-07-23T15:58:36Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.80%. Comparing base (a8d66a8) to head (ea7193b).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1055      +/-   ##
==========================================
+ Coverage   77.69%   77.80%   +0.10%     
==========================================
  Files          44       44              
  Lines        2834     2834              
==========================================
+ Hits         2202     2205       +3     
+ Misses        525      522       -3     
  Partials      107      107

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Makefile.core.mk

tools/crd-schema-checker.sh

dgn · 2025-07-29T14:26:22Z

I updated this, it was pretty broken before. Now it actually outputs all the errors, warnings and info messages and has a proper global error count, and a separate one for errors in stable APIs (as only those will fail the check)

mkolesnik · 2025-07-30T08:13:44Z

Makefile.core.mk

 	@test -s $(LOCALBIN)/misspell || GOBIN=$(LOCALBIN) go install github.com/client9/misspell/cmd/misspell@$(MISSPELL_VERSION)

+.PHONY: crd-schema-checker
+crd-schema-checker: $(CRD_SCHEMA_CHECKER) ## Download crd-schema-checker to bin directory.


Why even have the phony target?

It seems to me lint-crds can just depend on $(CRD_SCHEMA_CHECKER) directly

technically yes, but that's how we're doing it for all other tools as well

Makefile.core.mk

tools/crd-schema-checker.sh

mkolesnik · 2025-07-30T08:40:35Z

tools/crd-schema-checker.sh

+            echo " - $line"
+        done <<< "$output"
+    fi
+    echo "--> ${errors} errors, ${warnings} warnings, ${infos} infos"


Why is it pertinent to count the errors/warn/info here?

Wouldn't it make more sense to push a PR to print that in the original command anyway?

yeah we need to count, if only to separate them by stable vs. non-stable APIs- we can allow breaking changes in v1alpha1 CRDs, but never in v1 CRDs, for example

tools/crd-schema-checker.sh

mkolesnik · 2025-07-30T13:30:01Z

tools/crd-schema-checker.sh

+
+        version=$(getLatestCRDVersion "${current_crd_map[$crd]}")
+        output_result "${crd}" "${version}" "${output}"
+        CHECKED_CRDS=$((CHECKED_CRDS + 1))


Suggested change

CHECKED_CRDS=$((CHECKED_CRDS + 1))

((CHECKED_CRDS++))

I understand there is no practical difference between the two. I noticed your comment fixed - did you forget to update the PR?

oh, that was a mistake. I tried this and it didn't work I think

tools/crd-schema-checker.sh

mkolesnik · 2025-07-30T13:40:00Z

tools/crd-schema-checker.sh

+echo "=== Results ==="
+echo "Checked $CHECKED_CRDS CRDs: $ERRORS errors ($STABLE_ERRORS errors in stable APIs), $WARNINGS warnings, $INFOS infos"
+
+if [[ $STABLE_ERRORS -gt 0 ]]; then


Generally to me parsing the output seems like the wrong approach.
We have the exit code of the checks, we should rely on those instead of trying to read and interpret the output.

again, we need to parse to get detailed results - warnings we can always ignore, for pre-release CRDs we can even ignore errors. yes, technically we could do all that based on just exit codes but we wouldn't be able to produce detailed counts

dgn · 2025-09-18T14:14:17Z

/retest

dgn · 2025-09-18T15:01:57Z

This seems to be a legitimate error:

  - ERROR: "NoEnumRemoval": crd/istiocnis.sailoperator.io version/v1 enum/"v1.26.1" may not be removed for field/^.spec.version

Did we forget to add 1.26.1 on the 1.27 branch?

We have previously released a version on the 1.26 stream that supported 1.26.1. In order to not break API guarantees, that enum value of 1.26.1 has to be supported in later releases as well, so I'm adding it here. This problem was found with the crd-schema-checker introduced in istio-ecosystem#1055. Signed-off-by: Daniel Grimm <[email protected]>

dgn · 2025-09-19T06:53:57Z

This seems to be a legitimate error:
  - ERROR: "NoEnumRemoval": crd/istiocnis.sailoperator.io version/v1 enum/"v1.26.1" may not be removed for field/^.spec.version 
Did we forget to add 1.26.1 on the 1.27 branch?

This was indeed the case. First legit problem the checker found 👍 Fix is in #1228, until it merges the lint job will fail on this PR.

We have previously released a version on the 1.26 stream that supported 1.26.1. In order to not break API guarantees, that enum value of 1.26.1 has to be supported in later releases as well, so I'm adding it here. This problem was found with the crd-schema-checker introduced in #1055. Signed-off-by: Daniel Grimm <[email protected]>

dgn · 2025-10-02T14:10:28Z

/retest

Makefile.core.mk

sridhargaddam · 2025-10-03T12:56:54Z

tools/crd-schema-checker.sh

+
+        version=$(getLatestCRDVersion "${current_crd_map[$crd]}")
+        output_result "${crd}" "${version}" "${output}"
+        CHECKED_CRDS=$((CHECKED_CRDS + 1))


I understand there is no practical difference between the two. I noticed your comment fixed - did you forget to update the PR?

sridhargaddam · 2025-10-03T12:57:28Z

tools/crd-schema-checker.sh

+        version=$(getLatestCRDVersion "${previous_crd_map[$crd]}")
+        if ! isStableVersion "$version"; then
+            echo "WARNING: CRD $crd was removed ($version)"
+            WARNINGS=$((WARNINGS + 1))


nit: In case you want to make the same change here as well as in Line 166 below.

FilipB · 2025-10-07T13:05:48Z

LGTM, not approving to wait for other comments to be resolved.

dgn · 2025-10-16T14:12:06Z

currently if not on a release branch, the linter will compare the last two release branches. We should change that to compare the current HEAD against the latest release branch

dgn · 2025-10-22T19:59:49Z

lint job will fail until #1301 is merged

This is a script that uses openshift/crd-schema-checker to verify that we're not breaking any CRD API guarantees across releases. Signed-off-by: Daniel Grimm <[email protected]>

dgn · 2025-10-23T09:17:17Z

/retest

dgn requested a review from a team as a code owner July 23, 2025 15:47

istio-testing added the size/L label Jul 23, 2025

dgn force-pushed the crd-compatibility-check branch from a002f63 to a5334e7 Compare July 23, 2025 15:48

dgn force-pushed the crd-compatibility-check branch 9 times, most recently from 26f855e to e4babbe Compare July 28, 2025 14:07

FilipB reviewed Jul 29, 2025

View reviewed changes

dgn changed the title ~~introduce crd-schema-checker~~ [WIP] introduce crd-schema-checker Jul 29, 2025

istio-testing added the do-not-merge/work-in-progress label Jul 29, 2025

dgn force-pushed the crd-compatibility-check branch 3 times, most recently from 3d32594 to 085fd03 Compare July 29, 2025 14:23

dgn force-pushed the crd-compatibility-check branch from 085fd03 to 0063f73 Compare July 29, 2025 14:40

dgn changed the title ~~[WIP] introduce crd-schema-checker~~ [introduce crd-schema-checker Jul 29, 2025

istio-testing removed the do-not-merge/work-in-progress label Jul 29, 2025

dgn changed the title ~~[introduce crd-schema-checker~~ introduce crd-schema-checker Jul 29, 2025

mkolesnik reviewed Jul 30, 2025

View reviewed changes

FilipB added the cherrypick/release-1.27 label Sep 11, 2025

dgn force-pushed the crd-compatibility-check branch from 0063f73 to fc2bcf3 Compare September 19, 2025 06:33

dgn mentioned this pull request Sep 19, 2025

[release-1.27] Add 1.26.1 #1228

Merged

dgn force-pushed the crd-compatibility-check branch from fc2bcf3 to 8fb3f20 Compare September 19, 2025 06:52

sridhargaddam reviewed Oct 3, 2025

View reviewed changes

dgn force-pushed the crd-compatibility-check branch from 8fb3f20 to b4e4ede Compare October 8, 2025 15:34

dgn force-pushed the crd-compatibility-check branch from b4e4ede to 2be5426 Compare October 22, 2025 19:55

introduce crd-schema-checker

ea7193b

This is a script that uses openshift/crd-schema-checker to verify that we're not breaking any CRD API guarantees across releases. Signed-off-by: Daniel Grimm <[email protected]>

dgn force-pushed the crd-compatibility-check branch from 2be5426 to ea7193b Compare October 22, 2025 20:06

introduce crd-schema-checker #1055

Are you sure you want to change the base?

introduce crd-schema-checker #1055

Uh oh!

Conversation

dgn commented Jul 23, 2025

Uh oh!

codecov bot commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dgn commented Jul 29, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dgn Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dgn commented Sep 18, 2025

Uh oh!

dgn commented Sep 18, 2025

Uh oh!

dgn commented Sep 19, 2025

Uh oh!

dgn commented Oct 2, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FilipB commented Oct 7, 2025

Uh oh!

dgn commented Oct 16, 2025

Uh oh!

dgn commented Oct 22, 2025

Uh oh!

dgn commented Oct 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov bot commented Jul 23, 2025 •

edited

Loading

dgn Sep 19, 2025 •

edited

Loading