🌱 Switch e2e to kind #2209

lentzi90 · 2025-01-21T13:58:58Z

What this PR does / why we need it:

Minikube is having troubles starting sometimes.
It was nice to work with since the VM could easily be attached to the same network as the BMH VMs, but it is possible to work around that also with kind.

It also removes overlays that are no longer used and the upgrade tests for ironic-24.0 as these are not supported/needed anymore. With that we also get rid of the last inspector code from the scripts 🎉

Fixture tests were only uploading artifacts on success. New they upload also on failure, just like e2e.

I have also tried stabilizing the fixture tests. It seems like there is some condition when namespace deletion takes a long time. From the BMO logs I saw that BMO was trying to create some resources in the terminating namespace. In an attempt to avoid this I added BMH deletion to the cleanup before deleting the namespace. That way at least BMO should be aware what is coming. Not sure if that is the root cause of the slow namespace deletion though.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #1783

metal3-io-bot · 2025-01-21T13:59:02Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

lentzi90 · 2025-01-22T07:21:41Z

Required tests have passed locally for redfish, redfish-virtualmedia and IPMI. Let's see what the CI thinks.

tuminoid

Cleanup suggestion. Less work to update those.

config/overlays/e2e-release-0.4/ironic.env

config/overlays/e2e-release-0.5/ironic.env

ironic-deployment/overlays/e2e-release-24.0-with-inspector/ironic_bmo_configmap.env

Remove BMO overlays no longer used in the e2e tests. Drop upgrade from ironic-24.0 as it is out of support and not needed to be tested anymore. Signed-off-by: Lennart Jern <[email protected]>

Minikube is having troubles starting sometimes. It was nice to work with since the VM could easily be attached to the same network as the BMH VMs, but it is possible to work around that also with kind. Signed-off-by: Lennart Jern <[email protected]>

Signed-off-by: Lennart Jern <[email protected]>

Sometimes the fixture tests hit the timeout for namespace deletion. The BMO logs indicate that BMO is trying to create new objects while the namespace is terminating. For example HardwareDetails. To avoid this, I think we can trigger deletion of the BMHs before we delete the namespace. We are running a bit close to the 1m deadline on successful runs in the re-inspection test. I believe this is explained by an extra reconcile loop when the hardwaredetails are updated because of the inspection. No other fixture test is close to this deadline normally. Signed-off-by: Lennart Jern <[email protected]>

lentzi90 · 2025-01-23T12:09:17Z

Alright I think this is ready for review. It became a bit larger than planned... Some of it can definitely be split into separate PRs if you wish.
I have run every test the is locally (including fixture and optional upgrade jobs) and all have passed. The fixture test is actually what I am most unsure about. It was quite flaky so I am not completely sure if I got it stable with this. The flakiness is anyway unrelated to this PR as far as I can tell.

tuminoid

One nit.

Also, @kashifest brought up https://github.com/medyagh/setup-minikube as a way to set up minikube in a GH action. I did not check it out myself if it would actually even be flexible enough for our needs. Did you check that?

tuminoid · 2025-01-23T13:34:40Z

test/e2e/config/ironic.yaml

@@ -2,6 +2,8 @@ images:
 # Use locally built e2e images
 - name: quay.io/metal3-io/baremetal-operator:e2e
  loadBehavior: tryLoad
+# - name: quay.io/metal3-io/ironic:local


Why have this added but commented out?

Oh I forgot about it!
@Rozzii you added the option to use a locally built ironic image recently. This is basically all that is required for it as far as I understand. If you uncomment this, the image is loaded. The rest is handled by changing the manifests directly or by pointing to another kustomization to deploy below.
Is this acceptable?

Previous process:

Set LOAD_LOCAL_IRONIC=true and build the image

Change e2e config to deploy ironic from e2e-local-ironic

Process after this PR:

Build the image and comment out these lines in e2e config

Change e2e config to deploy ironic from e2e-local-ironic

If we move to kind then the image transfer is not needed anymore as kind runs on the host VM so in that context I am fine with the changes.

I am fine with doing it via the config file although I prefer env variable based config because I am a bit worried that it is easy to accidentally commit changes to ironic.yaml.

lentzi90 · 2025-01-24T06:05:22Z

/cc @Rozzii

Rozzii

/lgtm

Rozzii · 2025-01-24T14:19:55Z

/lgtm cancel

Rozzii

/approve

metal3-io-bot · 2025-01-24T14:20:18Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Rozzii

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [Rozzii]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tuminoid

/lgtm

🚀 let's get rid of the minikube/docker flake!
Thank you @lentzi90 for taking time to do this.

lentzi90 · 2025-01-24T19:44:40Z

/cherry-pick release-0.9

metal3-io-bot · 2025-01-24T19:45:16Z

@lentzi90: #2209 failed to apply on top of branch "release-0.9":

Applying: E2E: Clean up unused overlays
Using index info to reconstruct a base tree...
M	hack/ci-e2e.sh
M	test/e2e/config/ironic.yaml
Falling back to patching base and 3-way merge...
Auto-merging test/e2e/config/ironic.yaml
CONFLICT (content): Merge conflict in test/e2e/config/ironic.yaml
Removing ironic-deployment/overlays/e2e-with-inspector/kustomization.yaml
Removing ironic-deployment/overlays/e2e-with-inspector/ironic_bmo_configmap.env
Removing ironic-deployment/overlays/e2e-with-inspector/ironic-patch.yaml
Removing ironic-deployment/overlays/e2e-release-24.0-with-inspector/kustomization.yaml
Removing ironic-deployment/overlays/e2e-release-24.0-with-inspector/ironic_bmo_configmap.env
Removing ironic-deployment/overlays/e2e-release-24.0-with-inspector/ironic-patch.yaml
Auto-merging hack/ci-e2e.sh
Removing config/overlays/fixture-release-0.5/kustomization.yaml
Removing config/overlays/fixture-release-0.4/kustomization.yaml
Removing config/overlays/e2e-release-0.5/kustomization.yaml
Removing config/overlays/e2e-release-0.5/ironic.env
Removing config/overlays/e2e-release-0.4/kustomization.yaml
Removing config/overlays/e2e-release-0.4/ironic.env
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config advice.mergeConflict false"
Patch failed at 0001 E2E: Clean up unused overlays

In response to this:

/cherry-pick release-0.9

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

metal3-io-bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 21, 2025

metal3-io-bot requested review from mquhuy and zaneb January 21, 2025 13:59

metal3-io-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jan 21, 2025

lentzi90 marked this pull request as ready for review January 22, 2025 07:21

metal3-io-bot requested review from iurygregory and tuminoid January 22, 2025 07:21

tuminoid reviewed Jan 22, 2025

View reviewed changes

config/overlays/e2e-release-0.4/ironic.env Outdated Show resolved Hide resolved

config/overlays/e2e-release-0.5/ironic.env Outdated Show resolved Hide resolved

ironic-deployment/overlays/e2e-release-24.0-with-inspector/ironic_bmo_configmap.env Outdated Show resolved Hide resolved

lentzi90 force-pushed the lentzi90/e2e-kind branch 2 times, most recently from 6480c59 to 8e3272e Compare January 23, 2025 08:44

metal3-io-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jan 23, 2025

lentzi90 force-pushed the lentzi90/e2e-kind branch from 8e3272e to 543ed55 Compare January 23, 2025 10:50

lentzi90 added 3 commits January 23, 2025 11:30

E2E: Clean up unused overlays

352ab75

Remove BMO overlays no longer used in the e2e tests. Drop upgrade from ironic-24.0 as it is out of support and not needed to be tested anymore. Signed-off-by: Lennart Jern <[email protected]>

E2E: Use kind instead of minikube

ace7702

Minikube is having troubles starting sometimes. It was nice to work with since the VM could easily be attached to the same network as the BMH VMs, but it is possible to work around that also with kind. Signed-off-by: Lennart Jern <[email protected]>

Fixture: Upload artifacts also on failure

a4cab90

Signed-off-by: Lennart Jern <[email protected]>

lentzi90 force-pushed the lentzi90/e2e-kind branch from 543ed55 to 192fc86 Compare January 23, 2025 11:31

lentzi90 changed the title ~~WIP: 🌱 Switch e2e to kind~~ 🌱 Switch e2e to kind Jan 23, 2025

metal3-io-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 23, 2025

lentzi90 force-pushed the lentzi90/e2e-kind branch from 192fc86 to 2f49b57 Compare January 23, 2025 11:46

lentzi90 requested a review from tuminoid January 23, 2025 12:15

tuminoid reviewed Jan 23, 2025

View reviewed changes

metal3-io-bot requested a review from Rozzii January 24, 2025 06:05

Rozzii reviewed Jan 24, 2025

View reviewed changes

metal3-io-bot added the lgtm Indicates that a PR is ready to be merged. label Jan 24, 2025

metal3-io-bot removed the lgtm Indicates that a PR is ready to be merged. label Jan 24, 2025

Rozzii reviewed Jan 24, 2025

View reviewed changes

metal3-io-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 24, 2025

tuminoid reviewed Jan 24, 2025

View reviewed changes

metal3-io-bot added the lgtm Indicates that a PR is ready to be merged. label Jan 24, 2025

metal3-io-bot merged commit 0a49d5a into metal3-io:main Jan 24, 2025
29 checks passed

metal3-io-bot deleted the lentzi90/e2e-kind branch January 24, 2025 14:23

metal3-io-bot added this to the BMO - v0.10 milestone Jan 24, 2025

lentzi90 mentioned this pull request Jan 28, 2025

🌱 Switch e2e to kind #2222

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🌱 Switch e2e to kind #2209

🌱 Switch e2e to kind #2209

lentzi90 commented Jan 21, 2025 •

edited

Loading

metal3-io-bot commented Jan 21, 2025

lentzi90 commented Jan 22, 2025

tuminoid left a comment

lentzi90 commented Jan 23, 2025

tuminoid left a comment

tuminoid Jan 23, 2025

lentzi90 Jan 23, 2025

Rozzii Jan 24, 2025

lentzi90 commented Jan 24, 2025

Rozzii left a comment

Rozzii commented Jan 24, 2025

Rozzii left a comment

metal3-io-bot commented Jan 24, 2025

tuminoid left a comment

lentzi90 commented Jan 24, 2025

metal3-io-bot commented Jan 24, 2025

🌱 Switch e2e to kind #2209

🌱 Switch e2e to kind #2209

Conversation

lentzi90 commented Jan 21, 2025 • edited Loading

metal3-io-bot commented Jan 21, 2025

lentzi90 commented Jan 22, 2025

tuminoid left a comment

Choose a reason for hiding this comment

lentzi90 commented Jan 23, 2025

tuminoid left a comment

Choose a reason for hiding this comment

tuminoid Jan 23, 2025

Choose a reason for hiding this comment

lentzi90 Jan 23, 2025

Choose a reason for hiding this comment

Rozzii Jan 24, 2025

Choose a reason for hiding this comment

lentzi90 commented Jan 24, 2025

Rozzii left a comment

Choose a reason for hiding this comment

Rozzii commented Jan 24, 2025

Rozzii left a comment

Choose a reason for hiding this comment

metal3-io-bot commented Jan 24, 2025

tuminoid left a comment

Choose a reason for hiding this comment

lentzi90 commented Jan 24, 2025

metal3-io-bot commented Jan 24, 2025

lentzi90 commented Jan 21, 2025 •

edited

Loading