Skip to content

test(smoke): skip @vm_only scenarios in container lanes - #740

Open
castrojo wants to merge 1 commit into
mainfrom
fix/vm-only-container-lane
Open

test(smoke): skip @vm_only scenarios in container lanes#740
castrojo wants to merge 1 commit into
mainfrom
fix/vm-only-container-lane

Conversation

@castrojo

@castrojo castrojo commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

offline_boot.feature drives the device under test entirely over SSH — its own
header says "Runner: plain SSH behave (no qecore — no GUI interaction needed)".
Container lanes run behave inside the nested target and have no sshd, so all 8
scenarios failed their Background with:

AssertionError: Cannot reach VM at 127.0.0.1 over SSH after 5 attempts: rc=255, stdout=''

These were invisible while the GNOME Shell readiness gate kept container lanes from
executing any scenario at all (fixed in projectbluefin/lab#614/#615). They report a
runner limitation as an image regression.

Change

  • Tag offline_boot.feature @vm_only.
  • Skip @vm_only scenarios in the smoke before_scenario hook when the lane is a
    container lane, following the existing @bluefin / @dakota_only / @wifi skip
    pattern already in that hook.
  • New tests/shared/runtime_env.py detects a container lane via /run/.containerenv,
    which podman writes into every container it starts and a KubeVirt VM never has.

The existing _IN_CONTAINER heuristic cannot be reused: it probes for
/usr/bin/bootc, which is present in the container lane too because the target is
a bootc image. Confirmed on a live lane:

$ podman exec bluefin-qa-target ls -l /usr/bin/bootc /run/.containerenv
-rwxr-xr-x. … /usr/bin/bootc
-rw-r--r--. … /run/.containerenv
IN_CONTAINER= False        # the old heuristic

Not done

No timeout is lengthened and no assertion is weakened or deleted. The scenarios still
run unchanged in VM lanes, where they are meaningful. Scenario totals are unchanged
(skipping is not removal), so the coverage snapshot is already current.

Companion infrastructure fixes (/dev/uinput mode, setuptools for qecore's
pkg_resources import) are in projectbluefin/lab#619 — deliberately kept in a
separate PR per the repository-ownership boundary.

Validation

  • ruff check tests/ --select E,F,W --ignore E501
  • behave --dry-run tests/smoke/features ✅ (25 features, 196 scenarios)
  • python3 -m pytest tests/unit/ -q1255 passed
  • scripts/update_coverage_snapshot.py → "No change — snapshot already current"

Assisted-by: Claude Opus 5 via GitHub Copilot CLI

offline_boot.feature drives the device under test entirely over SSH --
its own header says "Runner: plain SSH behave". Container lanes run
behave *inside* the nested target and have no sshd, so every scenario
failed its Background with

  Cannot reach VM at 127.0.0.1 over SSH after 5 attempts: rc=255

Those failures were invisible while the GNOME readiness gate kept
container lanes from executing any scenario at all; they report a runner
limitation as an image regression.

Tag the feature @vm_only and skip it in the smoke hook when the lane is a
container lane. Detection uses /run/.containerenv, which podman writes
into every container it starts and a KubeVirt VM never has. The existing
_IN_CONTAINER heuristic cannot be reused here: it probes for
/usr/bin/bootc, which is present in the container lane too because the
target *is* a bootc image.

No timeout is lengthened and no assertion is weakened -- the scenarios
still run unchanged in VM lanes, where they are meaningful.

Assisted-by: Claude Opus 5 via GitHub Copilot CLI
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Straightforward and correctly scoped. offline_boot.feature is genuinely SSH-only per its own header, and a container lane has no sshd to reach — the Cannot reach VM ... over SSH failure really is a runner-capability mismatch being misreported as an image regression, not a weakened assertion.

The new /run/.containerenv detection in tests/shared/runtime_env.py is a better signal than the existing _IN_CONTAINER heuristic (which probes for /usr/bin/bootc — present in both lanes since the container-lane target is itself a bootc image); the PR includes a live-lane confirmation of that distinction in the description, and it's independently plausible since podman writes that marker into every container it starts.

No timeouts are lengthened, no assertions are dropped, and the tag only skips scenarios that cannot be meaningful in a container lane — this is a coverage-reporting fix, not a coverage reduction: the scenarios still run fully in VM lanes. TestInContainerLane unit tests cover the marker-present/absent/path cases directly. GitHub Actions checks (lint, behave dry-run, pytest, coverage-snapshot freshness) are all green on this one.

The ghost-lab external status is red here too, but it's red identically across every other open PR in this review batch (including this one, which has otherwise clean CI and a trivial, well-reasoned diff), so I'm reading that as broken/flaky external lab infra rather than a signal about this change.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants