Skip to content

Commit c86a253

Browse files
fix(vm): resolve PATH shadowing for pyelftools and Cargo in build-libkrun.sh
Signed-off-by: Vincent Caux-Brisebois <vcauxbrisebo@nvidia.com>
1 parent 1eaece1 commit c86a253

File tree

4 files changed

+597
-63
lines changed

4 files changed

+597
-63
lines changed

architecture/custom-vm-runtime.md

Lines changed: 6 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -122,11 +122,10 @@ graph LR
122122
123123
subgraph Linux["Linux CI (build-libkrun.sh)"]
124124
BUILD_L["Build kernel + libkrunfw.so + libkrun.so"]
125-
KERNELC["kernel.c\nKernel as C byte array"]
126125
end
127126
128127
subgraph macOS["macOS CI (build-libkrun-macos.sh)"]
129-
BUILD_M["Compile kernel.c -> libkrunfw.dylib\nBuild libkrun.dylib"]
128+
BUILD_M["Build libkrunfw.dylib + libkrun.dylib"]
130129
end
131130
132131
subgraph Output["target/libkrun-build/"]
@@ -136,8 +135,7 @@ graph LR
136135
137136
KCONF --> BUILD_L
138137
BUILD_L --> LIB_SO
139-
BUILD_L --> KERNELC
140-
KERNELC --> BUILD_M
138+
KCONF --> BUILD_M
141139
BUILD_M --> LIB_DY
142140
```
143141

@@ -228,18 +226,15 @@ supported platforms. Runs on-demand or when the kernel config / pinned versions
228226

229227
| Platform | Runner | Build Method |
230228
|----------|--------|-------------|
231-
| Linux ARM64 | `build-arm64` (self-hosted) | Native `build-libkrun.sh` (also exports kernel.c) |
229+
| Linux ARM64 | `build-arm64` (self-hosted) | Native `build-libkrun.sh` |
232230
| Linux x86_64 | `build-amd64` (self-hosted) | Native `build-libkrun.sh` |
233-
| macOS ARM64 | `macos-latest-xlarge` (GitHub-hosted) | `build-libkrun-macos.sh --kernel-dir` (uses pre-built kernel.c from ARM64) |
231+
| macOS ARM64 | `macos-latest-xlarge` (GitHub-hosted) | `build-libkrun-macos.sh` |
234232

235233
Artifacts: `vm-runtime-{platform}.tar.zst` containing libkrun, libkrunfw, gvproxy, and
236234
provenance metadata.
237235

238-
The aarch64 Linux kernel is compiled once on the Linux ARM64 runner. The resulting
239-
`kernel.c` (a C source file containing the kernel as a byte array) is passed to the
240-
macOS job, which compiles it into `libkrunfw.dylib` with Apple's `cc`. This eliminates
241-
the need for krunvm/Fedora VM and cuts macOS CI from ~45 min to ~5 min. The kernel
242-
inside libkrunfw is always Linux regardless of host platform.
236+
Each platform builds its own libkrunfw and libkrun natively. The kernel inside
237+
libkrunfw is always Linux regardless of host platform.
243238

244239
### VM Binary (`release-vm-dev.yml`)
245240

crates/openshell-vm/runtime/README.md

Lines changed: 8 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -27,21 +27,20 @@ runtime/
2727

2828
## Build Pipeline
2929

30-
The kernel is compiled on Linux CI runners. macOS reuses the pre-built `kernel.c`
31-
artifact from the Linux ARM64 build — no krunvm or Fedora VM needed.
30+
Each platform builds its own kernel and runtime natively.
3231

3332
```
34-
Linux ARM64: builds aarch64 kernel -> .so + exports kernel.c (parallel)
35-
Linux AMD64: builds x86_64 kernel -> .so (parallel)
36-
macOS ARM64: reuses aarch64 kernel.c -> .dylib (depends on ARM64)
33+
Linux ARM64: builds aarch64 kernel -> .so (parallel)
34+
Linux AMD64: builds x86_64 kernel -> .so (parallel)
35+
macOS ARM64: builds aarch64 kernel -> .dylib
3736
```
3837

3938
### Build Scripts
4039

4140
| Script | Platform | What it does |
4241
|--------|----------|-------------|
43-
| `tasks/scripts/vm/build-libkrun.sh` | Linux | Builds libkrunfw + libkrun from source, exports kernel.c |
44-
| `tasks/scripts/vm/build-libkrun-macos.sh` | macOS | Compiles pre-built kernel.c into .dylib, builds libkrun |
42+
| `tasks/scripts/vm/build-libkrun.sh` | Linux | Builds libkrunfw + libkrun from source |
43+
| `tasks/scripts/vm/build-libkrun-macos.sh` | macOS | Builds libkrunfw + libkrun from source |
4544
| `tasks/scripts/vm/package-vm-runtime.sh` | Any | Packages runtime tarball (libs + gvproxy + provenance) |
4645

4746
### Quick Build (Linux)
@@ -56,14 +55,12 @@ FROM_SOURCE=1 mise run vm:setup
5655

5756
### Quick Build (macOS)
5857

59-
On macOS, you need a pre-built `kernel.c` from a Linux ARM64 build:
60-
6158
```bash
6259
# Download pre-built runtime (recommended, ~30s):
6360
mise run vm:setup
6461

65-
# Or if you have kernel.c from a Linux build:
66-
tasks/scripts/vm/build-libkrun-macos.sh --kernel-dir target/libkrun-build
62+
# Or build from source:
63+
FROM_SOURCE=1 mise run vm:setup
6764
```
6865

6966
### Output
@@ -74,8 +71,6 @@ Build artifacts are placed in `target/libkrun-build/`:
7471
target/libkrun-build/
7572
libkrun.so / libkrun.dylib # The VMM library
7673
libkrunfw.so* / libkrunfw.dylib # Kernel firmware library
77-
kernel.c # Linux kernel as C byte array (Linux only)
78-
ABI_VERSION # ABI version number (Linux only)
7974
```
8075

8176
## Networking

tasks/scripts/vm/build-libkrun.sh

Lines changed: 170 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -93,17 +93,33 @@ install_deps() {
9393
echo " libelf-dev libssl-dev bc curl cpio" >&2
9494
fi
9595

96-
# Ensure pyelftools is importable by the Python that will run bin2cbundle.py.
97-
# The apt package may install to a different Python than the default python3.
98-
if ! python3 -c "import elftools" &>/dev/null; then
99-
echo " pyelftools not importable, installing via pip..."
100-
python3 -m pip install --break-system-packages pyelftools 2>/dev/null || \
101-
python3 -m pip install pyelftools || true
102-
fi
10396
}
10497

10598
install_deps
10699

100+
# libkrunfw's Makefile invokes `python3` from PATH for bin2cbundle.py. A mise shim,
101+
# project venv, or other early PATH entry often shadows /usr/bin/python3 and does
102+
# not ship pyelftools even when python3-pyelftools is installed for the distro.
103+
ensure_python3_with_pyelftools_for_libkrunfw() {
104+
echo " Checking Python 3 + pyelftools (libkrunfw bin2cbundle.py)..."
105+
if python3 -c 'from elftools.elf.elffile import ELFFile' 2>/dev/null; then
106+
echo " OK ($(command -v python3))"
107+
return 0
108+
fi
109+
if [ -x /usr/bin/python3 ] && /usr/bin/python3 -c 'from elftools.elf.elffile import ELFFile' 2>/dev/null; then
110+
export PATH="/usr/bin:${PATH}"
111+
echo " Using /usr/bin/python3 (PATH python3 lacked pyelftools; system Python has it)."
112+
return 0
113+
fi
114+
echo "ERROR: Python 3 with pyelftools is required to build libkrunfw (kernel.c generation)." >&2
115+
echo " Install: Debian/Ubuntu: sudo apt-get install -y python3-pyelftools" >&2
116+
echo " Fedora/RHEL: sudo dnf install -y python3-pyelftools" >&2
117+
echo " pip: python3 -m pip install --user pyelftools" >&2
118+
echo " If the package is installed but this still fails, PATH may point at another python3 (mise, venv)." >&2
119+
echo " Try: PATH=/usr/bin:\$PATH mise run vm:setup" >&2
120+
exit 1
121+
}
122+
107123
# ── Setup build directory ───────────────────────────────────────────────
108124

109125
mkdir -p "$BUILD_DIR"
@@ -114,6 +130,8 @@ cd "$BUILD_DIR"
114130
echo ""
115131
echo "==> Building libkrunfw with custom kernel config..."
116132

133+
ensure_python3_with_pyelftools_for_libkrunfw
134+
117135
if [ ! -d libkrunfw ]; then
118136
echo " Cloning libkrunfw (pinned: ${LIBKRUNFW_REF:-HEAD})..."
119137
git clone https://github.com/containers/libkrunfw.git
@@ -221,54 +239,168 @@ make -j"$(nproc)"
221239
cp libkrunfw.so* "$OUTPUT_DIR/"
222240
echo " Built: $(ls "$OUTPUT_DIR"/libkrunfw.so* | xargs -n1 basename | tr '\n' ' ')"
223241

224-
# ── Export kernel.c for cross-platform builds ───────────────────────────
225-
# kernel.c is a C source file containing the compiled Linux kernel as a byte
226-
# array. It is architecture-specific (aarch64 vs x86_64) but OS-agnostic —
227-
# any C compiler can turn it into a .so or .dylib. We export it so the macOS
228-
# job can produce libkrunfw.dylib without rebuilding the kernel.
242+
cd "$BUILD_DIR"
229243

230-
ABI_VERSION="$(grep -oE 'ABI_VERSION\s*=\s*[0-9]+' Makefile | head -1 | sed 's/[^0-9]//g')"
244+
# ── Build libkrun (VMM) ─────────────────────────────────────────────────
231245

232-
if [ -f kernel.c ]; then
233-
cp kernel.c "$OUTPUT_DIR/kernel.c"
234-
echo "${ABI_VERSION}" > "$OUTPUT_DIR/ABI_VERSION"
235-
echo " Exported kernel.c ($(du -sh kernel.c | cut -f1)) and ABI_VERSION=${ABI_VERSION}"
236-
else
237-
echo "Warning: kernel.c not found — cross-platform builds will not work" >&2
238-
fi
246+
# libkrun's Makefile invokes plain `cargo`. Ubuntu/Debian often put /usr/bin/cargo
247+
# (e.g. 1.75) ahead of mise/rustup; upstream requires edition 2024 (Cargo >= 1.85).
248+
ensure_cargo_for_libkrun() {
249+
local min_ver="${LIBKRUN_MIN_CARGO_VERSION:-1.85}"
250+
local have ver_line bindir candidates_mise candidates_home
251+
252+
cargo_meets_min() {
253+
local bin="$1"
254+
local v
255+
[ -x "$bin" ] || return 1
256+
v="$("$bin" --version 2>/dev/null | awk '{print $2}')"
257+
[ -n "$v" ] || return 1
258+
[ "$(printf '%s\n' "${min_ver}" "$v" | sort -V | head -n1)" = "${min_ver}" ]
259+
}
260+
261+
echo " Checking Cargo (libkrun needs >= ${min_ver}, edition 2024)..."
262+
if cargo_meets_min "$(command -v cargo 2>/dev/null || true)"; then
263+
echo " OK ($(command -v cargo)$(cargo --version))"
264+
return 0
265+
fi
239266

240-
cd "$BUILD_DIR"
267+
candidates_mise=""
268+
if command -v mise &>/dev/null; then
269+
if ver_line="$(mise which cargo 2>/dev/null)" && [ -n "${ver_line}" ]; then
270+
candidates_mise="$(dirname "${ver_line}")"
271+
fi
272+
fi
273+
candidates_home="${HOME}/.cargo/bin"
274+
275+
for bindir in "${candidates_mise}" "${candidates_home}"; do
276+
[ -n "${bindir}" ] || continue
277+
if cargo_meets_min "${bindir}/cargo"; then
278+
export PATH="${bindir}:${PATH}"
279+
echo " Using ${bindir}/cargo ($("${bindir}/cargo" --version))"
280+
return 0
281+
fi
282+
done
241283

242-
# ── Build libkrun (VMM) ─────────────────────────────────────────────────
284+
echo "ERROR: Cargo >= ${min_ver} is required to build libkrun (Rust edition 2024)." >&2
285+
echo " Current: $(command -v cargo 2>/dev/null || echo '(no cargo in PATH)') $(cargo --version 2>/dev/null || true)" >&2
286+
echo " Typical fix: run vm:setup via mise from the repo so Rust stable is on PATH," >&2
287+
echo " or: rustup update stable && export PATH=\"\$HOME/.cargo/bin:\$PATH\"" >&2
288+
echo " Override minimum: LIBKRUN_MIN_CARGO_VERSION=…" >&2
289+
exit 1
290+
}
291+
292+
# Directory must contain libclang.so or libclang-<ver>.so (what clang-sys expects
293+
# for linking; bare .so.N sonames alone are not enough).
294+
_libclang_dir_usable() {
295+
local d="$1"
296+
[ -n "$d" ] && [ -d "$d" ] || return 1
297+
if [ -e "$d/libclang.so" ]; then
298+
return 0
299+
fi
300+
local f base
301+
for f in "$d"/libclang-*.so; do
302+
[ -e "$f" ] || continue
303+
base="$(basename "$f")"
304+
case "$base" in
305+
*-cpp.so*) continue ;;
306+
esac
307+
if [[ "$base" == libclang-*.so ]] && [[ "$base" != *.so.[0-9]* ]]; then
308+
return 0
309+
fi
310+
done
311+
return 1
312+
}
313+
314+
ensure_libclang_for_libkrun() {
315+
local user_libclang="${LIBCLANG_PATH:-}"
316+
317+
if [ -n "$user_libclang" ] && _libclang_dir_usable "$user_libclang"; then
318+
export LIBCLANG_PATH="$user_libclang"
319+
echo " LIBCLANG_PATH=$LIBCLANG_PATH (from environment)"
320+
return 0
321+
fi
322+
323+
if [ -n "$user_libclang" ]; then
324+
echo " Warning: LIBCLANG_PATH='$user_libclang' has no libclang.so or libclang-*.so symlink;" >&2
325+
echo " those are required for clang-sys. Searching other system locations..." >&2
326+
fi
327+
unset LIBCLANG_PATH
328+
329+
local llvm_lib
330+
if command -v llvm-config &>/dev/null; then
331+
llvm_lib="$(llvm-config --libdir 2>/dev/null)" || true
332+
if [ -n "${llvm_lib}" ] && _libclang_dir_usable "$llvm_lib"; then
333+
export LIBCLANG_PATH="$llvm_lib"
334+
echo " LIBCLANG_PATH=$LIBCLANG_PATH (from llvm-config --libdir)"
335+
return 0
336+
fi
337+
fi
338+
339+
shopt -s nullglob
340+
local candidates=(/usr/lib/llvm-*/lib)
341+
shopt -u nullglob
342+
while IFS= read -r llvm_lib; do
343+
[ -n "$llvm_lib" ] || continue
344+
if _libclang_dir_usable "$llvm_lib"; then
345+
export LIBCLANG_PATH="$llvm_lib"
346+
echo " LIBCLANG_PATH=$LIBCLANG_PATH (from /usr/lib/llvm-*/lib)"
347+
return 0
348+
fi
349+
done < <(printf '%s\n' "${candidates[@]}" | sort -rV)
350+
351+
local multi
352+
multi="$(gcc -print-multiarch 2>/dev/null || true)"
353+
if [ -n "$multi" ] && _libclang_dir_usable "/usr/lib/${multi}"; then
354+
export LIBCLANG_PATH="/usr/lib/${multi}"
355+
echo " LIBCLANG_PATH=$LIBCLANG_PATH (from gcc multiarch /usr/lib/${multi})"
356+
return 0
357+
fi
358+
359+
if _libclang_dir_usable "/usr/lib64"; then
360+
export LIBCLANG_PATH="/usr/lib64"
361+
echo " LIBCLANG_PATH=$LIBCLANG_PATH (from /usr/lib64)"
362+
return 0
363+
fi
364+
365+
echo "ERROR: libclang is required to build libkrun (Rust bindgen / clang-sys) but was not found." >&2
366+
if [ -n "$user_libclang" ]; then
367+
echo " You had LIBCLANG_PATH='$user_libclang' (ignored after search failed)." >&2
368+
fi
369+
echo " Install LLVM/Clang development packages, then re-run vm:setup:" >&2
370+
echo " Debian/Ubuntu: sudo apt-get install -y libclang-dev" >&2
371+
echo " Fedora/RHEL: sudo dnf install -y clang-devel" >&2
372+
echo " Then unset LIBCLANG_PATH or set it to a directory that contains libclang.so." >&2
373+
exit 1
374+
}
243375

244376
echo ""
245377
echo "==> Building libkrun..."
246378

379+
ensure_cargo_for_libkrun
380+
ensure_libclang_for_libkrun
381+
382+
LIBKRUN_REF="${LIBKRUN_REF:-v1.17.4}"
383+
247384
if [ ! -d libkrun ]; then
248385
echo " Cloning libkrun..."
249-
git clone --depth 1 https://github.com/containers/libkrun.git
386+
git clone https://github.com/containers/libkrun.git
250387
fi
251388

252389
cd libkrun
253390

254-
# Build with NET support for gvproxy networking and BLK support for the
255-
# host-backed state disk.
256-
echo " Building libkrun with NET=1 BLK=1..."
391+
if [ -n "${LIBKRUN_REF:-}" ]; then
392+
echo " Checking out pinned ref: ${LIBKRUN_REF}"
393+
git fetch origin "${LIBKRUN_REF}" 2>/dev/null || git fetch origin
394+
git checkout "${LIBKRUN_REF}" 2>/dev/null || git checkout "origin/${LIBKRUN_REF}" 2>/dev/null || true
395+
fi
257396

258-
# Locate libclang for clang-sys if LIBCLANG_PATH isn't already set.
259-
# clang-sys looks for libclang.so or libclang-*.so; on Debian/Ubuntu the
260-
# versioned file (e.g. libclang-18.so.18) lives under the LLVM lib dir.
261-
if [ -z "${LIBCLANG_PATH:-}" ]; then
262-
for llvm_lib in /usr/lib/llvm-*/lib; do
263-
if ls "$llvm_lib"/libclang*.so* &>/dev/null; then
264-
export LIBCLANG_PATH="$llvm_lib"
265-
echo " LIBCLANG_PATH=$LIBCLANG_PATH"
266-
break
267-
fi
268-
done
397+
if [ -f init/Makefile ] || grep -q 'init/init' Makefile 2>/dev/null; then
398+
echo " Building init/init binary..."
399+
make init/init
269400
fi
270401

271-
make NET=1 BLK=1 -j"$(nproc)"
402+
echo " Building libkrun with NET=1 BLK=1..."
403+
cargo build --release --features blk --features net --target-dir="$(pwd)/target"
272404

273405
# Copy output
274406
cp target/release/libkrun.so "$OUTPUT_DIR/"
@@ -283,7 +415,6 @@ echo "==> Build complete!"
283415
echo " Output directory: ${OUTPUT_DIR}"
284416
echo ""
285417
echo " Artifacts:"
286-
ls -lah "$OUTPUT_DIR"/*.so* "$OUTPUT_DIR"/kernel.c "$OUTPUT_DIR"/ABI_VERSION 2>/dev/null || \
287418
ls -lah "$OUTPUT_DIR"/*.so*
288419

289420
echo ""

0 commit comments

Comments
 (0)