OCI sandboxes get an undocumented 512 MiB tmpfs at `/tmp` that ignores `--root-disk`

[nopcorn]

Summary

Every OCI-image sandbox gets an automatic RAM-backed tmpfs at /tmp, sized min(memory_mib / 4, 512) MiB. It is undocumented, it is not tunable except by replacing it with another mount, and it silently overrides the root-disk size the caller asked for. There is no way to opt out and have /tmp live on the writable root overlay.

Reproduce

msb run alpine --memory 32G --root-disk 20G -- df -h / /tmp

Actual:

Filesystem                Size      Used Available Use% Mounted on
overlay                  19.6G     52.0K     19.6G   0% /
tmpfs                   512.0M         0    512.0M   0% /tmp

In the guest, /proc/mounts shows tmpfs /tmp tmpfs rw,relatime,size=524288k, and both the kernel command line and agentd's environment carry MSB_TMPFS="/tmp:size=512".

Expected: /tmp on the 20 GiB overlay, like /var, /root and every other unmounted path.

Cause

sdk/rust/lib/sandbox/config.rs:23:

const DEFAULT_OCI_TMPFS_PATH: &str = "/tmp";
const DEFAULT_OCI_TMPFS_MAX_SIZE_MIB: u32 = 512;
const DEFAULT_OCI_TMPFS_MEMORY_DIVISOR: u32 = 4;

config.rs:450 and config.rs:547:

pub(crate) fn apply_runtime_defaults(&mut self) {
    if !matches!(self.spec.image, RootfsSource::Oci(_)) { return; }
    if self.spec.mounts.iter().any(|mount| guest_mount_is(mount, DEFAULT_OCI_TMPFS_PATH)) { return; }
    self.spec.mounts.push(VolumeMount::Tmpfs {
        guest: DEFAULT_OCI_TMPFS_PATH.to_string(),
        size_mib: Some(default_oci_tmpfs_size_mib(self.spec.resources.memory_mib)),
        options: MountOptions::default(),
    });
}

fn default_oci_tmpfs_size_mib(memory_mib: u32) -> u32 {
    (memory_mib / DEFAULT_OCI_TMPFS_MEMORY_DIVISOR).clamp(1, DEFAULT_OCI_TMPFS_MAX_SIZE_MIB)
}

Called from backend/local/sandbox/create.rs:100 and backend/local/sandbox/mod.rs:141, so it applies on create and on every start, and it reaches all SDKs since they are shims over this crate.

Introduced in #548 ("block-backed OCI rootfs via VMDK + EROFS fsmeta"), which records both the change and its motivation: "Auto-mount /tmp as tmpfs for OCI sandboxes, sized memory_mib / 4 and capped at 512 MiB, unless the user overrides it", and "bench_fs.py mounts Docker /tmp as tmpfs to match the new default".

Why this is a problem

The cap makes the formula inert. clamp(memory/4, 1, 512) saturates at 2 GiB of RAM. Every sandbox with 2 GiB or more gets exactly 512 MiB, whether it has 2 GiB or 128 GiB. Below that the formula does apply — --memory 1G yields a 256 MiB /tmp — but root_disk never has any effect.

It contradicts the root-disk knob. A caller who sets a 20 GiB root disk because builds run out of space still gets 512 MiB for the most common scratch location. Go link steps, cargo, npm, pip and image builds all stage in /tmp and hit ENOSPC at 512 MiB no matter how large the root disk is.

It is charged to guest RAM. A full /tmp costs 512 MiB of guest memory, which is not what a caller asking for disk space expects to spend.

It is undocumented. Nothing in the docs states that OCI sandboxes get an automatic /tmp tmpfs. docs/sandboxes/volumes.mdx shows --tmpfs /tmp:1G:noexec,nosuid,nodev as an example without noting that it replaces a default. The three constants carry no doc comments, in contrast to DEFAULT_BIND_QUOTA_MIB a few lines below, which is documented and points at its override.

There is no opt-out. apply_runtime_defaults skips only when a mount already targets /tmp, and the mount kinds are tmpfs (RAM), bind (host filesystem), named volume (host filesystem or separate image) and disk image (separate image). None of them is the root overlay, and the overlay is what you get when nothing is mounted — so /tmp on the root disk is unreachable. --tmpfs /tmp:0 does not help either; it suppresses the default and mounts a 0-byte tmpfs instead:

$ msb run alpine --memory 4G --root-disk 20G --tmpfs /tmp:0 -- df -h /tmp
Filesystem                Size      Used Available Use% Mounted on
tmpfs                        0         0         0   0% /tmp

Suggested fixes

Any one of these would resolve it:

1. Skip the default when the caller explicitly sized the OCI root disk — the strongest signal that a caller has thought about capacity.
2. Add an explicit opt-out: SandboxBuilder::no_default_tmpfs(), --no-default-tmpfs, or treating --tmpfs /tmp:0 as "do not mount".
3. Make the size configurable and uncapped, so --tmpfs /tmp:8G is a first-class tuning knob rather than an accidental discovery.
4. At minimum, document the behaviour in docs/sandboxes/volumes.mdx and add doc comments to the three constants explaining the performance rationale and the override.

Option 1 or 2 would be the most useful; 4 is worth doing regardless.

Environment

• Reproduced on msb 0.6.8, macOS arm64 host, libkrun backend
• Code paths above are from v0.6.9; the tmpfs logic is unchanged between 0.6.8 and 0.6.9
• SDK-independent — the behaviour lives in the shared Rust crate