Prevent region allocation from filling pools (#7912)

Recent customer issues have highlighted problems with storage
accounting, namely that while there are quotas and reservations for
individual Crucible regions, there's nothing set for the whole Crucible
dataset. Crucible _could_ end up using the whole disk, or some large
fraction of it, such that other users of the same U2 could be starved
out.

This commit adds a buffer to each zpool that the Crucible region
allocation query will not allocate into. This overhead will be set to
250G initially (see #7875 for reasoning) but could
also be modified with omdb.

Part of this commit's changes include using a CTE with
`regions_hard_delete`, which is much more efficient than the previous
for loop but has the effect of overwriting `size_used` for all datasets,
which will undo any time this column value was manually set to prevent
allocation for particular datasets / pools. Because of this, this commit
also adds a `no_provision` flag for a Crucible dataset: if it is set,
then the region allocation query will not allocate into that dataset.
This flag can be toggled with omdb.

Part of the upgrade to R14 will include a support procedure to address
if the addition of the control plane storage buffer of 250G causes a
Crucible dataset to be "overprovisioned", necessitating manually
requested region replacement requests to reduce the size allocated for a
particular Crucible dataset. This commit adds an omdb command to show
all overprovisioned crucible datasets, and changes the region listing
command so it can list regions for a particular dataset.

Fixes #3480

Author: jmpesp

.github/buildomat/jobs/deploy.sh

@@ -270,6 +270,9 @@ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml pkg/config.toml
 sed -E -i~ "s/(m2|u2)(.*\.vdev)/\/scratch\/\1\2/g" pkg/config.toml
 diff -u pkg/config.toml{~,} || true
 
+EXPECTED_ZPOOL_COUNT=$(grep -c -E 'u2.*\.vdev' pkg/config.toml)
+echo "expected number of zpools is ${EXPECTED_ZPOOL_COUNT}"
+
 SILO_NAME="$(sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml)"
 EXTERNAL_DNS_DOMAIN="$(sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml)"
 
@@ -397,6 +400,47 @@ until zoneadm list | grep nexus; do
 done
 echo "Waited for nexus: ${retry}s"
 
+# Wait for handoff, as zpools as inserted into the database during
+# `rack_initialize`, and the next omdb command requires them to exist in the
+# db.
+retry=0
+until grep "Handoff to Nexus is complete" /var/svc/log/oxide-sled-agent:default.log; do
+	if [[ $retry -gt 300 ]]; then
+		echo "Failed to handoff to Nexus after 300 seconds"
+		exit 1
+	fi
+	sleep 1
+	retry=$((retry + 1))
+done
+echo "Waited for handoff: ${retry}s"
+
+# Wait for the number of expected U2 zpools
+retry=0
+ACTUAL_ZPOOL_COUNT=$(pfexec zlogin oxz_switch /opt/oxide/omdb/bin/omdb db zpool list -i | wc -l)
+until [[ "${ACTUAL_ZPOOL_COUNT}" -eq "${EXPECTED_ZPOOL_COUNT}" ]];
+do
+	pfexec zlogin oxz_switch /opt/oxide/omdb/bin/omdb db zpool list
+	if [[ $retry -gt 300 ]]; then
+		echo "Failed to wait for ${EXPECTED_ZPOOL_COUNT} zpools after 300 seconds"
+		exit 1
+	fi
+	sleep 1
+	retry=$((retry + 1))
+	ACTUAL_ZPOOL_COUNT=$(pfexec zlogin oxz_switch /opt/oxide/omdb/bin/omdb db zpool list -i | wc -l)
+done
+
+# The bootstrap command creates a disk, so before that: adjust the control plane
+# storage buffer to 0 as the virtual hardware only creates 20G pools
+
+pfexec zlogin oxz_switch /opt/oxide/omdb/bin/omdb db zpool list
+
+for ZPOOL in $(pfexec zlogin oxz_switch /opt/oxide/omdb/bin/omdb db zpool list -i);
+do
+	pfexec zlogin oxz_switch /opt/oxide/omdb/bin/omdb -w db zpool set-storage-buffer "${ZPOOL}" 0
+done
+
+pfexec zlogin oxz_switch /opt/oxide/omdb/bin/omdb db zpool list
+
 export RUST_BACKTRACE=1
 export E2E_TLS_CERT IPPOOL_START IPPOOL_END
 eval "$(./target/debug/bootstrap)"