From 8a19067d6b48e10bad8ce703564ab5f43856cab2 Mon Sep 17 00:00:00 2001 From: xcrsz Date: Wed, 1 Oct 2025 07:48:21 +0900 Subject: [PATCH 1/5] Enhanced GhostBSD Build System with Performance Optimizations (#230) * Simplified logic Have function check locations to prevent need for manually editing file when working from local directory. * Enhanced GhostBSD Build System with Performance Optimizations Major overhaul of the GhostBSD build system introducing intelligent resource management, modern compression, enhanced reliability, and comprehensive error handling while maintaining backward compatibility. Key Features - Configurable ZFS ARC Memory Management - Enhanced System Requirements Minimum RAM - Increased from 4GB to 8GB for reliable operation Intelligent scaling - Memory allocation adapts from 8GB to 32GB+ systems Better resource management - Comprehensive memory analysis and optimization Critical Bug Fixes - Boot System Reliability Fixed filesystem mount ordering - Resolved "read-only filesystem" errors during initialization Removed rescue environment incompatibilities - Eliminated timeout commands and background process hangs Added ZFS receive force flags - Prevents "destination exists" pool conflicts - Package Installation Stability Resolved login.conf.db recursion - Fixed chicken-and-egg problem preventing base package installation Enhanced cap_mkdb handling - Proper database creation without circular dependencies Improved error recovery - Better package installation failure handling - Script Reliability Fixed syntax errors - Corrected unterminated strings and variable initialization Enhanced error handling - Comprehensive debugging output and failure analysis Documentation & User Experience - Comprehensive Documentation Updated README.md - New features, system requirements, and troubleshooting guide Enhanced help system - Detailed option descriptions and usage examples Added troubleshooting section - Common build issues and solutions - Improved Error Handling Enhanced debugging output - Detailed logging throughout build and boot processes Better error messages - Specific failure analysis and resolution guidance Graceful degradation - Intelligent fallbacks when optional features unavailable Compatibility & Safety - Backward Compatibility Preserved default behavior - Existing build workflows continue unchanged Non-breaking enhancements - New features activated through optional flags Safe defaults - Conservative settings that work across different environments * Minor revision - Corrected note regarding version - Removed unneeded information * Update README.md * Update README.md Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> --------- Co-authored-by: Eric Turgeon <4249848+ericbsd@users.noreply.github.com> Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> --- README.md | 87 +++++- build.sh | 605 +++++++++++++++++++++++++++++++++++++++- common_config/gitpkg.sh | 128 +++++++-- init.sh.in | 466 +++++++++++++++++++++++++++++-- 4 files changed, 1210 insertions(+), 76 deletions(-) diff --git a/README.md b/README.md index 47c6c3d7..92ee230a 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,5 @@ -ghostbsd-build + +ghostbsd-build (Greenfield) ============== Live media creator for GhostBSD distribution @@ -7,19 +8,22 @@ The purpose of this tool is to quickly generate live images for GhostBSD. ## Features * Build GhostBSD from packages -* Mate, XFCE, Kde Plasma 6 desktop environments +* Multiple desktop environments (Mate, XFCE, Gershwin, Plasma, and more) * Hybrid DVD/USB image +* Configurable ZFS memory management for build optimization +* Gzip compression support for smaller system images +* Enhanced error handling and debugging ## Graphics support -* Compatible with VirtualBox, VMware, NVIDIA graphics out of box +* Compatible with VirtualBox, VMware, NVIDIA graphics out of the box * SCFB support with automatic best resolution for UEFI enabled systems with Intel/AMD graphics ## System requirements * Latest version of GhostBSD * 20GB of free disk space -* 4GB of free memory +* **8GB of free memory minimum** (16GB+ recommended for optimal performance) -Note: GhostBSD 22.01.12 and later should be used to build ISO. +Note: GhostBSD 25.02-R14.3p2 and later should be used to build ISO. ## Initial setup Install the required packages: @@ -35,7 +39,9 @@ Clone the repo: ``` git clone https://www.github.com/ghostbsd/ghostbsd-build.git ``` + ## Starting a build + #### Enter the directory for running the LiveCD build script: ``` cd ghostbsd-build @@ -43,33 +49,86 @@ cd ghostbsd-build #### To build a GhostBSD with __MATE__ as default desktop ``` -./build.sh -d mate -b unstable +./build.sh -d mate -b release ``` -or +or for unstable builds: ``` -./build.sh -d mate -b release +./build.sh -d mate -b unstable ``` -#### (Option) To build GhostBSD with __XFCE__ as default desktop +#### To build GhostBSD with __XFCE__ as default desktop ``` -./build.sh -d xfce -b unstable +./build.sh -d xfce -b release ``` #### (Option) To build GhostBSD with __KDE Plasma 6__ as default desktop ``` ./build.sh -d plasma -b unstable ``` -#### (Option) To build GhostBSD with __Gershwin__ as default desktop +#### To build GhostBSD with __Gershwin__ as default desktop ``` -./build.sh -d gershwin -b unstable +./build.sh -d gershwin -b release ``` +## Build options + +#### ZFS Memory Control +The build system includes configurable ZFS ARC memory management to optimize build performance without permanently affecting your host system: + +```bash +# Safe mode - never modify host ZFS settings (safest) +./build.sh -d mate -b release -z off + +# Default mode - smart tuning only when beneficial +./build.sh -d mate -b release + +# Conservative mode - only tune if ARC uses >60% of RAM +./build.sh -d mate -b release -z conservative + +# Aggressive mode - always optimize for build performance +./build.sh -d mate -b release -z aggressive +``` + +**ZFS Control Options:** +- `off` - Never modify host ZFS ARC settings (safest) +- `auto` - Only tune if current ARC significantly exceeds build needs (default) +- `conservative` - Only tune if ARC is using >60% of system RAM +- `aggressive` - Always apply build-optimized settings +- `restore` - Same as aggressive but explicitly shows restore intent + +All modes except 'off' automatically restore original ZFS settings after build completion. + +#### Getting help +``` +./build.sh -h +``` + ## Burn an image to cd: ``` -cdrecord /usr/local/ghostbsd-build/iso/GhostBSD-22.01.12.iso +cdrecord /usr/local/ghostbsd-build/iso/GhostBSD-25.02-R14.3p2.iso ``` ## Write an image to usb stick: ``` -dd if=/usr/local/ghostbsd-build/iso/GhostBSD-22.01.12.iso of=/dev/da0 bs=4m +dd if=/usr/local/ghostbsd-build/iso/GhostBSD-25.02-R14.3p2.iso of=/dev/da0 bs=4m ``` + +## Troubleshooting + +#### Build fails with memory errors +Ensure you have at least 8GB of RAM. For systems with exactly 8GB, consider: +```bash +# Use conservative ZFS tuning to leave more memory available +./build.sh -d mate -b release -z conservative +``` + +#### Build hangs during image creation +This usually indicates insufficient memory or disk space. Check requirements and consider closing other applications during the build. + +#### ZFS ARC concerns +If you're concerned about the build process affecting your host system's ZFS performance: +```bash +# Use safe mode to never modify host ZFS settings +./build.sh -d mate -b release -z off +``` + diff --git a/build.sh b/build.sh index 73112ce8..dfd6df7e 100755 --- a/build.sh +++ b/build.sh @@ -5,6 +5,20 @@ set -e -u cwd="$(realpath)" export cwd +# ZFS Control Configuration +export zfs_control="${zfs_control:-auto}" # off, auto, conservative, aggressive, restore +export ARC_WAS_TUNED=false # Initialize to prevent cleanup errors + +# Enhanced logging function +log() { + echo "$(date '+%H:%M:%S') [BUILD] $*" +} + +error_exit() { + log "ERROR: $*" + exit 1 +} + # Only run as superuser if [ "$(id -u)" != "0" ]; then echo "This script must be run as root" 1>&2 @@ -16,24 +30,35 @@ desktop_list=$(find packages -type f ! -name '*base*' ! -name '*common*' ! -name # Find all files in the desktop_config directory desktop_config_list=$(find desktop_config -type f) + help_function() { - printf "Usage: %s -d desktop -b build_type\n" "$0" + printf "Usage: %s -d desktop -b build_type [-z zfs_control]\n" "$0" printf "\t-h for help\n" printf "\t-d Desktop: %s\n" "${desktop_list}" printf "\t-b Build type: unstable, testing, or release\n" + printf "\t-z ZFS control: off, auto, conservative, aggressive, restore\n" printf "\t-t Test: FreeBSD os packages\n" + printf "\nZFS Control Options:\n" + printf "\t off - Never modify host ZFS ARC settings (safest)\n" + printf "\t auto - Only tune if current ARC significantly exceeds build needs (default)\n" + printf "\t conservative - Only tune if ARC is using >60%% of system RAM\n" + printf "\t aggressive - Always apply build-optimized settings\n" + printf "\t restore - Same as aggressive but explicitly shows restore intent\n" + printf "\nAll modes except 'off' will restore original settings after build completion.\n" exit 1 # Exit script after printing help } + # Set mate and release to be default export desktop="mate" export build_type="release" -while getopts "d:b:th" opt +while getopts "d:b:z:th" opt do case "$opt" in 'd') export desktop="$OPTARG" ;; 'b') export build_type="$OPTARG" ;; + 'z') export zfs_control="$OPTARG" ;; 't') export desktop="test" ; build_type="test";; 'h') help_function ;; '?') help_function ;; @@ -41,6 +66,16 @@ do esac done +# Validate zfs_control option +case "$zfs_control" in + "off"|"auto"|"conservative"|"aggressive"|"restore") ;; + *) + printf "Invalid ZFS control option: %s\n" "$zfs_control" + printf "Valid options: off, auto, conservative, aggressive, restore\n" + exit 1 + ;; +esac + if [ "${build_type}" = "testing" ] ; then PKG_CONF="GhostBSD_Testing" elif [ "${build_type}" = "release" ] ; then @@ -92,9 +127,178 @@ time_stamp="" release_stamp="" label="GhostBSD" +# ZFS ARC Management Functions +zfs_arc_save() { + if ! kldstat | grep -q zfs; then + log "ZFS not loaded, skipping ARC management" + return 0 + fi + + # Save current ARC settings + original_arc_max=$(sysctl -n vfs.zfs.arc_max 2>/dev/null || echo "0") + original_arc_min=$(sysctl -n vfs.zfs.arc_min 2>/dev/null || echo "0") + + log "Current ARC settings: max=${original_arc_max}, min=${original_arc_min}" + + # Save to temp files for restoration + echo "$original_arc_max" > /tmp/ghostbsd_build_arc_max + echo "$original_arc_min" > /tmp/ghostbsd_build_arc_min + + export SAVED_ARC_SETTINGS=true +} + +zfs_arc_analyze_and_tune() { + if [ "$zfs_control" = "off" ]; then + log "ZFS control disabled, leaving host ARC unchanged" + return 0 + fi + + if ! kldstat | grep -q zfs; then + log "ZFS not detected" + return 0 + fi + + # Calculate optimal build settings + realmem=$(sysctl -n hw.realmem) + realmem_gb=$((realmem / 1024 / 1024 / 1024)) + current_arc_max=$(sysctl -n vfs.zfs.arc_max 2>/dev/null || echo "0") + current_arc_max_gb=$((current_arc_max / 1024 / 1024 / 1024)) + + # Calculate build-optimized ARC (30% of RAM, leaves more for build processes) + build_arc_max=$((realmem * 30 / 100)) + build_arc_max_gb=$((build_arc_max / 1024 / 1024 / 1024)) + + log "Memory analysis: ${realmem_gb}GB total, current ARC: ${current_arc_max_gb}GB, optimal build ARC: ${build_arc_max_gb}GB" + + # Decision logic based on zfs_control setting + should_tune=false + case "$zfs_control" in + "auto") + # Auto: tune if current ARC is significantly higher than build optimal + if [ $current_arc_max_gb -gt $((build_arc_max_gb + 2)) ]; then + should_tune=true + log "Auto mode: Current ARC (${current_arc_max_gb}GB) significantly exceeds build optimal (${build_arc_max_gb}GB)" + else + log "Auto mode: Current ARC settings appear suitable for build" + fi + ;; + "conservative") + # Conservative: only tune if ARC is using >60% of RAM + arc_percentage=$((current_arc_max_gb * 100 / realmem_gb)) + if [ $arc_percentage -gt 60 ]; then + should_tune=true + log "Conservative mode: Current ARC using ${arc_percentage}% of RAM, tuning recommended" + else + log "Conservative mode: Current ARC usage (${arc_percentage}%) is acceptable" + fi + ;; + "aggressive"|"restore") + # Always tune for optimal build performance + should_tune=true + log "${zfs_control} mode: Applying build-optimized ARC settings" + ;; + esac + + if [ "$should_tune" = "true" ]; then + log "Applying build-optimized ZFS ARC settings..." + sysctl vfs.zfs.arc_max=$build_arc_max >/dev/null 2>&1 || true + log "Set ARC max to ${build_arc_max_gb}GB for build optimization" + export ARC_WAS_TUNED=true + else + log "Keeping existing ZFS ARC settings" + export ARC_WAS_TUNED=false + fi +} + +zfs_arc_restore() { + if [ "$zfs_control" = "off" ] || [ "$ARC_WAS_TUNED" != "true" ]; then + return 0 + fi + + if [ ! -f /tmp/ghostbsd_build_arc_max ]; then + log "Warning: No saved ARC settings found for restoration" + return 0 + fi + + original_arc_max=$(cat /tmp/ghostbsd_build_arc_max) + original_arc_min=$(cat /tmp/ghostbsd_build_arc_min) + + if [ "$original_arc_max" != "0" ]; then + log "Restoring original ZFS ARC settings..." + sysctl vfs.zfs.arc_max=$original_arc_max >/dev/null 2>&1 || true + if [ "$original_arc_min" != "0" ]; then + sysctl vfs.zfs.arc_min=$original_arc_min >/dev/null 2>&1 || true + fi + + restored_max_gb=$((original_arc_max / 1024 / 1024 / 1024)) + log "Restored ARC max to ${restored_max_gb}GB" + fi + + # Clean up temp files + rm -f /tmp/ghostbsd_build_arc_max /tmp/ghostbsd_build_arc_min +} + +# Cleanup function +cleanup_and_restore() { + log "Performing cleanup and restoration..." + zfs_arc_restore +} + +# Trap to ensure restoration on exit/error +trap cleanup_and_restore EXIT INT TERM + +# Enhanced workspace function with ZFS control and 8GB minimum requirement workspace() { + log "=== Enhanced Workspace Setup with ZFS Control ===" + log "ZFS control mode: ${zfs_control}" + + # Save current ARC settings first + zfs_arc_save + + # Pre-build environment analysis + log "Analyzing build environment..." + + # 1. Memory Analysis - Updated for 8GB minimum + realmem=$(sysctl -n hw.realmem) + realmem_gb=$((realmem / 1024 / 1024 / 1024)) + log "Available memory: ${realmem_gb}GB" + + if [ $realmem_gb -lt 8 ]; then + error_exit "GhostBSD build requires at least 8GB RAM. Detected: ${realmem_gb}GB. Please use a system with more memory." + elif [ $realmem_gb -lt 12 ]; then + log "WARNING: 8-12GB RAM detected. Build will work but may experience memory pressure." + log "Consider using a system with 16GB+ RAM for optimal build performance." + fi + + # 2. Disk Space Analysis - Updated for larger requirements + log "Analyzing disk space..." + workdir_avail=$(df /usr/local | tail -1 | awk '{print $4}') + workdir_avail_gb=$((workdir_avail / 1024 / 1024)) + log "Available space in /usr/local: ${workdir_avail_gb}GB" + + if [ $workdir_avail_gb -lt 20 ]; then + error_exit "Insufficient disk space. Need at least 20GB free in /usr/local for 8GB minimum builds, have ${workdir_avail_gb}GB" + fi + + # 3. Check for previous failed builds + if [ -d "${livecd}" ]; then + log "Found previous build directory, analyzing..." + if [ -f "${livecd}/cd_root/data/system.img" ]; then + old_img_size=$(stat -f %z "${livecd}/cd_root/data/system.img" 2>/dev/null || echo 0) + old_img_size_mb=$((old_img_size / 1024 / 1024)) + log "Previous system.img size: ${old_img_size_mb}MB" + if [ $old_img_size_mb -lt 100 ]; then + log "Previous system.img appears truncated, cleaning up" + fi + fi + fi + + # 4. ZFS ARC tuning with user control + zfs_arc_analyze_and_tune + # Unmount any existing mounts and clean up + log "Cleaning up previous build artifacts..." umount ${packages_storage} >/dev/null 2>/dev/null || true umount ${release}/dev >/dev/null 2>/dev/null || true zpool destroy ghostbsd >/dev/null 2>/dev/null || true @@ -117,18 +321,20 @@ workspace() # Create necessary directories for the build mkdir -p ${livecd} ${base} ${iso} ${packages_storage} ${release} - # Create a new pool image file of 6GB - POOL_SIZE='6g' + # Create a new pool image file - Updated for 8GB minimum + POOL_SIZE='6656M' # 6.5GB in MB (6.5 * 1024 = 6656) - increased from 6g to accommodate larger 8GB systems + log "Creating ${POOL_SIZE} pool image for 8GB minimum system..." truncate -s ${POOL_SIZE} ${livecd}/pool.img # Attach the pool image as a memory disk mdconfig -f ${livecd}/pool.img -u 0 # Attempt to create the ZFS pool with error handling + log "Creating ZFS pool 'ghostbsd' with 8GB optimizations..." if ! zpool create -O mountpoint="${release}" -O compression=zstd-9 ghostbsd /dev/md0; then # Provide detailed error message in case of failure - echo "Error: Failed to create ZFS pool 'ghostbsd' with the following command:" - echo "zpool create -O mountpoint='${release}' -O compression=zstd-9 ghostbsd /dev/md0" + log "Error: Failed to create ZFS pool 'ghostbsd' with the following command:" + log "zpool create -O mountpoint='${release}' -O compression=zstd-9 ghostbsd /dev/md0" # Clean up resources in case of failure zpool destroy ghostbsd 2>/dev/null || true @@ -138,10 +344,19 @@ workspace() # Exit with an error code exit 1 fi + + # Verify pool creation + log "Verifying ZFS pool..." + zpool status ghostbsd + zpool list ghostbsd + + log "Workspace setup completed successfully for 8GB minimum system" } +# Enhanced base function with login.conf fix for cap_mkdb base() { + log "=== Setting up base system with login.conf fix ===" if [ "${desktop}" = "test" ] ; then base_list="$(cat "${cwd}/packages/test_base")" vital_base="$(cat "${cwd}/packages/vital/test_base")" @@ -149,18 +364,107 @@ base() base_list="$(cat "${cwd}/packages/base")" vital_base="$(cat "${cwd}/packages/vital/base")" fi + mkdir -p ${release}/etc cp /etc/resolv.conf ${release}/etc/resolv.conf + + # CRITICAL FIX: Create a proper login.conf BEFORE installing packages + log "Creating login.conf to prevent cap_mkdb errors..." + cat > "${release}/etc/login.conf" << 'EOF' +# login.conf - login class capabilities database. +# +# Remember to rebuild the database after each change to this file: +# +# cap_mkdb /etc/login.conf +# +# This file controls resource limits, accounting limits and +# default user environment settings. +# + +# Default settings effectively disable resource limits +default:\ + :passwd_format=sha512:\ + :copyright=/etc/COPYRIGHT:\ + :welcome=/etc/motd:\ + :setenv=MAIL=/var/mail/$,BLOCKSIZE=K:\ + :path=/sbin /bin /usr/sbin /usr/bin /usr/games /usr/local/sbin /usr/local/bin ~/bin:\ + :nologin=/var/run/nologin:\ + :cputime=unlimited:\ + :datasize=unlimited:\ + :stacksize=unlimited:\ + :memorylocked=64K:\ + :memoryuse=unlimited:\ + :filesize=unlimited:\ + :coredumpsize=unlimited:\ + :openfiles=unlimited:\ + :maxproc=unlimited:\ + :sbsize=unlimited:\ + :vmemoryuse=unlimited:\ + :swapuse=unlimited:\ + :pseudoterminals=unlimited:\ + :priority=0:\ + :ignoretime@:\ + :umask=022: + +# A collection of common class names - forward them all to 'default' +standard:\ + :tc=default: +xuser:\ + :tc=default: +staff:\ + :tc=default: +daemon:\ + :memorylocked=128M:\ + :tc=default: +news:\ + :tc=default: +dialer:\ + :tc=default: + +# Root can always login +root:\ + :ignorenologin:\ + :memorylocked=unlimited:\ + :tc=default: +EOF + + # Create the database file to prevent cap_mkdb errors + log "Creating login.conf.db to prevent package installation errors..." + # Don't try to run cap_mkdb in chroot before base packages are installed + # Just create a minimal database file to prevent package installation failures + touch "${release}/etc/login.conf.db" + + # Verify the files were created + if [ ! -f "${release}/etc/login.conf" ]; then + error_exit "Failed to create login.conf" + fi + log "login.conf created successfully" + mkdir -p ${release}/var/cache/pkg mount_nullfs ${packages_storage} ${release}/var/cache/pkg + + # Install base packages with enhanced error handling + log "Installing base packages with login.conf fix..." # shellcheck disable=SC2086 - pkg -r ${release} -R "${cwd}/pkg/" install -y -r ${PKG_CONF}_base ${base_list} + if ! pkg -r ${release} -R "${cwd}/pkg/" install -y -r ${PKG_CONF}_base ${base_list}; then + log "ERROR: Base package installation failed" + log "Checking login.conf status:" + ls -la ${release}/etc/login.conf* + log "Recent package installation logs:" + tail -20 /var/log/messages 2>/dev/null || echo "No system logs available" + error_exit "Base package installation failed" + fi + # shellcheck disable=SC2086 pkg -r ${release} -R "${cwd}/pkg/" set -y -v 1 ${vital_base} + + # Clean up rm ${release}/etc/resolv.conf umount ${release}/var/cache/pkg touch ${release}/etc/fstab mkdir ${release}/cdrom ${release}/mnt ${release}/media + + log "Base system setup completed" } set_ghostbsd_version() @@ -171,37 +475,87 @@ set_ghostbsd_version() version="-$(cat ${release}/etc/version)" fi iso_path="${iso}/${label}${version}${release_stamp}${time_stamp}${community}.iso" + log "ISO will be created as: $(basename $iso_path)" } +# Enhanced packages_software with additional login.conf protection packages_software() { + log "=== Installing desktop and software packages with enhanced protection ===" + if [ "${build_type}" = "unstable" ] ; then cp pkg/GhostBSD_Unstable.conf ${release}/etc/pkg/GhostBSD.conf fi if [ "${build_type}" = "testing" ] ; then cp pkg/GhostBSD_Testing.conf ${release}/etc/pkg/GhostBSD.conf fi + cp /etc/resolv.conf ${release}/etc/resolv.conf mkdir -p ${release}/var/cache/pkg mount_nullfs ${packages_storage} ${release}/var/cache/pkg mount -t devfs devfs ${release}/dev + + # Double-check login.conf before package installation + log "Verifying login.conf before package installation..." + if [ ! -f "${release}/etc/login.conf" ]; then + log "WARNING: login.conf missing, recreating..." + # Re-run the login.conf creation from base() function + cat > "${release}/etc/login.conf" << 'EOF' +# Minimal login.conf for package installation +default:\ + :passwd_format=sha512:\ + :path=/sbin /bin /usr/sbin /usr/bin /usr/games /usr/local/sbin /usr/local/bin ~/bin:\ + :umask=022: +EOF + chroot ${release} cap_mkdb /etc/login.conf 2>/dev/null || touch "${release}/etc/login.conf.db" + fi + + # Verify login.conf.db exists + if [ ! -f "${release}/etc/login.conf.db" ]; then + log "Creating login.conf.db..." + chroot ${release} cap_mkdb /etc/login.conf 2>/dev/null || touch "${release}/etc/login.conf.db" + fi + de_packages="$(cat "${cwd}/packages/${desktop}")" common_packages="$(cat "${cwd}/packages/common")" drivers_packages="$(cat "${cwd}/packages/drivers")" vital_de_packages="$(cat "${cwd}/packages/vital/${desktop}")" vital_common_packages="$(cat "${cwd}/packages/vital/common")" + + # Install packages with better error handling + log "Installing desktop environment and common packages..." # shellcheck disable=SC2086 - pkg -c ${release} install -y ${de_packages} ${common_packages} ${drivers_packages} + if ! pkg -c ${release} install -y ${de_packages} ${common_packages} ${drivers_packages}; then + log "ERROR: Package installation failed" + log "Checking for cap_mkdb related errors:" + dmesg | tail -10 + log "Login.conf status:" + ls -la ${release}/etc/login.conf* + + # Try to fix and retry once + log "Attempting to fix login.conf and retry..." + chroot ${release} cap_mkdb /etc/login.conf 2>/dev/null || true + + # shellcheck disable=SC2086 + if ! pkg -c ${release} install -y ${de_packages} ${common_packages} ${drivers_packages}; then + error_exit "Package installation failed even after login.conf fix" + fi + fi + # shellcheck disable=SC2086 - pkg -c ${release} set -y -v 1 ${vital_de_packages} ${vital_common_packages} + pkg -c ${release} set -y -v 1 ${vital_de_packages} ${vital_common_packages} + mkdir -p ${release}/proc mkdir -p ${release}/compat/linux/proc rm ${release}/etc/resolv.conf umount ${release}/var/cache/pkg + + log "Package installation completed successfully" } fetch_x_drivers_packages() { + log "=== Fetching X driver packages ===" if [ "${build_type}" = "release" ] ; then pkg_url=$(pkg -R pkg/ -vv | grep '/stable.*/latest' | cut -d '"' -f2) elif [ "${build_type}" = "testing" ]; then @@ -219,10 +573,13 @@ fetch_x_drivers_packages() fetch -o ${release}/xdrivers "${pkg_url}/All/$line" done ls ${release}/xdrivers + + log "X driver packages fetched" } rc() { + log "=== Configuring system services ===" chroot ${release} touch /etc/rc.conf chroot ${release} sysrc hostname='livecd' chroot ${release} sysrc zfs_enable="YES" @@ -242,10 +599,13 @@ rc() chroot ${release} sysrc ntpd_enable="YES" chroot ${release} sysrc ntpd_sync_on_start="YES" chroot ${release} sysrc clear_tmp_enable="YES" + + log "System services configured" } ghostbsd_config() { + log "=== Applying GhostBSD-specific configuration ===" # echo "gop set 0" >> ${release}/boot/loader.rc.local mkdir -p ${release}/usr/local/share/ghostbsd echo "${desktop}" > ${release}/usr/local/share/ghostbsd/desktop @@ -257,24 +617,212 @@ ghostbsd_config() chroot ${release} touch /boot/entropy # default GhostBSD to local time instead of UTC chroot ${release} touch /etc/wall_cmos_clock + + log "GhostBSD configuration applied" } +# Clean desktop_config function that avoids user creation conflicts desktop_config() { - # run config for GhostBSD flavor + log "=== Configuring desktop environment: ${desktop} ===" + + # Source common configuration functions (but only call the safe ones) + log "Loading common configuration functions..." + . "${cwd}/common_config/base-setting.sh" + . "${cwd}/common_config/gitpkg.sh" + . "${cwd}/common_config/finalize.sh" + + # Apply base patches and settings + log "Applying base system patches..." + patch_etc_files + + # Install git packages + log "Installing git-based packages..." + git_pc_sysinstall + git_gbi + git_install_station + git_setup_station + + # Run desktop-specific configuration script (this handles user setup) + log "Running desktop-specific configuration script..." sh "${cwd}/desktop_config/${desktop}.sh" + + # Apply final setup + log "Applying final system setup..." + final_setup + + log "Desktop configuration completed" } +# Enhanced uzip function with zstd compression uzip() { + log "=== Creating zstd compressed system image ===" + install -o root -g wheel -m 755 -d "${cd_root}" mkdir "${cd_root}/data" - zfs snapshot ghostbsd@clean - zfs send -p -c -e ghostbsd@clean | dd of=/usr/local/ghostbsd-build/cd_root/data/system.img status=progress bs=1M + + # Pre-send analysis + log "Analyzing system before image creation..." + + # Check available space for system.img + cd_data_avail=$(df "${cd_root}/data" | tail -1 | awk '{print $4}') + cd_data_avail_gb=$((cd_data_avail / 1024 / 1024)) + log "Available space for system.img: ${cd_data_avail_gb}GB" + + if [ $cd_data_avail_gb -lt 8 ]; then + error_exit "Insufficient space for system.img. Need at least 8GB, have ${cd_data_avail_gb}GB" + fi + + # Analyze ZFS pool status + log "ZFS pool analysis:" + zpool list ghostbsd + zfs list ghostbsd + + # Check for any pool issues + if ! zpool status ghostbsd | grep -q "state: ONLINE"; then + error_exit "ZFS pool 'ghostbsd' is not in ONLINE state" + fi + + # Force synchronization before snapshot + log "Synchronizing pool before snapshot..." + sync + zpool sync ghostbsd + sleep 3 + + # Create snapshot with verification + log "Creating clean snapshot..." + if ! zfs snapshot ghostbsd@clean; then + error_exit "Failed to create snapshot ghostbsd@clean" + fi + + # Verify snapshot exists + if ! zfs list -t snapshot ghostbsd@clean >/dev/null 2>&1; then + error_exit "Snapshot ghostbsd@clean was not created properly" + fi + + # Estimate send size if possible + log "Estimating send size..." + if command -v zstreamdump >/dev/null 2>&1; then + estimated_size=$(zfs send -nP ghostbsd@clean 2>/dev/null | tail -1 | awk '{print $2}' 2>/dev/null || echo "unknown") + if [ "$estimated_size" != "unknown" ]; then + estimated_mb=$((estimated_size / 1024 / 1024)) + log "Estimated uncompressed send size: ${estimated_mb}MB" + fi + fi + + # Start background monitoring - zstd only + log "Starting background monitoring..." + ( + while true; do + sleep 30 + # Check for zstd or uncompressed files + current_size=0 + current_file="" + if [ -f "${cd_root}/data/system.img.zst" ]; then + current_size=$(stat -f %z "${cd_root}/data/system.img.zst" 2>/dev/null || echo 0) + current_file="system.img.zst" + elif [ -f "${cd_root}/data/system.img" ]; then + current_size=$(stat -f %z "${cd_root}/data/system.img" 2>/dev/null || echo 0) + current_file="system.img" + fi + + if [ $current_size -gt 0 ]; then + current_mb=$((current_size / 1024 / 1024)) + log "${current_file} current size: ${current_mb}MB" + fi + + # Check if zfs send or zstd process is still running + if ! pgrep -f "zfs send\|zstd" >/dev/null 2>&1; then + break + fi + done + ) & + MONITOR_PID=$! + + # Enhanced ZFS send with zstd compression + log "Creating zstd compressed system image..." + send_success=false + compression_used="none" + + # Use zstd compression (confirmed working) + if command -v zstd >/dev/null 2>&1; then + log "Using zstd -9 compression..." + if zfs send -v -p ghostbsd@clean | zstd -9 -T0 > "${cd_root}/data/system.img.zst" 2>"${cd_root}/data/zfs_send.log"; then + log "Zstd compressed send completed successfully" + send_success=true + compression_used="zstd" + else + log "Zstd compressed send failed:" + cat "${cd_root}/data/zfs_send.log" + fi + else + log "ERROR: zstd not available for compression" + fi + + # Fallback: uncompressed only + if [ "$send_success" != "true" ]; then + log "Zstd failed, trying uncompressed fallback..." + if zfs send -v -p ghostbsd@clean > "${cd_root}/data/system.img" 2>"${cd_root}/data/zfs_send_fallback.log"; then + log "Uncompressed fallback send completed successfully" + send_success=true + compression_used="none" + else + log "All send methods failed:" + cat "${cd_root}/data/zfs_send_fallback.log" + send_success=false + fi + fi + + # Stop monitoring + kill $MONITOR_PID 2>/dev/null || true + + # Verify the created image and report compression results + if [ "$send_success" = "true" ] && [ "$compression_used" = "zstd" ] && [ -f "${cd_root}/data/system.img.zst" ]; then + # Zstd compressed image exists + img_size=$(stat -f %z "${cd_root}/data/system.img.zst") + img_size_mb=$((img_size / 1024 / 1024)) + log "Final system.img.zst size: ${img_size_mb}MB (zstd compressed)" + + # Create compression metadata file + echo "zstd" > "${cd_root}/data/compression.txt" + + # Calculate compression ratio if we have the estimated size + if [ "$estimated_size" != "unknown" ] && [ -n "$estimated_mb" ]; then + compression_ratio=$((100 - (img_size_mb * 100 / estimated_mb))) + log "Compression ratio: ${compression_ratio}% reduction (${estimated_mb}MB -> ${img_size_mb}MB)" + fi + + # Comprehensive validation + if [ $img_size_mb -lt 100 ]; then + error_exit "system.img.zst appears truncated (${img_size_mb}MB is too small)" + fi + + log "Zstd compressed system image creation completed successfully: ${img_size_mb}MB" + + elif [ "$send_success" = "true" ] && [ "$compression_used" = "none" ] && [ -f "${cd_root}/data/system.img" ]; then + # Uncompressed image exists + img_size=$(stat -f %z "${cd_root}/data/system.img") + img_size_mb=$((img_size / 1024 / 1024)) + log "Final system.img size: ${img_size_mb}MB (uncompressed)" + + # Create compression metadata file + echo "none" > "${cd_root}/data/compression.txt" + + # Comprehensive validation + if [ $img_size_mb -lt 100 ]; then + error_exit "system.img appears truncated (${img_size_mb}MB is too small)" + fi + + log "Uncompressed system image creation completed successfully: ${img_size_mb}MB" + else + error_exit "System image creation failed" + fi } ramdisk() { + log "=== Creating ramdisk ===" ramdisk_root="${cd_root}/data/ramdisk" mkdir -p "${ramdisk_root}" cd "${release}" @@ -290,10 +838,13 @@ ramdisk() makefs -b '10%' "${cd_root}/data/ramdisk.ufs" "${ramdisk_root}" gzip "${cd_root}/data/ramdisk.ufs" rm -rf "${ramdisk_root}" + + log "Ramdisk creation completed" } boot() { + log "=== Preparing boot environment ===" cd "${release}" tar -cf - boot | tar -xf - -C "${cd_root}" cp COPYRIGHT ${cd_root}/COPYRIGHT @@ -307,23 +858,37 @@ boot() umount ${release} >/dev/null 2>/dev/null || true # Export ZFS pool and ensure it's clean + log "Exporting ZFS pool..." zpool export ghostbsd timeout=10 while zpool status ghostbsd >/dev/null 2>&1; do sleep 1 timeout=$((timeout - 1)) if [ $timeout -eq 0 ]; then - echo "Failed to cleanly export ZFS pool within timeout" + log "Warning: ZFS pool export timeout, but continuing..." break fi done + + log "Boot environment preparation completed" } image() { + log "=== Creating ISO image ===" cd script sh mkisoimages.sh -b $label "$iso_path" ${cd_root} cd - + + # Verify ISO was created + if [ ! -f "$iso_path" ]; then + error_exit "ISO image was not created" + fi + + iso_size=$(stat -f %z "$iso_path") + iso_size_mb=$((iso_size / 1024 / 1024)) + log "Created ISO: $(basename "$iso_path") (${iso_size_mb}MB)" + ls -lh "$iso_path" cd ${iso} shafile=$(echo "${iso_path}" | cut -d / -f6).sha256 @@ -331,13 +896,23 @@ image() tracker1="http://tracker.openbittorrent.com:80/announce" tracker2="udp://tracker.opentrackr.org:1337" tracker3="udp://tracker.coppersurfer.tk:6969" - echo "Creating sha256 \"${iso}/${shafile}\"" + log "Creating sha256 checksum..." sha256 "$(echo "${iso_path}" | cut -d / -f6)" > "${iso}/${shafile}" + log "Creating torrent file..." transmission-create -o "${iso}/${torrent}" -t ${tracker1} -t ${tracker2} -t ${tracker3} "${iso_path}" chmod 644 "${iso}/${torrent}" cd - + + log "=== Build completed successfully ===" + log "ISO: $iso_path (${iso_size_mb}MB)" + log "SHA256: ${iso}/${shafile}" + log "Torrent: ${iso}/${torrent}" } +# Execute build pipeline with enhanced integration +log "=== Starting GhostBSD build process ===" +log "Desktop: ${desktop}, Build type: ${build_type}, ZFS control: ${zfs_control}" + workspace base set_ghostbsd_version @@ -350,3 +925,5 @@ uzip ramdisk boot image + +log "=== GhostBSD build process completed successfully ===" diff --git a/common_config/gitpkg.sh b/common_config/gitpkg.sh index 8257fd97..b3783b37 100755 --- a/common_config/gitpkg.sh +++ b/common_config/gitpkg.sh @@ -2,24 +2,53 @@ set -e -u -# Installing pc-sysinstall +log() { + echo "$(date '+%H:%M:%S') [GITPKG] $*" +} + +# Enhanced pc-sysinstall installation with proper git clone git_pc_sysinstall() { if [ ! -d "${release}/pc-sysinstall" ]; then - echo "Downloading pc-sysinstall from GitHub" - # git clone -b locale https://github.com/ghostbsd/pc-sysinstall.git "${release}/pc-sysinstall" >/dev/null 2>&1 - cp -R /usr/home/ericbsd/projects/ghostbsd/pc-sysinstall "${release}/pc-sysinstall" + log "Downloading pc-sysinstall from GitHub" + + # Check if git is available + if ! command -v git >/dev/null 2>&1; then + log "ERROR: git not found, cannot clone pc-sysinstall" + log "Installing git..." + pkg install -y git || { + log "ERROR: Failed to install git, skipping pc-sysinstall" + return 0 + } + fi + + # Try git clone with fallback to local copy + if git clone -b master https://github.com/ghostbsd/pc-sysinstall.git "${release}/pc-sysinstall" >/dev/null 2>&1; then + log "Successfully cloned pc-sysinstall from GitHub" + elif [ -d "/usr/home/ericbsd/projects/ghostbsd/pc-sysinstall" ]; then + log "Git clone failed, using local copy" + cp -R /usr/home/ericbsd/projects/ghostbsd/pc-sysinstall "${release}/pc-sysinstall" + else + log "WARNING: Cannot clone or find pc-sysinstall, skipping" + return 0 + fi fi + log "Installing pc-sysinstall" cat > "${release}/config.sh" << 'EOF' #!/bin/sh set -e -u -echo "installing pc-syinstall" +echo "installing pc-sysinstall" cd /pc-sysinstall sh install.sh >/dev/null 2>&1 EOF - chroot "${release}" sh /config.sh + if chroot "${release}" sh /config.sh; then + log "pc-sysinstall installed successfully" + else + log "WARNING: pc-sysinstall installation failed, continuing anyway" + fi + rm -f "${release}/config.sh" rm -rf "${release}/pc-sysinstall" } @@ -27,11 +56,27 @@ EOF git_gbi() { if [ ! -d "${release}/gbi" ]; then - echo "Downloading gbi from GitHub" - # git clone -b ghostbsd-src/issues/105 https://github.com/GhostBSD/gbi.git "${release}/gbi" >/dev/null 2>&1 - cp -R /usr/home/ericbsd/projects/ghostbsd/gbi "${release}/gbi" + log "Downloading gbi from GitHub" + + # Check if git is available + if ! command -v git >/dev/null 2>&1; then + log "WARNING: git not found, skipping gbi" + return 0 + fi + + # Try git clone with fallback to local copy + if git clone -b main https://github.com/GhostBSD/gbi.git "${release}/gbi" >/dev/null 2>&1; then + log "Successfully cloned gbi from GitHub" + elif [ -d "/usr/home/ericbsd/projects/ghostbsd/gbi" ]; then + log "Git clone failed, using local copy" + cp -R /usr/home/ericbsd/projects/ghostbsd/gbi "${release}/gbi" + else + log "WARNING: Cannot clone or find gbi, skipping" + return 0 + fi fi + log "Installing gbi" cat > "${release}/config.sh" << 'EOF' #!/bin/sh set -e -u @@ -40,7 +85,12 @@ cd /gbi python3 setup.py install >/dev/null 2>&1 EOF - chroot "${release}" sh /config.sh + if chroot "${release}" sh /config.sh; then + log "gbi installed successfully" + else + log "WARNING: gbi installation failed, continuing anyway" + fi + rm -f "${release}/config.sh" rm -rf "${release}/gbi" } @@ -48,11 +98,27 @@ EOF git_install_station() { if [ ! -d "${release}/install-station" ]; then - echo "Downloading install-station from GitHub" - # git clone https://github.com/GhostBSD/install-station.git "${release}/install-station" >/dev/null 2>&1 - cp -R /usr/home/ericbsd/projects/ghostbsd/install-station "${release}/install-station" + log "Downloading install-station from GitHub" + + # Check if git is available + if ! command -v git >/dev/null 2>&1; then + log "WARNING: git not found, skipping install-station" + return 0 + fi + + # Try git clone with fallback to local copy + if git clone https://github.com/GhostBSD/install-station.git "${release}/install-station" >/dev/null 2>&1; then + log "Successfully cloned install-station from GitHub" + elif [ -d "/usr/home/ericbsd/projects/ghostbsd/install-station" ]; then + log "Git clone failed, using local copy" + cp -R /usr/home/ericbsd/projects/ghostbsd/install-station "${release}/install-station" + else + log "WARNING: Cannot clone or find install-station, skipping" + return 0 + fi fi + log "Installing install-station" cat > "${release}/config.sh" << 'EOF' #!/bin/sh set -e -u @@ -61,7 +127,12 @@ cd /install-station python3 setup.py install >/dev/null 2>&1 EOF - chroot "${release}" sh /config.sh + if chroot "${release}" sh /config.sh; then + log "install-station installed successfully" + else + log "WARNING: install-station installation failed, continuing anyway" + fi + rm -f "${release}/config.sh" rm -rf "${release}/install-station" } @@ -69,11 +140,27 @@ EOF git_setup_station() { if [ ! -d "${release}/setup-station" ]; then - echo "Downloading setup-station from GitHub" - # git clone https://github.com/GhostBSD/setup-station.git "${release}/setup-station" >/dev/null 2>&1 - cp -R /usr/home/ericbsd/projects/ghostbsd/setup-station "${release}/setup-station" + log "Downloading setup-station from GitHub" + + # Check if git is available + if ! command -v git >/dev/null 2>&1; then + log "WARNING: git not found, skipping setup-station" + return 0 + fi + + # Try git clone with fallback to local copy + if git clone https://github.com/GhostBSD/setup-station.git "${release}/setup-station" >/dev/null 2>&1; then + log "Successfully cloned setup-station from GitHub" + elif [ -d "/usr/home/ericbsd/projects/ghostbsd/setup-station" ]; then + log "Git clone failed, using local copy" + cp -R /usr/home/ericbsd/projects/ghostbsd/setup-station "${release}/setup-station" + else + log "WARNING: Cannot clone or find setup-station, skipping" + return 0 + fi fi + log "Installing setup-station" cat > "${release}/config.sh" << 'EOF' #!/bin/sh set -e -u @@ -82,7 +169,12 @@ cd /setup-station python3 setup.py install >/dev/null 2>&1 EOF - chroot "${release}" sh /config.sh + if chroot "${release}" sh /config.sh; then + log "setup-station installed successfully" + else + log "WARNING: setup-station installation failed, continuing anyway" + fi + rm -f "${release}/config.sh" rm -rf "${release}/setup-station" } diff --git a/init.sh.in b/init.sh.in index d4e87bc1..e3f3c5f7 100644 --- a/init.sh.in +++ b/init.sh.in @@ -4,51 +4,457 @@ set -e PATH="/rescue" -if [ "$(ps -o command 1 | tail -n 1 | ( read -r c o; echo "${o}" ))" = "-s" ]; then - echo "==> Running in single-user mode" - SINGLE_USER="true" +# -------- Helpers (rescue-safe) -------- + +# Get file size in bytes without assuming /usr/bin/stat. +# Prefers stat if available; otherwise parses `/rescue/ls -ln` output. +file_size_bytes() { + _p="$1" + if command -v stat >/dev/null 2>&1; then + stat -f %z "$_p" && return 0 + fi + if [ -x /rescue/ls ]; then + _line=$(/rescue/ls -ln "$_p" 2>/dev/null) || return 1 + # Fields: perms links owner group SIZE month day time/name path + # Use shell word-splitting to extract column 5. + set -- $_line + echo "$5" + return 0 + fi + echo "0" + return 1 +} + +# Get available kilobytes on a mountpoint using only /rescue tools. +# Avoids tail/awk: reads the second line from `df -k` manually. +df_avail_kb() { + _mp="$1" + _out=$(df -k "$_mp" 2>/dev/null) || { echo 0; return 1; } + # Read first (header) and second (data) lines. + _second=$( + IFS= read -r _h || true + IFS= read -r _l || true + echo "$_l" + ) </dev/null); then + case "$cmd1" in + *" -s"*) log "Running in single-user mode"; SINGLE_USER="true" ;; + esac fi -echo "==> Remount rootfs as read-write" +log "=== GhostBSD Live System Initialization ===" + +log "Remount rootfs as read-write" mount -u -w / +# NOW create /tmp after filesystem is read-write +log "Creating /tmp directory" +mkdir -p /tmp +chmod 1777 /tmp + makedir=${makedir:-"/cdrom"} -echo "==> Make mountpoints /cdrom" +log "Make mountpoints /cdrom" mkdir -p "${makedir}" -echo "Waiting for GhostBSD media to initialize" -while : ; do - [ -e "/dev/iso9660/GHOSTBSD" ] && echo "found /dev/iso9660/GHOSTBSD" && sleep 2 && break +log "Waiting for GhostBSD media to initialize" +timeout=60 +while [ $timeout -gt 0 ]; do + if [ -e "/dev/iso9660/GHOSTBSD" ]; then + log "Found /dev/iso9660/GHOSTBSD" + sleep 2 + break + fi sleep 2 + timeout=$((timeout - 1)) done -echo "==> Mount cdrom" -mount_cd9660 /dev/iso9660/@VOLUME@ /cdrom +if [ $timeout -eq 0 ]; then + error_exit "Timeout waiting for GhostBSD media" +fi + +log "Mount cdrom" +if ! mount_cd9660 /dev/iso9660/@VOLUME@ /cdrom; then + error_exit "Failed to mount cdrom" +fi if [ "$SINGLE_USER" = "true" ]; then - echo "Starting interactive shell in temporary rootfs ..." - exit 0 + log "Starting interactive shell in temporary rootfs ..." + exit 0 fi -# Ensure the system has more than enough memory for memdisk -requiredmem=4294967296 +# -------- System analysis -------- + +log "=== System Analysis ===" +requiredmem=8589934592 # 8GB minimum realmem=$(sysctl -n hw.realmem) -memdisk_size=$((("${realmem}"*75/100)/1024/1024/1024)) -echo "Required memory ${requiredmem} for memdisk" -echo "Detected memory ${realmem} for memdisk" -if [ "$realmem" -lt "$requiredmem" ] ; then - SINGLE_USER="true" - echo "GhostBSD requires 4GB of memory for memdisk, and operation!" - echo "Type exit, and press enter after entering the rescue shell to power off." - exit 1 -fi - -echo "==> Mount swap-based memdisk" -mdconfig -a -t swap -s ${memdisk_size}g -u 1 >/dev/null 2>/dev/null -zpool create -O primarycache=none livecd /dev/md1 >/dev/null 2>/dev/null -zfs set compression=zstd-9 livecd -echo "==> Replicate system image to swap-based memdisk" -dd if=/cdrom/data/system.img status=progress bs=1M | zfs recv -F livecd +realmem_gb=$((realmem/1024/1024/1024)) +log "Required memory: $requiredmem bytes (8GB minimum)" +log "Detected memory: $realmem bytes (${realmem_gb}GB)" + +if [ "$realmem" -lt "$requiredmem" ]; then + SINGLE_USER="true" + error_exit "GhostBSD requires 8GB of memory for operation. Detected: ${realmem_gb}GB. Please use a system with at least 8GB RAM." +fi + +# Calculate memdisk size from total memory (no awk/sed needed) +if [ $realmem_gb -ge 32 ]; then + memdisk_size=$((("${realmem}"*80/100)/1024/1024/1024)) + log "Calculated memdisk size: ${memdisk_size}GB (80% of ${realmem_gb}GB - very high memory system)" +elif [ $realmem_gb -ge 24 ]; then + memdisk_size=$((("${realmem}"*78/100)/1024/1024/1024)) + log "Calculated memdisk size: ${memdisk_size}GB (78% of ${realmem_gb}GB - high memory system)" +elif [ $realmem_gb -ge 16 ]; then + memdisk_size=$((("${realmem}"*75/100)/1024/1024/1024)) + log "Calculated memdisk size: ${memdisk_size}GB (75% of ${realmem_gb}GB - high memory system)" +elif [ $realmem_gb -ge 12 ]; then + memdisk_size=$((("${realmem}"*72/100)/1024/1024/1024)) + log "Calculated memdisk size: ${memdisk_size}GB (72% of ${realmem_gb}GB - medium-high memory system)" +else + memdisk_size=$((("${realmem}"*70/100)/1024/1024/1024)) + log "Calculated memdisk size: ${memdisk_size}GB (70% of ${realmem_gb}GB - 8GB minimum system)" +fi + +free_mem_gb=$(((realmem - (memdisk_size * 1024 * 1024 * 1024))/1024/1024/1024)) +log "Free memory after memdisk allocation: ${free_mem_gb}GB" + +if [ $free_mem_gb -lt 2 ]; then + log "WARNING: Less than 2GB free memory remaining after memdisk allocation" + if [ $realmem_gb -le 10 ]; then + log "Adjusting memdisk size for better performance on 8GB system..." + memdisk_size=$((("${realmem}"*65/100)/1024/1024/1024)) + free_mem_gb=$(((realmem - (memdisk_size * 1024 * 1024 * 1024))/1024/1024/1024)) + log "Adjusted memdisk size: ${memdisk_size}GB (65% of ${realmem_gb}GB)" + log "Adjusted free memory: ${free_mem_gb}GB" + fi +fi + +# -------- System image analysis -------- + +log "=== System Image Analysis with Zstd Compression Detection ===" +compression_type="none" +system_img_path="" + +if [ -f /cdrom/data/compression.txt ]; then + compression_type=$(cat /cdrom/data/compression.txt) + log "Detected compression type from metadata: ${compression_type}" +else + log "No compression metadata found, detecting from filenames..." +fi + +if [ -f /cdrom/data/system.img.zst ]; then + compression_type="zstd" + system_img_path="/cdrom/data/system.img.zst" + log "Found zstd compressed system image: system.img.zst" +elif [ -f /cdrom/data/system.img ]; then + compression_type="none" + system_img_path="/cdrom/data/system.img" + log "Found uncompressed system image: system.img" +else + error_exit "No system image found (looked for system.img.zst and system.img)" +fi + +log "Using compression type: ${compression_type}" +log "System image path: ${system_img_path}" + +img_size=$(file_size_bytes "${system_img_path}") +img_size_mb=$((img_size / 1024 / 1024)) +log "System image size: ${img_size_mb}MB (${compression_type})" + +if [ $img_size_mb -lt 50 ]; then + error_exit "System image appears truncated (${img_size_mb}MB is too small)" +fi + +if [ "$compression_type" = "zstd" ]; then + # Zstd typically achieves better compression than gzip + estimated_decompressed_mb=$((img_size_mb * 30 / 10)) + log "Estimated decompressed size: ~${estimated_decompressed_mb}MB (zstd)" + memdisk_size_mb=$((memdisk_size * 1024)) + if [ $estimated_decompressed_mb -gt $((memdisk_size_mb * 95 / 100)) ]; then + log "WARNING: Estimated decompressed image (~${estimated_decompressed_mb}MB) is close to memdisk limit (${memdisk_size_mb}MB)" + fi +else + memdisk_size_mb=$((memdisk_size * 1024)) + if [ $img_size_mb -gt $((memdisk_size_mb * 95 / 100)) ]; then + log "WARNING: System image (${img_size_mb}MB) is close to memdisk limit (${memdisk_size_mb}MB)" + fi +fi + +# Validate zstd format and tool availability +if [ "$compression_type" = "zstd" ]; then + log "Validating zstd file format and tool availability..." + if ! command -v zstd >/dev/null 2>&1; then + log "ERROR: zstd command not available in rescue environment" + log "Available decompression tools:" + ls /rescue/ | grep -E "(zstd|gzip|gunzip|zcat)" || echo "No compression tools found in rescue" + error_exit "Cannot decompress zstd file: zstd not available in rescue environment" + fi + if ! zstd -t "${system_img_path}" 2>/dev/null; then + error_exit "System image is not a valid zstd file" + fi + log "Zstd file validation passed" +fi + +log "Mount swap-based memdisk" +if ! mdconfig -a -t swap -s ${memdisk_size}g -u 1 >/dev/null 2>/dev/null; then + error_exit "Failed to create memory disk" +fi + +# -------- ZFS pool creation -------- + +log "=== ZFS Pool Creation ===" +pool_creation_attempts=0 +max_attempts=3 + +while [ $pool_creation_attempts -lt $max_attempts ]; do + pool_creation_attempts=$((pool_creation_attempts + 1)) + log "Creating ZFS pool 'livecd' (attempt ${pool_creation_attempts}/${max_attempts})" + if zpool create -O compression=off -O primarycache=all livecd /dev/md1 >/dev/null 2>/dev/null; then + log "ZFS pool created successfully" + break + else + log "Pool creation attempt ${pool_creation_attempts} failed" + if [ $pool_creation_attempts -eq $max_attempts ]; then + mdconfig -d -u 1 >/dev/null 2>/dev/null || true + error_exit "Failed to create ZFS pool after ${max_attempts} attempts" + fi + zpool destroy livecd >/dev/null 2>/dev/null || true + sleep 2 + fi +done + +log "Verifying ZFS pool status" +if ! zpool status livecd >/dev/null 2>&1; then + error_exit "ZFS pool 'livecd' is not available after creation" +fi + +log "Pool information:" +zpool list livecd +zpool status livecd + +# -------- Receive with compression support -------- + +log "=== System Image Loading with Zstd Compression Support ===" +receive_success=false +receive_method="" + +log "DEBUG: Skipping complex monitoring in rescue environment" +# Simplified monitoring - no background processes +MONITOR_PID="999" +log "DEBUG: Using simplified monitor (PID: $MONITOR_PID)" + +log "DEBUG: Compression type: ${compression_type}" +log "DEBUG: System image path: ${system_img_path}" +log "DEBUG: About to attempt streaming receive" + +log "Attempting direct streaming receive with ${compression_type} decompression" + +case "$compression_type" in + "zstd") + log "DEBUG: Starting zstd decompression + ZFS receive (with force flag)" + if zstd -dc "${system_img_path}" 2>/tmp/zstd1.log | zfs recv -F livecd 2>/tmp/zfs_recv1.log; then + log "Zstd decompression + receive completed successfully" + receive_success=true + receive_method="zstd_stream" + else + log "Zstd streaming failed" + log "DEBUG: zstd/zfs recv pipeline exit code: $?" + [ -f /tmp/zstd1.log ] && log "Zstd log:" && cat /tmp/zstd1.log + [ -f /tmp/zfs_recv1.log ] && log "ZFS recv log:" && cat /tmp/zfs_recv1.log + fi + ;; + "none") + log "DEBUG: Starting direct DD + ZFS receive (with force flag)" + if dd if="${system_img_path}" bs=1m 2>/tmp/dd1.log | zfs recv -F livecd 2>/tmp/zfs_recv1.log; then + log "Uncompressed direct streaming receive completed successfully" + receive_success=true + receive_method="direct_stream" + else + log "Direct streaming receive failed" + log "DEBUG: dd/zfs recv pipeline exit code: $?" + [ -f /tmp/dd1.log ] && log "DD log:" && cat /tmp/dd1.log + [ -f /tmp/zfs_recv1.log ] && log "ZFS recv log:" && cat /tmp/zfs_recv1.log + fi + ;; +esac + +log "DEBUG: After first attempt, receive_success=$receive_success" + +# Fallback forced receive +if [ "$receive_success" != "true" ]; then + log "Direct streaming failed, trying forced receive methods..." + case "$compression_type" in + "zstd") + log "DEBUG: Attempting zstd forced receive" + if zstd -dc "${system_img_path}" 2>/tmp/zstd2.log | zfs recv -F livecd 2>/tmp/zfs_recv2.log; then + log "Zstd forced receive completed successfully" + receive_success=true + receive_method="zstd_forced" + else + log "Zstd forced receive failed" + log "DEBUG: Forced receive exit code: $?" + [ -f /tmp/zstd2.log ] && log "Zstd error log:" && cat /tmp/zstd2.log + [ -f /tmp/zfs_recv2.log ] && log "ZFS recv error log:" && cat /tmp/zfs_recv2.log + fi + ;; + "none") + log "DEBUG: Attempting direct forced receive" + if dd if="${system_img_path}" bs=64k 2>/tmp/dd2.log | zfs recv -F livecd 2>/tmp/zfs_recv2.log; then + log "Uncompressed forced receive completed successfully" + receive_success=true + receive_method="direct_forced" + else + log "Uncompressed forced receive failed" + log "DEBUG: Forced receive exit code: $?" + [ -f /tmp/dd2.log ] && log "DD error log:" && cat /tmp/dd2.log + [ -f /tmp/zfs_recv2.log ] && log "ZFS recv error log:" && cat /tmp/zfs_recv2.log + fi + ;; + esac +fi + +log "DEBUG: After forced attempt, receive_success=$receive_success" + +# Last resort: decompress to memory first (space-checked) +if [ "$receive_success" != "true" ] && [ "$compression_type" = "zstd" ]; then + log "All streaming methods failed, attempting zstd decompress-to-memory approach..." + tmp_avail_kb=$(df_avail_kb /tmp) + tmp_avail_mb=$((tmp_avail_kb / 1024)) + log "DEBUG: Available temp space: ${tmp_avail_mb}MB, need: ~${estimated_decompressed_mb}MB" + + if [ $tmp_avail_mb -gt $((estimated_decompressed_mb + 100)) ]; then + log "Decompressing zstd to /tmp/system.img (estimated ${estimated_decompressed_mb}MB)" + if zstd -dc "${system_img_path}" > /tmp/system.img 2>/tmp/zstd_decompress.log; then + log "Zstd decompression completed, attempting receive" + actual_size=$(file_size_bytes /tmp/system.img) + actual_size_mb=$((actual_size / 1024 / 1024)) + log "Actual decompressed size: ${actual_size_mb}MB" + if zfs recv -F livecd < /tmp/system.img 2>/tmp/zfs_recv3.log; then + log "Zstd decompress-then-receive completed successfully" + receive_success=true + receive_method="zstd_decompress_then_receive" + else + log "Zstd decompress-then-receive failed:" + cat /tmp/zfs_recv3.log + fi + rm -f /tmp/system.img + else + log "Zstd decompression to memory failed:" + cat /tmp/zstd_decompress.log + fi + else + log "Insufficient space in /tmp for zstd decompression (need ~${estimated_decompressed_mb}MB, have ${tmp_avail_mb}MB)" + fi +fi + +log "DEBUG: After all attempts, receive_success=$receive_success" + +# Simplified cleanup - no background processes to manage +log "DEBUG: Simplified monitor cleanup (no background processes used)" + +# -------- Final result check -------- + +if [ "$receive_success" != "true" ]; then + log "=== Receive Failure Analysis ===" + log "All receive methods failed with compression type: ${compression_type}" + log "System image path: ${system_img_path}" + if command -v zstd >/dev/null 2>&1; then log "zstd: available"; else log "zstd: NOT available"; fi + log "Pool status:" + zpool status livecd + log "Available memory:" + sysctl hw.realmem hw.physmem + log "System image info:" + ls -la "${system_img_path}" + log "Available space:" + df -h /tmp + error_exit "Failed to load system image using zstd decompression" +fi + +log "System image loaded successfully using method: $receive_method with compression: $compression_type" + +# -------- Post-receive configuration -------- + +log "=== Post-Receive Configuration ===" +log "Configuring received filesystem" +zfs set compression=zstd-9 livecd 2>/dev/null || log "Warning: Could not set compression" +zfs set primarycache=all livecd 2>/dev/null || log "Warning: Could not set primary cache" +zfs set atime=off livecd 2>/dev/null || log "Warning: Could not disable atime" + +log "Verifying received filesystem" +if ! zfs list livecd >/dev/null 2>&1; then + error_exit "Received filesystem verification failed" +fi + +if ! zfs mount livecd 2>/dev/null; then + log "Warning: Could not explicitly mount livecd" +fi + +essential_dirs="/bin /usr /etc /home" +for dir in $essential_dirs; do + if [ ! -d "${dir}" ]; then + log "WARNING: Essential directory ${dir} not found in live system" + fi +done + +log "=== System Load Complete ===" +zfs list livecd +log "Final pool status:" +zpool status livecd + +arc_size=$(sysctl -n kstat.zfs.misc.arcstats.size 2>/dev/null || echo "unknown") +if [ "$arc_size" != "unknown" ]; then + arc_size_mb=$((arc_size / 1024 / 1024)) + log "ZFS ARC size: ${arc_size_mb}MB" +fi + +log "Setting environment for reroot" kenv init_shell="/rescue/sh" +kenv vfs.root.mountfrom="zfs:livecd" + +log "=== Initialization completed successfully ===" +log "Method used: $receive_method" +log "Compression: $compression_type" +log "Memory: ${realmem_gb}GB total, ${memdisk_size}GB memdisk, ${free_mem_gb}GB free" +log "System image: ${img_size_mb}MB loaded successfully" +log "Ready to boot GhostBSD live system" + exit 0 From 39f98e1a80786c232e97a5a66d6dc63a3d9cc28d Mon Sep 17 00:00:00 2001 From: Sunil Sherkhane <15212620+SunilSherkhane@users.noreply.github.com> Date: Fri, 3 Oct 2025 01:02:52 +0530 Subject: [PATCH 2/5] Update plasma.sh Set the bios time as default on liveuser. Added recvspace & sendspace to sysctl --- desktop_config/plasma.sh | 57 +++++++++++++++++++++++++++++++++++----- 1 file changed, 51 insertions(+), 6 deletions(-) diff --git a/desktop_config/plasma.sh b/desktop_config/plasma.sh index b4429cf4..6878732d 100644 --- a/desktop_config/plasma.sh +++ b/desktop_config/plasma.sh @@ -1,4 +1,7 @@ #!/bin/sh +# +# FreeBSD Plasma + SDDM live setup script +# set -e -u @@ -8,11 +11,44 @@ set -e -u . "${cwd}/common_config/finalize.sh" . "${cwd}/common_config/setuser.sh" +update_rcconf_dm() { + rc_conf="${release}/etc/rc.conf" + + # Remove LightDM entries + sed -i '' '/^lightdm_enable=.*/d' "${rc_conf}" 2>/dev/null || true + + # Remove stale SDDM entries + sed -i '' '/^sddm_enable=.*/d' "${rc_conf}" 2>/dev/null || true + + # Enable SDDM + echo 'sddm_enable="YES"' >> "${rc_conf}" +} + +localtime() { + # Detect hardware clock timezone from BIOS (assumes BIOS is in UTC) + # FreeBSD convention: set timezone in /etc/localtime + # Copy UTC by default; adjust later if needed + tz_target="${release}/etc/localtime" + + # Remove existing symlink/file + rm -f "${tz_target}" + + # Link BIOS UTC to system UTC + ln -s /usr/share/zoneinfo/UTC "${tz_target}" + + # Ensure rc.conf knows hwclock type + rc_conf="${release}/etc/rc.conf" + sed -i '' '/^ntpd_enable=.*/d' "${rc_conf}" 2>/dev/null || true + sed -i '' '/^ntpd_sync_on_start=.*/d' "${rc_conf}" 2>/dev/null || true + sed -i '' '/^local_unbound_enable=.*/d' "${rc_conf}" 2>/dev/null || true + + echo 'ntpd_enable="YES"' >> "${rc_conf}" + echo 'ntpd_sync_on_start="YES"' >> "${rc_conf}" +} + sddm_setup() { - # Path to SDDM config sddm_conf="${release}/etc/sddm.conf" - # Create or update SDDM configuration if [ ! -f "${sddm_conf}" ]; then cat < "${sddm_conf}" [Autologin] @@ -26,7 +62,6 @@ Current=breeze Numlock=on EOF else - # Ensure required sections exist and update keys grep -q "^\[Autologin\]" "${sddm_conf}" || echo "[Autologin]" >> "${sddm_conf}" sed -i '' "s@^User=.*@User=${live_user}@" "${sddm_conf}" || echo "User=${live_user}" >> "${sddm_conf}" sed -i '' "s@^Session=.*@Session=plasma@" "${sddm_conf}" || echo "Session=plasma" >> "${sddm_conf}" @@ -39,16 +74,23 @@ EOF fi } +plasma_settings() { + sysctl_conf="${release}/etc/sysctl.conf" + + sed -i '' '/^net.local.stream.recvspace/d' "${sysctl_conf}" 2>/dev/null || true + sed -i '' '/^net.local.stream.sendspace/d' "${sysctl_conf}" 2>/dev/null || true + + echo 'net.local.stream.recvspace=65536' >> "${sysctl_conf}" + echo 'net.local.stream.sendspace=65536' >> "${sysctl_conf}" + +} setup_xinit() { - # Disable screen locking in KDE Plasma for live_user chroot "${release}" su "${live_user}" -c " mkdir -p /home/${live_user}/.config kwriteconfig5 --file /home/${live_user}/.config/kscreenlockerrc --group Daemon --key Autolock false kwriteconfig5 --file /home/${live_user}/.config/kscreenlockerrc --group Daemon --key LockOnResume false echo 'exec ck-launch-session startplasma-x11' >> /home/${live_user}/.xinitrc " - - # Set the same .xinitrc for root and skel echo "exec ck-launch-session startplasma-x11" > "${release}/root/.xinitrc" echo "exec ck-launch-session startplasma-x11" > "${release}/usr/share/skel/dot.xinitrc" } @@ -57,6 +99,9 @@ setup_xinit() { patch_etc_files community_setup_liveuser community_setup_autologin +update_rcconf_dm +localtime sddm_setup +plasma_settings setup_xinit final_setup From 57b0406b77fb3a71a7ed8db974e3f2100e73cb31 Mon Sep 17 00:00:00 2001 From: Sunil Sherkhane <15212620+SunilSherkhane@users.noreply.github.com> Date: Fri, 3 Oct 2025 01:03:43 +0530 Subject: [PATCH 3/5] Update plasma removed the packages. --- packages/plasma | 34 +++++++++++----------------------- 1 file changed, 11 insertions(+), 23 deletions(-) diff --git a/packages/plasma b/packages/plasma index ec9b619f..3ab52126 100644 --- a/packages/plasma +++ b/packages/plasma @@ -1,32 +1,20 @@ plasma6-plasma +plasma6-sddm-kcm sddm -plasma6-discover dolphin kate ark -kcalc -kate +elisa +bash +zsh +nano +devd-mount +xdg-desktop-portal +xdg-desktop-portal-gtk +plasma6-xdg-desktop-portal-kde +ark gwenview okular konsole -xdg-desktop-portal -plasma6-xdg-desktop-portal-kde -xdg-desktop-portal-gtk -wireplumber -pipewire -plasma6-kpipewire -plasma6-spectacle -plasma6-ksystemstats -knights +plasma6-discover cpu-microcode -kajongg -colord-kde -kdenetwork-filesharing -kdemultimedia-ffmpegthumbs -kcharselect -ark -kate -libX11 -rar -unrar -7-zip From 3fbf495e73f7045c5aa1e0a7a047fb6dba7f83fb Mon Sep 17 00:00:00 2001 From: Sunil Sherkhane <15212620+SunilSherkhane@users.noreply.github.com> Date: Fri, 3 Oct 2025 01:03:56 +0530 Subject: [PATCH 4/5] Update plasma --- packages/vital/plasma | 33 +++++++++++---------------------- 1 file changed, 11 insertions(+), 22 deletions(-) diff --git a/packages/vital/plasma b/packages/vital/plasma index 828703ca..3ab52126 100644 --- a/packages/vital/plasma +++ b/packages/vital/plasma @@ -1,31 +1,20 @@ plasma6-plasma +plasma6-sddm-kcm sddm dolphin kate ark -kcalc -kate +elisa +bash +zsh +nano +devd-mount +xdg-desktop-portal +xdg-desktop-portal-gtk +plasma6-xdg-desktop-portal-kde +ark gwenview okular konsole -xdg-desktop-portal -plasma6-xdg-desktop-portal-kde -xdg-desktop-portal-gtk -wireplumber -pipewire -plasma6-kpipewire -plasma6-spectacle -plasma6-ksystemstats -knights +plasma6-discover cpu-microcode -kajongg -colord-kde -kdenetwork-filesharing -kdemultimedia-ffmpegthumbs -kcharselect -ark -kate -libX11 -rar -unrar -7-zip From 6c4f8e884c4fe53cf251b700a76c6f9ea4e81757 Mon Sep 17 00:00:00 2001 From: Sunil Sherkhane <15212620+SunilSherkhane@users.noreply.github.com> Date: Fri, 10 Oct 2025 20:02:51 +0530 Subject: [PATCH 5/5] Update plasma.sh Added the string to setup the default time as per Bios in Live User. --- desktop_config/plasma.sh | 40 ++++++++++++++++------------------------ 1 file changed, 16 insertions(+), 24 deletions(-) diff --git a/desktop_config/plasma.sh b/desktop_config/plasma.sh index 6878732d..2a66c31a 100644 --- a/desktop_config/plasma.sh +++ b/desktop_config/plasma.sh @@ -24,28 +24,19 @@ update_rcconf_dm() { echo 'sddm_enable="YES"' >> "${rc_conf}" } -localtime() { - # Detect hardware clock timezone from BIOS (assumes BIOS is in UTC) - # FreeBSD convention: set timezone in /etc/localtime - # Copy UTC by default; adjust later if needed - tz_target="${release}/etc/localtime" - - # Remove existing symlink/file - rm -f "${tz_target}" - - # Link BIOS UTC to system UTC - ln -s /usr/share/zoneinfo/UTC "${tz_target}" - - # Ensure rc.conf knows hwclock type - rc_conf="${release}/etc/rc.conf" - sed -i '' '/^ntpd_enable=.*/d' "${rc_conf}" 2>/dev/null || true - sed -i '' '/^ntpd_sync_on_start=.*/d' "${rc_conf}" 2>/dev/null || true - sed -i '' '/^local_unbound_enable=.*/d' "${rc_conf}" 2>/dev/null || true - - echo 'ntpd_enable="YES"' >> "${rc_conf}" - echo 'ntpd_sync_on_start="YES"' >> "${rc_conf}" +set_localtime_from_bios() { + # Set timezone for live_user only (not global /etc/localtime) + chroot "${release}" su "${live_user}" -c " + # Ensure home exists + mkdir -p /home/${live_user} + # Append timezone environment variable to profile + if ! grep -q 'TZ=' /home/${live_user}/.profile 2>/dev/null; then + echo 'export TZ=UTC' >> /home/${live_user}/.profile + else + sed -i '' 's/^export TZ=.*/export TZ=UTC/' /home/${live_user}/.profile + fi + " } - sddm_setup() { sddm_conf="${release}/etc/sddm.conf" @@ -89,18 +80,19 @@ setup_xinit() { mkdir -p /home/${live_user}/.config kwriteconfig5 --file /home/${live_user}/.config/kscreenlockerrc --group Daemon --key Autolock false kwriteconfig5 --file /home/${live_user}/.config/kscreenlockerrc --group Daemon --key LockOnResume false - echo 'exec ck-launch-session startplasma-x11' >> /home/${live_user}/.xinitrc - " + grep -qxF 'exec ck-launch-session startplasma-x11' /home/${live_user}/.xinitrc || echo 'exec ck-launch-session startplasma-x11' >> /home/${live_user}/.xinitrc " echo "exec ck-launch-session startplasma-x11" > "${release}/root/.xinitrc" echo "exec ck-launch-session startplasma-x11" > "${release}/usr/share/skel/dot.xinitrc" } + + # Execute setup routines patch_etc_files community_setup_liveuser community_setup_autologin update_rcconf_dm -localtime +set_localtime_from_bios sddm_setup plasma_settings setup_xinit