Building OpenFOAM on different computing systems

[colemanjs]

This discussion will serve as a centralized place for addressing OpenFOAM-specific build issues, keeping them separate from the main AdditiveFOAM branch, to better support users facing similar challenges.

[colemanjs]

Installation Notes for OpenFOAM and AdditiveFoam on HPC systems

These notes provide a guide for compiling OpenFOAM from the source code, with specific instructions for configuring it with a system-installed MPI library and preparing it for use on a High-Performance Computing (HPC) system.

First, the source code for OpenFOAM and its third-party packages need to be cloned into an OpenFOAM directory on the user system from their respective repositories. The user home directory is used here as an example:

# make the OpenFOAM directory
mkdir -p /home/OpenFOAM
cd /home/OpenFOAM

# download source code
git clone https://github.com/OpenFOAM/OpenFOAM-10.git
git clone https://github.com/OpenFOAM/ThirdParty-10.git

1. OpenFOAM Third-Party Dependencies

When compiling OpenFOAM, it's important to manage its third-party package (TPL) dependencies. The necessary source files and build scripts are located in the ThirdParty-10 repository, which can be found at: https://github.com/OpenFOAM/ThirdParty-10.

While the OpenFOAM documentation often mentions ParaView for visualization, it is not a mandatory dependency for compiling and running the software, especially on remote compute-only resources. Other essential packages like Scotch/PT-Scotch for domain decomposition are typically compiled automatically from the ThirdParty directory. [source: @rcarson3]

2. Configuring OpenFOAM with a System-Installed MPI

To ensure OpenFOAM uses a system-installed MPI library such as MPICH, you will need to modify the shell environment. While you can directly edit the OpenFOAM/etc/bashrc file, a cleaner and recommended approach is to create a prefs.sh file in your $WM_PROJECT_DIR/etc/ directory to override the default settings.

MPICH Example

Add the following configuration to your prefs.sh file to link against a system MPI library and use the GCC compiler:

# Set mpi root directory
export MPI_ROOT=$MPICH_DIR

# To prevent conflicts with the C++ compiler bindings,
# it is recommended to set the following flag for MPICH-based MPIs.
export MPI_ARCH_FLAGS="-DMPICH_SKIP_MPICXX"

# Set the include and library paths to your system's MPI installation.
# The $MPI_ROOT environment variable should point to the base directory of your MPI installation.
export MPI_ARCH_INC="-I$MPI_ROOT/include"
export MPI_ARCH_LIBS="-L$MPI_ROOT/lib -lmpich -lrt"

# Instruct OpenFOAM to use the system-installed MPI library.
export WM_MPLIB=SYSTEMMPI

Note: The paths and library names (-lmpich) may need to be adjusted based on your specific MPI installation. The WM_MPLIB variable can be set to other values depending on the MPI implementation, such as SYSTEMOPENMPI for OpenMPI.

3. Managing Floating-Point Exceptions

OpenFOAM includes a feature to trap floating-point exceptions (FPE), which can be useful for debugging but may cause simulations to terminate unexpectedly. This is controlled by the FOAM_SIGFPE environment variable. To disable this feature, you can add the following line to your prefs.sh file:

export FOAM_SIGFPE=unset

4. Platform-Specific Compiler Configuration (OLCF Frontier)

On certain HPC systems with specialized programming environments, such as OLCF Frontier which uses the Cray Programming Environment, you may need to modify the compiler flags and specify the compiler toolchain.

First, set the WM_COMPILER environment variable in your prefs.sh file to Clang to match Frontier's compiler suite.

# Use the Clang compiler toolchain
export WM_COMPILER=Clang

Setting this variable ensures that OpenFOAM's build system (wmake) will use the rules located in the wmake/rules/linux64Clang/ directory.

Next, you should edit the following files within that directory to add flags required by the Cray compiler wrappers:

1. In wmake/rules/linux64Clang/c, modify the cc definition to:

   cc          = cc -craype-verbose -m64
   

   and the LINKEXE definition:

   LINKEXE     = $(cc) -fuse-ld=ld -Wl,-add-needed
   

2. In wmake/rules/linux64Clang/c++, modify the CC definition to:

   CC          = CC -craype-verbose -std=c++14 -m64
   

   and the LINKEXE definition:

   LINKEXE     = $(CC) $(c++FLAGS) -fuse-ld=ld -Wl,-add-needed
   

These changes ensure that the Cray compiler wrappers are invoked correctly for the Frontier architecture.

5. Compiling OpenFOAM in Parallel

The standard ./Allwmake script builds OpenFOAM sequentially. To significantly reduce compilation time, you can perform a parallel build using the -j flag, which specifies the number of processor cores to use. The -q flag can also be used for a quicker build by compiling multiple libraries and executables at once.

For a parallel build, use the following command:

./Allwmake -j -q

6. Configuring AdditiveFoam for HPC Environments

When running the examples and tutorials for AdditiveFoam on an HPC system, you may need to adapt the run scripts to the specific job scheduler in use (e.g., SLURM, LSF). The tutorial scripts often utilize RunFunctions located in OpenFOAM/bin/tools/, which may default to using mpirun for parallel execution.

On many HPC systems, jobs must be submitted through a scheduler command like srun (for SLURM). If your system requires this, you will need to modify the RunFunctions file to replace instances of mpirun with the appropriate command for your system's job queue. This ensures that parallel jobs are correctly submitted to the HPC scheduler.