Running the examples

Go to one of the subdirectories of the Example folders, e.g. Examples/ScalarFieldBH and type:

make all -j 4

Assuming this compiles with no errors, an executable will have been produced with a name that depends on your compiler options but will look something like:

Main_ScalarFieldBH3d_ch.Linux.64.mpiicpc.ifort.OPTHIGH.MPI.OPENMPCC.COSMA8.Intel2022.ex

Normally you will not want to output the resulting hdf5 files to your home directory (where you would normally store the code), so you should amend the following lines in the params.txt file to give the (full, not relative) path to your data directory, where large output files are stored, e.g.

output_path = /path/to/your/directory/Outputs/
pout_path = /path/to/your/directory/pout/

The solver will output the final initial conditions as a hdf5 checkpoint file. You can rename it if you want by changing the following line in params.txt

output_filename = InitialDataFinal.3d.hdf5

The executable can be run in parallel (e.g. on 4 processors) using a command like

mpirun -np 4 ./Main_ScalarFieldBH3d_ch.Linux.64.mpiicpc.ifort.OPTHIGH.MPI.OPENMPCC.COSMA8.Intel2022.ex params.txt

However, usually on a cluster you are required to submit a batch job, so that the simulation is queued and run on the compute nodes. So you would write a jobscript and submit that to the queue. There should be examples for MPI and hybrid OpenMP/MPI jobs in your cluster documentation. Note that you may also need to reload the module files within the jobscript, depending on how your cluster works. Some examples for systems we use can be found here. The number of cores you need will depend on the job and the memory per core on your cluster. If you run out of memory, you may want to switch to using several OpenMP threads per task, rather than running a pure MPI job. You submit the job with a command which depends on the scheduler, again for example for the slurm scheduler it is:

sbatch jobscript.txt

This should produce a lot of hdf5 files with names like NL_iteration_*.hdf5. Each file corresponds to each solver's step and allows you to see the quantities used by the solver, and the ADM variables (see Visualising outputs). These can be useful where you have problems with convergence but in general you may wish to turn them off as they create a lot of data output. If the solver runs successfully, it will produce a final file called InitialDataFinal.3d.hdf5. This contains all the outputs using the variable conventions and formatting of the checkpoint files of the NR evolution code GRChombo. These hdf5 files can be viewed to check what has been simulated (see Visualising outputs). The solver also outputs Ham_and_Mom_errors.txt which contains a measure of the constraints error for each step. This should decrease over the course of the non-linear iteration steps, but note that it is a necessary but not sufficient condition for you to trust your output. See Convergence testing for more details on how to validate your results, which is an essential step.

Your job will also output files called pout.* for each processor. Looking at these files during the run allows you to see what is going on while the solver is running. More output can be produced by setting

verbose = 1

in params.txt. These files will also tell you how many boxes are being processed by each processor - for good load balancing and to avoid using more processors than you need, ideally all the processors should have one (or several) boxes each, and there should be at least one per processor.

Note that Chombo does not adjust the box sizes automatically based on the number of processes you choose so you almost certainly need to adjust them for your specific application and job configuration to avoid wasting resources and/or achieve good speeds.

More information on load balancing is given in the section on performance optimisation in the main GRChombo wiki. This is highly recommended reading.