Compiling STRUMPACK packages takes forever when updating petsc on Expanse cluster

[chunhuizhao478]

Check these boxes if you have followed the posting rules.

• I have consulted the Posting Guidelines.
• I have searched the Discussions Forum and MOOSE Framework Troubleshooting and have not found what I was looking for
• Q&A Getting Started is the most appropriate category for my question (trouble installing, beginner user, ...)

Issue or question about MOOSE

Hi all, I try to update MOOSE on Expanse to latest version using the approach discussed in #26162. However, when it comes to "STRUMPACK" package, it gets stuck and unable to move forward:

=============================================================================================
      Trying to download https://github.com/Reference-ScaLAPACK/scalapack for SCALAPACK
=============================================================================================
=============================================================================================
               Configuring SCALAPACK with CMake; this may take several minutes
=============================================================================================
=============================================================================================
              Compiling and installing SCALAPACK; this may take several minutes
=============================================================================================
=============================================================================================
            Trying to download https://github.com/pghysels/STRUMPACK for STRUMPACK
=============================================================================================
=============================================================================================
               Configuring STRUMPACK with CMake; this may take several minutes
=============================================================================================
=============================================================================================
              Compiling and installing STRUMPACK; this may take several minutes
=============================================================================================

I wonder if the approach in #26162 by @milljm needs to get updated, any suggestion? Thanks!

(Optional) code in question / simulation log / errors

No response

Encountering Errors? Please include diagnostic output

No response

[GiudGiud]

Do you need strumpack? You could just disable it

[GiudGiud]

in scripts/configure_petsc.h, set --download-strumpack=0\ at line 116

[chunhuizhao478]

Hi @GiudGiud, I tried this and it passed petsc build, but then the libmesh build also stucks at some point:

  CXX      src/mesh/libmesh_opt_la-mesh_output.lo
  CXX      src/mesh/libmesh_opt_la-mesh_refinement.lo
  CXX      src/mesh/libmesh_opt_la-mesh_refinement_flagging.lo
  CXX      src/mesh/libmesh_opt_la-mesh_refinement_smoothing.lo
  CXX      src/mesh/libmesh_opt_la-mesh_serializer.lo
  CXX      src/mesh/libmesh_opt_la-mesh_smoother.lo
  CXX      src/mesh/libmesh_opt_la-mesh_smoother_laplace.lo
  CXX      src/mesh/libmesh_opt_la-mesh_smoother_vsmoother.lo
  CXX      src/mesh/libmesh_opt_la-mesh_subdivision_support.lo
  CXX      src/mesh/libmesh_opt_la-mesh_tet_interface.lo

I have used export MOOSE_JOBS=6 METHODS=opt, I wonder if there are other options I should set, thanks!

[GiudGiud]

should not happen
I think maybe you are running out of memory and it s stalling? try less moose_jobs

[chunhuizhao478]

Yes, I use moose_jobs = 1 and pass the libmesh build, then when build moose test, I use single core, but still get killed by out of memory after some time:

[czhao1@login02 test]$ make 
Compiling C++ (in opt mode) /expanse/projects/qstore/usc143/qwxdev/apps/expanse/rocky8.8/moose_versions/moosed253789/framework/build/unity_src/userobjects_Unity.C...
g++: fatal error: Killed signal terminated program cc1plus
compilation terminated.
make: *** [/expanse/projects/qstore/usc143/qwxdev/apps/expanse/rocky8.8/moose_versions/moosed253789/framework/build.mk:151: /expanse/projects/qstore/usc143/qwxdev/apps/expanse/rocky8.8/moose_versions/moosed253789/framework/build/unity_src/userobjects_Unity.x86_64-pc-linux-gnu.opt.lo] Error 1

[GiudGiud]

Wow you’re really low on memory then

the lowest memory you can use is this

MOOSE_UNITY=false make