intel-mkl external packages have incorrect paths

[aidanheerdegen]

When attempting to install petsc:

$ time spack install petsc %intel@2021.6.0 ^openmpi@4.1.4

got errors like this:

==> Warning: cannot perform the requested write operation on module files [Trying to source non-existing file: /apps/intel-mkl/2022.2.0/compilers_and_libraries/linux/mkl/bin/mklvars.sh]                                                                                                                                                                      

There is no directory /apps/intel-mkl. The intel-mkl libraries are installed under /apps/intel-ct/:

$ module show intel-mkl/2019.3.199
-------------------------------------------------------------------
/apps/Modules/modulefiles/intel-mkl/2019.3.199:

setenv          MKLROOT /apps/intel-ct/2019.3.199/mkl
prepend-path    LIBRARY_PATH /apps/intel-ct/2019.3.199/mkl/lib/intel64
prepend-path    LD_LIBRARY_PATH /apps/intel-ct/2019.3.199/mkl/lib/intel64
prepend-path    LD_RUN_PATH /apps/intel-ct/2019.3.199/mkl/lib/intel64
prepend-path    C_INCLUDE_PATH /apps/intel-ct/2019.3.199/mkl/include
prepend-path    CPLUS_INCLUDE_PATH /apps/intel-ct/2019.3.199/mkl/include
prepend-path    CPATH /apps/intel-ct/2019.3.199/mkl/include
prepend-path    FPATH /apps/intel-ct/2019.3.199/mkl/include
prepend-path    MANPATH /apps/intel-ct/2019.3.199/mkl/man
prepend-path    MANPATH /apps/intel-ct/2019.3.199/mkl/man/en_US
prepend-path    NLSPATH /apps/intel-ct/2019.3.199/mkl/lib/intel64/locale/en_US/%N
prepend-path    NLSPATH /apps/intel-ct/2019.3.199/mkl/lib/intel64/locale/%l_%t/%N
setenv          MKL /apps/intel-ct/2019.3.199/mkl
module-whatis   {Intel Math Kernel Library (MKL), version 2019.3.199}
conflict        intel-mkl
setenv          INTEL_MKL_BASE /apps/intel-ct/2019.3.199/mkl
setenv          INTEL_MKL_ROOT /apps/intel-ct/2019.3.199/mkl
setenv          INTEL_MKL_VERSION 2019.3.199
-------------------------------------------------------------------

The external packages are assumed to be in /apps/${PKGNAME}, so would need some method to customise that path for each package.

[aidanheerdegen]

Still not working. Damn

$ time spack install petsc %intel@2021.2.0 ^openmpi@4.1.4   
[+] /usr (external diffutils-3.6-ykfeb7a4v7aftydz47p6u47ao2wj3t4p)                       
==> cmake@3.24.2 : has external module in ['cmake/3.24.2']                                                                                                                     
[+] /apps/cmake/3.24.2 (external cmake-3.24.2-2i623mjk2dzl3v3jf52ulp4ouaqkuitk)                                                                                                
==> openmpi@4.1.4 : has external module in ['openmpi/4.1.4']                                                                                                                   
[+] /apps/openmpi/4.1.4 (external openmpi-4.1.4-5tae55ignezjh232xehlkn564byuixmg)                                                                                              
[+] /usr (external pkgconf-1.4.2-ue2bk6wxd2is37lrik25xtvanex6dbka)                                                                                                             
[+] /scratch/tm70/aph502/opt/spack/linux-rocky8-cascadelake/intel-2021.2.0/zlib-1.2.13-puh5y2zn6xxgjmdenu2de5rhmwnnf4ul                                                        
==> intel-mkl@2022.2.0 : has external module in ['intel-mkl/2022.2.0']                                                                                                         
[+] /apps/intel-ct/2022.2.0/mkl (external intel-mkl-2022.2.0-2qrmbz7alxusqf6n6vvtij2ve2m3frw2)                                                                                 
==> python@3.11.0 : has external module in ['python3/3.11.0']                                                                                                                  
[+] /apps/python3/3.11.0 (external python-3.11.0-y2xonsxpss5oheskqnrvxlgzqqur26wh)                                                                                             
[+] /scratch/tm70/aph502/opt/spack/linux-rocky8-cascadelake/intel-2021.2.0/metis-5.1.0-snvmkgkkmk453x7bdddzo55wai5tmq4i                                                        
[+] /scratch/tm70/aph502/opt/spack/linux-rocky8-cascadelake/intel-2021.2.0/hdf5-1.12.2-66toiecle7ce4rganbpz44vlw2hnii6v                                                        
==> Installing hypre-2.26.0-jb5mgjxsaohuqlynmmj3wolxikvaifeb                             
==> No binary for hypre-2.26.0-jb5mgjxsaohuqlynmmj3wolxikvaifeb found: installing from source                                                                                  
==> Error: InstallError: Cannot locate core MKL libraries:                                                                                                                     
        ['libmkl_intel_lp64', 'libmkl_sequential', 'libmkl_core']                                                                                                              
        in:                                                                                                                                                                    
        /apps/intel-ct/2022.2.0/mkl/compilers_and_libraries/linux/mkl/lib/intel64                                                                                              
                                                                                                                                                                               
/scratch/tm70/aph502/0.19.0/lib/spack/spack/build_systems/intel.py:810, in blas_libs:                                                                                          
        807        )                                                                                                                                                           
        808        debug_print(mkl_libs)                                                                                                                                       
        809                                                                                                                                                                    
  >>    810        if len(mkl_libs) < 3:                                                                                                                                       
        811            raise_lib_error(                                                                                                                                        
        812                "Cannot locate core MKL libraries:",                                                                                                                
        813                mkl_libnames,                                                                                                                                       
                                                                                                                                                                               
                                                                                                                                                                               
==> Warning: Skipping build of petsc-3.18.1-dhwf7zrsszv5cxh3j4j5jbwmib5xxe2e since hypre-2.26.0-jb5mgjxsaohuqlynmmj3wolxikvaifeb failed
[+] /scratch/tm70/aph502/opt/spack/linux-rocky8-cascadelake/intel-2021.2.0/parmetis-4.0.3-tkvxmqkzxnnv2xg73rgbwj2j5bs3w3em                                                     
==> Installing superlu-dist-8.1.1-uk7acqytowwhaqyas7nptavt74pvcfrt                                                                                                             
==> No binary for superlu-dist-8.1.1-uk7acqytowwhaqyas7nptavt74pvcfrt found: installing from source                                                                            
==> Error: InstallError: Cannot locate core MKL libraries:                                                                                                                     
        ['libmkl_intel_lp64', 'libmkl_sequential', 'libmkl_core']                                                                                                              
        in:                                                                                                                                                                    
        /apps/intel-ct/2022.2.0/mkl/compilers_and_libraries/linux/mkl/lib/intel64                                                                                              
                                                                                                                                                                               
/scratch/tm70/aph502/0.19.0/lib/spack/spack/build_systems/intel.py:810, in blas_libs:                                                                                          
        807        )                                                                    
        808        debug_print(mkl_libs)                                                
        809                                                                             
  >>    810        if len(mkl_libs) < 3:                                                
        811            raise_lib_error(                                                 
        812                "Cannot locate core MKL libraries:",                                                                                                                
        813                mkl_libnames,                                                                                                                                       
                                                                                                                                                                               
                                                                                                                                                                               
==> Error: petsc-3.18.1-dhwf7zrsszv5cxh3j4j5jbwmib5xxe2e: Package was not installed     
==> Error: Installation request failed.  Refer to reported errors for failing package(s).
                                                                                                                                                                               
real    4m13.134s                                                                                                                                                              
user    1m18.723s                                                                                                                                                              
sys     0m15.921s                                                                         

Not sure where it is still picking up /apps/intel-ct/2022.2.0/mkl/compilers_and_libraries/linux/mkl/lib/intel64 path.

[access-hive-bot]

This issue has been mentioned on ACCESS Hive Community Forum. There might be relevant details there:

https://forum.access-hive.org.au/t/shared-access-spack-configuration/227/13

[harshula]

[gadi-login-05 ~]$ spack install intel-mkl %intel@2021.6.0
==> intel-mkl@2022.2.0 : has external module in ['intel-mkl/2022.2.0']
==> Warning: cannot perform the requested write operation on module files [Trying to source non-existing file: /apps/intel/compilers_and_libraries_2022.2.0/linux/mkl/bin/mklvars.sh]
[+] /apps/intel/compilers_and_libraries_2022.2.0/linux/mkl (external intel-mkl-2022.2.0-pmc5vjcnsup74x6td6z6ygi5f4ruhofi)

$ ls -d /apps/intel/compilers_and_libraries*
/apps/intel/compilers_and_libraries_2019.3.199
/apps/intel/compilers_and_libraries_2019.4.243
/apps/intel/compilers_and_libraries_2019.5.281
/apps/intel/compilers_and_libraries_2020.0.166
/apps/intel/compilers_and_libraries_2020.1.217
/apps/intel/compilers_and_libraries_2020.2.254
/apps/intel/compilers_and_libraries_2020.4.304

$ grep -r compilers_and_libraries_2022.2.0 /g/data/access/spack/*
/g/data/access/spack/config/spack_config/packages.yaml:      prefix: /apps/intel/compilers_and_libraries_2022.2.0/linux/mkl

/g/data/access/spack/config/spack_config/packages.yaml:

  intel-mkl:
    externals:
...
    - spec: intel-mkl@2022.2.0
      prefix: /apps/intel/compilers_and_libraries_2022.2.0/linux/mkl
      modules:
      - intel-mkl/2022.2.0