Turbsim uses up all available threads when not compiled with OpenMP

[mayankchetan]

Bug description

When running turbsim on a system that has OpenMP ( or OpenMP loaded in case of HPC), but is not compiled with -DOPENMP=ON, turbsim runs on all available threads. This can bog down the system, or in some cases crashes HPC nodes.

Current workaround is to use export OMP_NUM_THREADS=1, but this can affect other processes that rely on OpenMP like OLAF.

To Reproduce

Steps to reproduce the behavior:

1. Compile with -DOPENMP=OFF
2. have or load OpenMP on system.
3. Run turbsim with r-test/glue-codes/openfast/5MW_Baseline/Wind/90m_12mps_twr.inp
4. Use top/htop to view resources used by turbsim.
5. Also see ####### CPU seconds used. after turbsim completes running.

Expected behavior

turbsim does not use all threads when compiled with -DOPENMP=OFF & system has/loaded OpenMP.

Screenshots, if applicable

1. When OpenMP is not loaded on an HPC node:

 **************************************************************************************************
 TurbSim

 Copyright (C) 2025 National Renewable Energy Laboratory
 Copyright (C) 2025 Envision Energy USA LTD

 This program is licensed under Apache License Version 2.0 and comes with ABSOLUTELY NO WARRANTY.
 See the "LICENSE" file distributed with this software for details.
 **************************************************************************************************

  Running TurbSim (from OpenFAST-v4.1.2)


  Reading the input file "90m_12mps_twr.inp".
  Calculating the spectral and transfer function matrices:
     u-component matrices (IEC coherence method)
     v-component matrices (identity coherence)
     w-component matrices (identity coherence)
  Generating time series for all points:
     u-component
     v-component
     w-component
  Computing hub-height statistics
  Writing statistics to summary file
  Generating AeroDyn binary time-series file "90m_12mps_twr.bts"

  Processing complete.  38.71 CPU seconds used.

  TurbSim terminated normally.

2. When OpenMP is loaded on an HPC node:

 **************************************************************************************************
 TurbSim

 Copyright (C) 2025 National Renewable Energy Laboratory
 Copyright (C) 2025 Envision Energy USA LTD

 This program is licensed under Apache License Version 2.0 and comes with ABSOLUTELY NO WARRANTY.
 See the "LICENSE" file distributed with this software for details.
 **************************************************************************************************

  Running TurbSim (from OpenFAST-v4.1.2)


  Reading the input file "90m_12mps_twr.inp".
  Calculating the spectral and transfer function matrices:
     u-component matrices (IEC coherence method)
     v-component matrices (identity coherence)
     w-component matrices (identity coherence)
  Generating time series for all points:
     u-component
     v-component
     w-component
  Computing hub-height statistics
  Writing statistics to summary file
  Generating AeroDyn binary time-series file "90m_12mps_twr.bts"

  Processing complete.  4963.4 CPU seconds used.

  TurbSim terminated normally.

3. When OpenMP is loaded on an HPC node & we set OMP_NUM_THREADS using export OMP_NUM_THREADS=1:

 **************************************************************************************************
 TurbSim

 Copyright (C) 2025 National Renewable Energy Laboratory
 Copyright (C) 2025 Envision Energy USA LTD

 This program is licensed under Apache License Version 2.0 and comes with ABSOLUTELY NO WARRANTY.
 See the "LICENSE" file distributed with this software for details.
 **************************************************************************************************

  Running TurbSim (from OpenFAST-v4.1.2)


  Reading the input file "90m_12mps_twr.inp".
  Calculating the spectral and transfer function matrices:
     u-component matrices (IEC coherence method)
     v-component matrices (identity coherence)
     w-component matrices (identity coherence)
  Generating time series for all points:
     u-component
     v-component
     w-component
  Computing hub-height statistics
  Writing statistics to summary file
  Generating AeroDyn binary time-series file "90m_12mps_twr.bts"

  Processing complete.  202.22 CPU seconds used.

  TurbSim terminated normally.

OpenFAST Version

 **************************************************************************************************
 OpenFAST

 Copyright (C) 2025 National Renewable Energy Laboratory
 Copyright (C) 2025 Envision Energy USA LTD

 This program is licensed under Apache License Version 2.0 and comes with ABSOLUTELY NO WARRANTY.
 See the "LICENSE" file distributed with this software for details.
 **************************************************************************************************

 OpenFAST-v4.1.2
 Compile Info:
  - Compiler: GCC version 14.3.0
  - Architecture: 64 bit
  - Precision: single
  - OpenMP: No
  - Date: Sep 24 2025
  - Time: 02:13:53
 Execution Info:
  - Date: 10/01/2025
  - Time: 11:26:53-0600

System Information (please complete the following information):

• OS: Red Hat Enterprise Linux 8.8 (Ootpa) (NREL, Kestrel)
• Compiler: GCC version 14.3.0
• Compiler settings: -DOPENMP=OFF

Additional context

[andrew-platt]

TurbSim does not include any !$OMP directives in the code, so it is very likely that the library getting linked in is attempting to use OpenMP on millions of very small calculations. The overhead associated with that is likely killing all performance.

At a bare minimum, I suspect we need one !$OMP directive at a top level to prevent it from doing this, and likely get a bit of performance gain in the process.

[andrew-platt]

Documenting this here just so it is findable:
Not all compilers will handle derived types inside an !$OMP PARALLEL DO PRIVATE(<derived_type>) statement correctly. If we have a derived type with an allocatable in it, and allocate that before the loop, that may not actually get copied into each OMP thread. Some of those copies may stay uninitialized which will turn into a mess when used.

I ran into this trying to put !$OMP PARALLEL directives around an ApplyFFT call.

[andrew-platt]

Workaround: export MKL_NUM_THREADS=1 to keep the MKL library from trying to parallelize.

Note: #3020 will allow OpenMP with TurbSim, and will automatically turn down the MKL threading. But if TS is built without OpenMP, then setting the number of MKL threads will be necessary.

[bjonkman]

Just to be clear, is this just an issue when TurbSim is built with the Intel MKL libraries? Could you set the number of threads inside the code so that it applies only to TurbSim (and not OLAF)? See https://www.intel.com/content/www/us/en/docs/onemkl/developer-guide-linux/2025-1/onemkl-specific-env-vars-for-openmp-thread-ctrl.html

[andrew-platt]

@bjonkman, that's what I did in the Cot2H routine in PR #3020. But that requires omp_lib, so we must build with OpenMP enabled for that to work.

So we could potentially still run into this issue if we build without enabling OpenMP in OpenFAST, but ended up linking to an MKL with OpenMP.

[bjonkman]

Hmmm... I see Coh2H calls omp_set_max_active_levels, but I thought the documentation showed you could call mkl_set_num_threads without building with OpenMP.

[andrew-platt]

Oops. I missed that subtlety on the name. They probably will both achieve the same thing.

I think mkl_set_num_threads still requires using omp_lib though (not shown in the Fortran code, but the header file is included in the C code above).

edit: mkl_set_num_threads() is vendor specific for Intel. There is no equivalent in the Apple Accelerate library (that may be single threaded only), but there is an openblas_set_num_threads() routine for the openblas library (I'm not sure on others). So perhaps we put in support for the Intel MKL since that is where we saw the issue, then tackle the others later as we run into them.

I had one other thought on this: CMake knows if we are using an MKL with OpenMP, so we could introduce an _OMPmkl identifier which would indicate OpenMP is available on the system. Then selectively load omp_lib in that routine.

[bjonkman]

I was actually noticing a similar problem with a code built on linux yesterday. I've now linked it with the sequential MKL library instead of the default Intel OpenMP MKL library, and that has fixed the issue (without the need to set the number of threads anywhere)