Missing `nthreads` arguments and its inconsistent defaults

[anand-avinash]

At some places in the code, the argument for nthreads in relevant ducc functions are still missing. While it is quickly fixable, I noticed that its default values are set inconsistently. At some places the default is set with nthreads: int | None = None usually followed by a call nthreads = resolve_nthreads(nthreads), while at others the default is set with nthreads: int = 0. Also in resolve_threads(), when nthreads is None, it returns 0. With 0 default, ducc would use all available hardware threads. With multiple MPI processes, this may lead to thread contention and may affect the performance.

As for consistency in setting the defaults,I was curious what could be the correct way to resolve this. One would be to use consistent defaults in all cases, followed by the resolve_nthreads() call. Another solution would be to make nthreads a global variable in litebird_sim (much like MPI_COMM_WORLD) and use it consistently without having to call resolve_nthreads() every time nthreads is needed.

Please let me know what you think.

[mj-gomes]

Hi @anand-avinash, in my opinion, the global variable solution is the best. Since both nthreads and MPI_COMM_WORLD belong to the parallel computing "domain", dealing with both of them in same way seems the logical and well organized way to go. For now, I can't see any caveat with this option.

And I agree that we should be careful with thread contention, probably we should stop giving nthreads=0 to ducc. If it comes in handy for specific cases, users can specify it, but not as the default value.

[anand-avinash]

Also, if we use the global variable solution, should we then strip off all the nthreads argument from the public methods/functions? Or still allow the user to pass whatever nthreads argument they want to a given method/function?

[mj-gomes]

Also, if we use the global variable solution, should we then strip off all the nthreads argument from the public methods/functions? Or still allow the user to pass whatever nthreads argument they want to a given method/function?

I think if we are defining a global default, it should be obtained from the OMP_NUM_THREADS variable, as we do in some places already

I am not sure anyone will have a case in which they want to use a number smaller than the available OMP threads, right? In general we always want to use as much as we can, specially for the numba functions, and we don't risk having memory problems for these use cases, I believe, but I am not 100% sure. Maybe there are use cases in which we might want to specify the threads, otherwise I would say let's just strip the nthreads argument off everywhere.

[paganol]

Hi there, I went through the code, and it's even worst.. there are three different cases:

• nthreads: int | None = None + resolve_nthreads(nthreads) — e.g. add_convolved_sky (beam_convolution.py), Observation methods, scan_map.py, grasp2alm.py, the destriper/binner/pair-differencing map-makers.
• nthreads: int = 0 (literal "use everything") — the map/alm transform APIs in maps_and_harmonics.py (4 sites), input_sky.py, and a few internal helpers.
• nthreads: int = 1 — MuellerConvolver, a third, more conservative default that nobody else uses.

Also, few other cases we need to consider:

• grasp2alm.py's PseudoAnalysisSHT has no nthreads parameter at all — it hardcodes nthreads=0 in two direct ducc0 calls (lines 108, 126).
• hwp_harmonics.py (the OMP_NUM_THREADS-reading code referenced above) doesn't actually call resolve_nthreads() — it reimplements the same three lines inline. That's drifted duplicate logic, not a template to copy elsewhere.
• Separately, Simulation.numba_threads is already a third thread-count mechanism (per-Simulation value read from NUMBA_NUM_THREADS_ENVVAR).

So, we should:

1. Standardize every site on nthreads: int | None = None + resolve_nthreads().
2. Fix the contention risk in resolve_nthreads() itself, maybe changing its fallback from 0 (all available) to something like per-rank share.. or simply 1.
3. Add the missing nthreads parameter to grasp2alm.py's PseudoAnalysisSHT.
4. Address the overlap with numba, here I have a less clear view..

Happy to send a PR along these lines if this shape sounds reasonable.

[anand-avinash]

Hi @paganol! I agree with your observations. I have two comments:

1. Currently, every time resolve_nthreads() is called, it makes a call to fetch the envvar OMP_NUM_THREADS. It seems redundant since the envvar itself won't change ever during the runtime. So it would be better if we call resolve_nthreads() only once - while importing litebird_sim and store the value as a global variable (just like lbs.MPI_COMM_WORLD). Then we can use this global variable whenever nthreads is set to None. Here, resolve_nthreads() should take care of the thread contention.
2. Thread contention exist for numba as well if OMP_NUM_THREADS is not set and num_threads=None while creating a simulation object. It would be better if we could resolve this in a way similar to resolve_nthreads(), except that the value provided by OMP_NUM_THREADS should be ignored if NUMBA_NUM_THREADS is set in the envvar or nthreads is provided while creating a simulation object. While ducc relies on OMP_NUM_THREADS, numba uses NUMBA_NUM_THREADS. This will allow us to use different number of threads for numba and ducc.

Also, both OMP_NUM_THREADS and NUMBA_NUM_THREADS translate to the number of threads per MPI process.

If you can create a PR quickly, that would be very helpful!