Different results with and without OpenMP threads

[iomaganaris]

Tests from testcorenrn repository that simulate more than 1 cell produce different outputs from Neuron when run with CoreNeuron with SoA configuration and more than 1 OpenMP thread.
CoreNeuron was built using the following CMake options:

cmake \
   -G 'Unix Makefiles' \
   -DCMAKE_INSTALL_PREFIX=./install \
   -DCMAKE_BUILD_TYPE=Debug \
   -DCMAKE_C_FLAGS:STRING="-O0" \
   -DCMAKE_CXX_FLAGS:STRING="-O0" \
   -DENABLE_SOA=ON \
   -DCORENEURON_OPENMP=ON \
   -DTEST_MPI_EXEC_BIN="mpirun" \
   -DTEST_EXEC_PREFIX="mpirun;-n;2" \
   -DAUTO_TEST_WITH_SLURM=OFF \
   -DAUTO_TEST_WITH_MPIEXEC=OFF \
   .. /

To run a test (for example "watch" which produces different output):

git clone git@github.com:pramodk/testcorenrn.git
cd testcorenrn
nrnivmodl mod
nrnivmodl-core mod
mkdir testwatchdat
mpirun -n 6 ./x86_64/special -mpi -c sim_time=100 testwatch.hoc
cat out.dat | sort -k 1n,1n -k 2n,2n > out_nrn_watch.spk
export OMP_NUM_THREADS=1
mpirun -n 6 ./x86_64/special-core -mpi -d test${NAME}dat -e 100
diff -w -q out.dat out_nrn_watch.spk // Same results
export OMP_NUM_THREADS=2
mpirun -n 3 ./x86_64/special-core -mpi -d test${NAME}dat -e 100
diff -w -q out.dat out_nrn_watch.spk // Different results

[pramodk]

We have seen that nrn_setup phase2 has race conditions! look at global hash tables which are not protected by OMP critical section.

[olupton]

@iomaganaris do you know if this is still an issue? I ran a quick test locally where I ran all testcorenrn tests with multiple threads as well as 1, and nothing failed.

I wonder if this was fixed with #283?

[iomaganaris]

I am not sure whether this fixed the issue. I think it would be useful to run the test added in this PR #299 and see whether there is still an issue

[olupton]

OK, this, #292 and #299 all look related. I guess #299 should be updated to run in CTest/GitLab.