lsvec Shared Memory Race Condition (#674)

This patches another shared memory bug in `lsvec`, this time a race
condition. The bug is easiest explained by pointing at the relevant
lines. Below is the implementation of `lsvec` for
`TimeIndependentMDCObjectiveFunction`:

```
  lsvec = self.terms(paramvec, oob_check, "LS OBJECTIVE")
  if _np.any(lsvec < 0):
      bad_locs = _np.where(lsvec < 0)[0]
      msg = f"""
      lsvec is only defined when terms is elementwise nonnegative.
      We encountered negative values for terms[i] for indices i
      in {bad_locs}.
      """
      raise RuntimeError(msg)
  unit_ralloc = self.layout.resource_alloc('atom-processing')
  shared_mem_leader = unit_ralloc.is_host_leader
  if shared_mem_leader:
      lsvec **= 0.5
  if raw_objfn_lsvec_signs:
      if shared_mem_leader:
          raw_lsvec = self.raw_objfn.lsvec(self.probs, self.counts, self.total_counts, self.freqs)
          lsvec[:self.nelements][raw_lsvec < 0] *= -1
  unit_ralloc.host_comm_barrier()
```

The bug arises due to the interation of the positivity check ` if
_np.any(lsvec < 0)` and later on the update of `lsvec` using the
`raw_objfn` signs `lsvec[:self.nelements][raw_lsvec < 0] *= -1`. Since
the positivity check is happening on all ranks, but is with respect to
shared memory, if a rank that is the shared memory leader hits and runs
that sign update line before another rank looking at the same memory
hits the positivity check this would raise an error.

After staring at this code for a bit I couldn't see any reason why this
positivity check (or anything else in this function) should be happening
on anything other the shared memory leader, so the patch is to just add
a corresponding if statement which enforces that.

Thanks to @juangmendoza19 for reporting this error.

God I hate shared memory sometimes...

Co-authored-by: Corey Ostrove <[email protected]>

Author: coreyostrove

pygsti/objectivefns/objectivefns.py

@@ -4406,21 +4406,22 @@ def terms(self, paramvec=None, oob_check=False, profiler_str="TERMS OBJECTIVE"):
         return terms
 
     def lsvec(self, paramvec=None, oob_check=False, raw_objfn_lsvec_signs=True):
-        lsvec = self.terms(paramvec, oob_check, "LS OBJECTIVE")
-        if _np.any(lsvec < 0):
-            bad_locs = _np.where(lsvec < 0)[0]
-            msg = f"""
-            lsvec is only defined when terms is elementwise nonnegative.
-            We encountered negative values for terms[i] for indices i
-            in {bad_locs}.
-            """
-            raise RuntimeError(msg)
         unit_ralloc = self.layout.resource_alloc('atom-processing')
         shared_mem_leader = unit_ralloc.is_host_leader
+        lsvec = self.terms(paramvec, oob_check, "LS OBJECTIVE")
+
         if shared_mem_leader:
+            if _np.any(lsvec < 0):            
+                bad_locs = _np.where(lsvec < 0)[0]
+                msg = f"""
+                lsvec is only defined when terms is elementwise nonnegative.
+                We encountered negative values for terms[i] for indices i
+                in {bad_locs}.
+                """
+                raise RuntimeError(msg)
+
             lsvec **= 0.5
-        if raw_objfn_lsvec_signs:
-            if shared_mem_leader:
+            if raw_objfn_lsvec_signs:
                 raw_lsvec = self.raw_objfn.lsvec(self.probs, self.counts, self.total_counts, self.freqs)
                 lsvec[:self.nelements][raw_lsvec < 0] *= -1
         unit_ralloc.host_comm_barrier()