[Option 1] Fix race between handle_replica_event and start_pid_remotely

[d-b]

There appears to be a race condition leading the Swarm.Tracker process to die. When this happens the monitors that were acquired by Swarm.Tracker are lost and the process :down events will fail to be captured.

The race occurs when handle_replica_event attempts to find the remote process in the local registry. If it is not found, but is then registered in the mean time by start_pid_remotely an error will occur when handle_replica_event tries to register the process itself

swarm/lib/swarm/tracker/tracker.ex

Lines 1004 to 1008 in 7dd08d6

	:undefined ->
	ref = Process.monitor(pid)
	lclock = Clock.join(clock, rclock)
	Registry.new!(entry(name: name, pid: pid, ref: ref, meta: meta, clock: lclock))
	:keep_state_and_data

This solution attempts to remedy the problem by serializing the call to add_registration by start_pid_remotely within the Tracker GenStateMachine process. This will prevent the registration from executing concurrently with handle_replica_event

[tristan-secord]

https://github.com/scorebet/sportsbook-api/pull/2261

Lets let it soak on staging for a bit and I'll throw it up on UAT.
If no issues come up we can toss it onto prod.

Also not sure how far along you've gotten with the stress testing stuff, could hand it off to someone on the infrastructure team if you need to focus on Marketing Push