RemoteDispatcher unable to deserialize SoftSignal carrying a StrictEnum

[burkeds]

Ran into this during our commissioning beamtime this week. I am still not fully certain if this is a bug or if did not handle the Enum correctly. To recreate the issue, start off by running a 0MQ proxy and RemoteDispatcher in their own processes.

bluesky-0MQ-proxy 5577 5578

from bluesky.callbacks.zmq import RemoteDispatcher
d = RemoteDispatcher('localhost:5578')
d.subscribe(print)
d.start()

Then create some simple devices, one of which has a Config Signal to handle your Enum.

from bluesky import RunEngine
from bluesky.callbacks.zmq import Publisher
from bluesky.plans import scan
from ophyd_async.core import DeviceCollector, AsyncStatus
from ophyd_async.core import StrictEnum, soft_signal_rw, StandardReadable, StandardReadableFormat as Format
from ophyd_async.sim.demo import SimMotor

class TestEnum(StrictEnum):
    CONTINUOUS = "continuous"
    STEP = "step"

class TestDevice(StandardReadable):
    def __init__(self,
                 name: str,
                 mode: str = "step"):
        mode = TestEnum(mode)
        with self.add_children_as_readables():
            self.motor = SimMotor(name="motor")
        with self.add_children_as_readables(Format.CONFIG_SIGNAL):
            self.mode = soft_signal_rw(TestEnum, name="mode", initial_value=mode)
        super().__init__(name=name)
    
    @AsyncStatus.wrap
    async def set(self, value):
        await self.motor.set(value)


class TestSensor(StandardReadable):
    def __init__(self, name):
        with self.add_children_as_readables(Format.HINTED_SIGNAL):
            self.counts = soft_signal_rw(int, name="counts", initial_value=150)
        super().__init__(name=name)


pub = Publisher("localhost:5577")

RE = RunEngine({})
RE.subscribe(pub)

with DeviceCollector() as collector:
    det = TestSensor("det")
    dev = TestDevice("dev")

RE(scan([det], dev, -1, 1, 1))

The RemoteDispatcher should be unable to print the descriptor:

Failed to deserialize the descriptor document b'\x80\x04\x95V\x03\x00\x00\x00\x00\x00\x00}\x94(\x8c\rconfiguration\x94}\x94(\x8c\x03det\x94}\x94(\x8c\x04data\x94}\x94\x8c\ntimestamps\x94}\x94\x8c\tdata_keys\x94}\x94u\x8c\x03dev\x94}\x94(h\x05}\x94(\x8c\x0fdev-motor-units\x94\x8c\x02mm\x94\x8c\x12dev-motor-velocity\x94G\x00\x00\x00\x00\x00\x00\x00\x00\x8c\x08dev-mode\x94\x8c\x08builtins\x94\x8c\x07getattr\x94\x93\x94\x8c\x08__main__\x94\x8c\x08TestEnum\x94\x93\x94\x8c\x04STEP\x94\x86\x94R\x94uh\x07}\x94(h\x0eGA$l\x9c\xcf\xe8\x90\xc5h\x10GA$l\x9c\xcf\xe6\xf6\x08h\x11GA$l\x9c\xd0\x06=\\uh\t}\x94(h\x0e}\x94(\x8c\x05dtype\x94\x8c\x06string\x94\x8c\x05shape\x94]\x94\x8c\x0bdtype_numpy\x94\x8c\x04|S40\x94\x8c\x06source\x94\x8c\x16soft://dev-motor-units\x94uh\x10}\x94(h\x1e\x8c\x06number\x94h ]\x94h"\x8c\x03<f8\x94h$\x8c\x19soft://dev-motor-velocity\x94uh\x11}\x94(h\x1eh\x1fh ]\x94h"\x8c\x04|S40\x94h$\x8c\x0fsoft://dev-mode\x94\x8c\x07choices\x94]\x94(\x8c\ncontinuous\x94\x8c\x04step\x94euuuuh\t}\x94(\x8c\ndet-counts\x94}\x94(h\x1e\x8c\x07integer\x94h ]\x94h"\x8c\x03<i8\x94h$\x8c\x11soft://det-counts\x94\x8c\x0bobject_name\x94h\x03u\x8c\x17dev-motor-user_readback\x94}\x94(h\x1eh\'h ]\x94h"\x8c\x03<f8\x94h$\x8c\x1esoft://dev-motor-user_readback\x94h:h\x0buu\x8c\x04name\x94\x8c\x07primary\x94\x8c\x0bobject_keys\x94}\x94(h\x03]\x94h4ah\x0b]\x94h;au\x8c\trun_start\x94\x8c$4627e133-da61-4cc4-bde5-4cfd9713c542\x94\x8c\x04time\x94GA\xd9\xd6\x02\xdf\x1a\xc0:\x8c\x03uid\x94\x8c$ff6201a5-95fc-414e-94f7-1502fa426ca0\x94\x8c\x05hints\x94}\x94(h\x03}\x94\x8c\x06fields\x94]\x94h4ash\x0b}\x94hN]\x94h;asuu.' using <built-in function loads>. Dropping on floor and continuing

I am only able to see this issue in the RemoteDispatcher. For example, describe_configuration() behaves normally.

await dev.describe_configuration()

{'dev-motor-units': {'dtype': 'string',
  'shape': [],
  'dtype_numpy': '|S40',
  'source': 'soft://dev-motor-units'},
 'dev-motor-velocity': {'dtype': 'number',
  'shape': [],
  'dtype_numpy': '<f8',
  'source': 'soft://dev-motor-velocity'},
 'dev-mode': {'dtype': 'string',
  'shape': [],
  'dtype_numpy': '|S40',
  'source': 'soft://dev-mode',
  'choices': ['continuous', 'step']}}

The following is the descriptor document produced by the RunEngine before being sent to the Publisher.

{'configuration': {'det': {'data': {}, 'timestamps': {}, 'data_keys': {}},
  'dev': {'data': {'dev-motor-units': 'mm',
    'dev-motor-velocity': 0.0,
    'dev-mode': <TestEnum.STEP: 'step'>},
   'timestamps': {'dev-motor-units': 669262.406071209,
    'dev-motor-velocity': 669262.406058968,
    'dev-mode': 669262.406297605},
   'data_keys': {'dev-motor-units': {'dtype': 'string',
     'shape': [],
     'dtype_numpy': '|S40',
     'source': 'soft://dev-motor-units'},
    'dev-motor-velocity': {'dtype': 'number',
     'shape': [],
     'dtype_numpy': '<f8',
     'source': 'soft://dev-motor-velocity'},
    'dev-mode': {'dtype': 'string',
     'shape': [],
     'dtype_numpy': '|S40',
     'source': 'soft://dev-mode',
     'choices': ['continuous', 'step']}}}},
 'data_keys': {'det-counts': {'dtype': 'integer',
   'shape': [],
   'dtype_numpy': '<i8',
   'source': 'soft://det-counts',
   'object_name': 'det'},
  'dev-motor-user_readback': {'dtype': 'number',
   'shape': [],
   'dtype_numpy': '<f8',
   'source': 'soft://dev-motor-user_readback',
   'object_name': 'dev'}},
 'name': 'primary',
 'object_keys': {'det': ['det-counts'], 'dev': ['dev-motor-user_readback']},
 'run_start': '4924f356-a7af-4ba6-8598-97064653cbc6',
 'time': 1733824335.0764165,
 'uid': 'dca83233-afe3-4493-9f8b-4040390fe418',
 'hints': {'det': {'fields': ['det-counts']},
  'dev': {'fields': ['dev-motor-user_readback']}}}

[coretl]

Looks like it's using pickle as the serializer, so this is expected:
https://github.com/bluesky/bluesky/blob/1b8422beee465d4f78f285a902b170a47303a75a/src/bluesky/callbacks/zmq.py#L61

@danielballan we decided on the following types:

• scalars
• StrEnums
• numpy arrays
• Tables which are BaseModels that serialize into dict[str, numpy array]

JSON needs help on the last 2, YAML on the last 3, zmq seems to be using pickle so requires those imports to be present on the client side when unpickling. What is the best approach here?