Rename multiplier to exposures_per_event and move to first dim of shape

[thomashopkins32]

This PR does the following:

• Renames multiplier -> frames_per_event
• Add the frames_per_event as the first dimension of DataKey.shape
• Ensure that the index provided by DetectorWriter.get_indices_written() and DetectorWriter.observe_indices_written() is divided by frames_per_event so that it actually captures the correct amount of exposures in each index (except for PandA which explicitly says it only has 1 "frame" per event)
• Add unit tests showing that describe() works as intended
• Add unit tests showing that stream resources are actually batches of exposures
• Re-order self._writer.open() and self._writer.get_indices_written(). The writer needs to be opened in order to get the indices written. Otherwise, it has no idea what frames_per_event to use when returning the index last written.

I could not actually add tests using bluesky plans and inspecting the data afterword because TriggerInfo is hardcoded in StandardDetector. I think it is a separate issue that should be raised since it would enhance the scope of this PR. I will open an issue for this soon and mention it below.

Otherwise, I have a few open questions regarding my understanding of ophyd-async as well as the implementation which I will also leave as review comments. Please see below.

Closes #576
Closes #431

[thomashopkins32]

@coretl @jwlodek

This might seem like overkill but please help me check my understanding of the terms. Here is what I understand:

Level	Term	Meaning	Example
EPICS (or some other control system)	"capture"	One single data point collected from a detector	A singular frame is collected from an area detector
ophyd-async	"trigger"	One or many "captures" that returns a status	5 frames are collected from an area detector
bluesky & beyond	"event"	Occurrence of a single "trigger" that finishes with successful status	5 frames are collected and a document is emitted

The TriggerInfo schema is actually pretty confusing when you try to do the following:

trigger_info = TriggerInfo(number_of_triggers=4, frames_per_event=2)
RE(bps.prepare(det, trigger_info))
RE(bp.count([det], num=10))

This example currently fails to run without error.

Is this something that should be possible? I can't even tell what the expected behavior here should be...

This may be an unintended consequence of doing a step scan versus a fly scan but I need to dig a bit further.

Either way, if there is a real issue here, it should probably be handled in a separate PR.

[coretl]

This might seem like overkill but please help me check my understanding of the terms.

My take:

Level	Term	Meaning	Example
EPICS (or some other control system)	"exposure"	One single data point collected from a detector	A singular frame is collected from an area detector
ophyd-async	"trigger"	One software or hardware trigger that is sent to the detector to trigger an exposure (with the given livetime)	A singular frame is collected from an area detector
bluesky & beyond	"event"	An atomic row of data in a stream within a scan, synchronous data points from multiple detectors	5 frames are collected from one detector as a single stream datum index, 1 frame from another as a single stream datum index, and a document is emitted with the motor position at this scan point

The TriggerInfo schema is actually pretty confusing when you try to do the following:

trigger_info = TriggerInfo(number_of_triggers=4, frames_per_event=2)
RE(bps.prepare(det, trigger_info))
RE(bp.count([det], num=10))

Definitely. There are 3 terms here:

• number_of_triggers -> number_of_events which is the number of events to produce for each kickoff in a flyscan, it should always be set to 1 for a step scan as each step is an event
• frames_per_event -> triggers_per_event which is the number of triggers and therefore the number of frames the detector will produce in each event
• num argument to count plan which is the number of events to produce

So number_of_events and num are both specifying the same thing, but in a step scan the number has to be controlled from the plan, so number_of_events must equal 1

This example currently fails to run without error.

This should error with something like "Can only trigger() to contribute to a single event, but detector was prepared with number_of_events=4"

[thomashopkins32]

@coretl Okay I understand now. It makes sense that a step scan can only have 1 trigger, by your definition of trigger.

However, the term "trigger" seems overloaded with bluesky's definition in the sense that a StandardDetector is Triggerable and a bluesky trigger happens once per event (in the case of a step scan, not sure of a fly scan). I suggest we use exposures_per_event to replace frames_per_event.

I will do the renaming part in this PR and add checks + expected functionality (with tests) in another PR.

[jwlodek]

I agree with @coretl 's understanding. One thing I thought of however when mulling this over, is that currently we are missing one thing when operating in step scan mode - namely that if I set num_images to say 5 in my AD, and run count with num=3, I'd expect 3 events w/ 5 frames per event, but currently we don't actually have a mechanism for setting frames_per_event in a step scan with trigger - you'd end up with three events of 5 images, but the shape would be incorrect with a frames_per_event of 1.

I think the easiest solution to this would be to add a property to StandardDetector that basically just returns the currently configured frames_per_event for a step scan, with it just returning 1 by default, with the expectation that the implementation should override this. So then for AD based detectors this would get the value of num_images for example, and would set frames_per_event=THE_VALUE_OF_THIS_PROPERTY in the implicit prepare in the trigger method.

Thoughts?

[coretl]

I think the easiest solution to this would be to add a property to StandardDetector that basically just returns the currently configured frames_per_event for a step scan, with it just returning 1 by default, with the expectation that the implementation should override this. So then for AD based detectors this would get the value of num_images for example, and would set frames_per_event=THE_VALUE_OF_THIS_PROPERTY in the implicit prepare in the trigger method.

How about we make frames_per_event optional? If it isn't set then the detector should read num_images and use it as the value of frames_per_event, if it isn't then it should set num_images to be frames_per_event.

I suggest we use exposures_per_event to replace frames_per_event.

I'm not sure whether frames or exposures is better, @jwlodek thoughts?

[thomashopkins32]

How about we make frames_per_event optional? If it isn't set then the detector should read num_images and use it as the value of frames_per_event, if it isn't then it should set num_images to be frames_per_event.

I think this should be handled in a different PR. There is some complexity in doing this. Something like this is already done for the TriggerInfo.deadtime but I'm not sure we want to do it the same way for frames_per_event.

We would basically be doing the following:

frames_per_event <- num_images
num_images <- number_of_events * frames_per_event

So if someone re-uses a TriggerInfo that has frames_per_event=None, then the num_images could increase over-time if number_of_events > 1. I think this user would have to call prepare twice in the same bluesky run though?

[coretl]

I think this should be handled in a different PR. There is some complexity in doing this. Something like this is already done for the TriggerInfo.deadtime but I'm not sure we want to do it the same way for frames_per_event.

Agreed.

We would basically be doing the following:

frames_per_event <- num_images
num_images <- number_of_events * frames_per_event

I think I've got some terminology confused. In my mind we're mapping:

• TriggerInfo.livetime -> ADAcquireTime
• TriggerInfo.livetime + TriggerInfo.deadtime -> ADAcquirePeriod
• TriggerInfo.number_of_events -> ADNumImages
• TriggerInfo.frames_per_event -> ADNumExposures

As defined in https://areadetector.github.io/areaDetector/ADCore/ADDriver.html

All of those fields in TriggerInfo should be optional apart from number_of_events. If there is an optional field, it should take its value from the currently set areaDetector value. number_of_events should always be 1 in a step scan.

So if someone re-uses a TriggerInfo that has frames_per_event=None, then the num_images could increase over-time if number_of_events > 1. I think this user would have to call prepare twice in the same bluesky run though?

I think I made a mistake in the current implementation, the passed in TriggerInfo should be immutable and we make a copy with a different deadtime. That would stop any surprises when a TriggerInfo was reused.

[thomashopkins32]

I think I've got some terminology confused. In my mind we're mapping:

* `TriggerInfo.livetime` -> `ADAcquireTime`

* `TriggerInfo.livetime + TriggerInfo.deadtime` -> `ADAcquirePeriod`

* `TriggerInfo.number_of_events` -> `ADNumImages`

* `TriggerInfo.frames_per_event` -> `ADNumExposures`

As defined in https://areadetector.github.io/areaDetector/ADCore/ADDriver.html

My understanding of the ADDriver is not great, so I will defer to @jwlodek on this one. My understanding was that we want multiple images (or data points) per event. So maybe "exposure" is not the right word and we just use "point" instead.

All of those fields in TriggerInfo should be optional apart from number_of_events. If there is an optional field, it should take its value from the currently set areaDetector value. number_of_events should always be 1 in a step scan.

Agreed! I think this would be great.

I think I made a mistake in the current implementation, the passed in TriggerInfo should be immutable and we make a copy with a different deadtime. That would stop any surprises when a TriggerInfo was reused.

Yes currently it modifies the existing TriggerInfo

[jwlodek]

Tested w/ the sister bluesky PR, in a variety of cases, and they all seem to work. @coretl OK to merge?

[coretl]

Tested w/ the sister bluesky PR, in a variety of cases, and they all seem to work. @coretl OK to merge?

Just discussed in the collab call, and there was widespread concern about what else might break when these descriptors change from [] to [1]. @genematx also raised a point about 1D arrays that I didn't understand, so I think we need more discussions with @danielballan before we merge. Maybe we discuss at next collab call?