Proposed types for Scientific Metadata

[nitrosx]

This issue is an attempt to summarize the issues and discussions following PR #925 and issues #924 and #940.

In the biweekly meeting, the community expressed support in having multi-dimensional quantities in the metadata, although it is concerned by the fact that there is no limit on the dimensionality of the array, opening up to the possibility to allow a user to save a time series as metadata value.

To address the use case presented in #924 but avoid lengthy time series in metadata, I proposed to allow the following metadata types (italics indicates new types):

• number: plain unit-less number
• string: plain string
• [1d] quantity: a one-dimensional quantity with associated unit. Example: length of 1m
• 2d quantity: a two-dimensional quantity with associated unit. Example: a point on a plane, [1,2]m
• 3d quantity: a three-dimensional quantity with associated unit. Example: a point in space, [1,2,3]m
• quantity range: Example: a frequency range like [1,10]Hz
• datetime: ISO 8601 string indicating a date and time. Example: 2024-01-02T17:06:04+07:00
• datetime range: 2 ISO 8601 strings indicating a date and time range. Example: [2024-01-02T13:06:04+07:00, 2024-01-03T19:06:04+07:00]
• date: ISO 8601 string indicating a date: Example: 2024-01-02
• date range: 2 ISO 8601 strings indicating a date range: Example: [2024-01-01, 2024-01-03]
• time: ISO 8601 string indicating a time. Example: T17:06:04+07:00
• time range: 2 ISO 8601 strings indicating a time range. Example: [T13:06:04+07:00, T19:06:04+07:00]
• object: container for nested metadata fields.

Search on types number, string and quantity will continue to behave as they currently behave.
I proposed the following search syntax and behavior for the new types:

• 2d quantity and 3d quantity:
  • q op v.
    It is expanded to an or operator of the same condition on all the elements of the quantity.
    It will be true if it is true for at least one of the element of the quantity
    Example:
    • Value: position = [1,2,3]m
    • Condition: position = 3m
    • Expansion: position[0] = 3m or position[1] = 3m or position[2] = 3m (False or False or True)
    • Result: True
  • q[i] op v
    This apply the condition directly to a single element of the quantity.
    Example:
    • Value: position = [1,2,3]m
    • Condition: position[2] = 3m
    • Expansion: position[2] = 3m (True)
    • Result: True
• datetime, date, and time
  All the date/time operators will be supported.
• range (all range types)
  • r = v, v in r
    Does the range r contains the value v
    Example:
    • Value: frequency = [1,10]Hz
    • Condition: frequency = 3Hz
    • Expansion: frequency[0]<3Hz and frequency[1]>3Hz (True and True)
    • Result: True
  • q > v
    Is range r greater than v?
    Example:
    • Value: frequency = [1,10]Hz
    • Condition: frequency > 0.5Hz
    • Expansion: frequency[0]>0.5Hz (True)
    • Result: True
  • q < v
    Is range r less than v?
    Example:
    • Value: frequency = [1,10]Hz
    • Condition: frequency < 11Hz
    • Expansion: frequency[1]<11Hz (True)
    • Result: True

All the relevant tests must be added in to the BE and FE.

This is just a proposal. Please comment below and elaborate further the ideas.

[bpedersen2]

Sounds good to me and should cover a good portion of our use cases.

[jkotan]

Hi @nitrosx,
not sure I understand idea. Would you like to add unit support for only the above mentioned types or forbid all other types in scientific metadata (hope not).
At DESY we have to types of Datasets: scan datasets and measurement datasets. The latter groups metadata from the scans. In the measurement metadata we often use lists to aggregate metadata from different scans e.g. a list of ScanCommand or inputDatasets (in the case measurement datasets are raw-like).
Also we had discussion with our beamline scientists how to store 3x3 hkl matrix. Currently we store it encoded in a string but often it is stored as a list of 9 numbers.
I could also imagine that some one would like to store 4-vectors which are much more natural in theory or particle physics.

[nitrosx]

@jkotan unit support will be available for the following types:

• quantity
• 2d quantity
• 3d quantity
• quantity range

Regarding your examples of scan and measurements dataset, could you please provide an example of them. I'm not sure I do understand what are their differences and how relevant they are regarding allowing multi-dimensional quantities in the metadata.

Regarding your last two points, my main concern (and the concern of most of the collaborators) is that allowing n-dimensional quantities in the metadata, we will exhaust the available space in the document and there will be a sprawling of time series in the metadata or data slowly sipping in the metadata.
I will be happy to discuss!!!

[jkotan]

Hello @nitrosx,
At DESY we've started with creating dataset for each scan (scan dataset). However, we perform a lot of scans, so our IT has reported that our scan datasets uses a lot of DB resources. Therefore we start to think to create a dataset not for a single scan but for a group of scans related to each other i.e. some kind of measurement e.g. 'calibration'.
For such measurement we create a measurement dataset where we group most important scientificMetadata from our scans. For the string quantities which are constant it is easy,

scientificMetadata:
   DOOR_proposalId: "99991173"

For number quantities which are almost constant we store an average, min, max and std.

scientificMetadata:
  source_current: 
    counts: 3
    max: 0.02578197419643402
    min: 0.025225341320037842
    std: 0.0002973251885217321
    unit: "mA"
    value: 0.02556405154367288
    valueSI: 0.00002556405154367288
    unitSI: "A"

However, important scan quantities which are different for each scan, e.g. like a scan command, we need to store as a list of strings.

scientificMetadata:
  ScanCommand: 
    - "ascan exp_mot02 0.0 6.0 6 0.1"
    - "ascan exp_mot01 0.1 6.0 6 0.1"
    - "ascan exp_mot02 0.0 5.0 6 0.1"

Similarly our users/beamline-scientiests have request to aggregate also numerical physical quantities which are changing from scan to scan, i.e. to store them in a list, one value for each scans in a measurement. For such quantities average is not useful e.g.
neither in Poisson nor in Gaussian distribution, e.,g. some motor positions.

A number of scan for each measurement could be different e.g. 1 or 1000. The aim to do it is to reduces data size storage, i.e. do not store duplicated metadata in a series of similar scans.

Of course all our solutions still under discussion so we don't know what will be the structure of our final production datasets.