Use case: products derived from multiple sources

[RobertPincus]

In some cases variables will be derived from multiple sources of information. How should the instrument for such variables be described if all sources are from the same platform? What if they are from different platforms?

[bjorn-stevens]

We had the idea of virtual platforms, one could also have virtual instruments, only requiring varialbes : instruments : platform : platform … platform : campaign. I think this could work, but I wonder if we make our life easier, and not too restrictive, if we first think about providing a meta-data standard for level1 data, and this is data that comes from a physical instrument.

…

On 12. May 2020, at 21:06, Robert Pincus ***@***.***> wrote: In some cases variables will be derived from multiple sources of information. How should the instrument for such variables be described if all sources are from the same platform? What if they are from different platforms? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#10>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APKHP3IT662MEJCV4YROASDRRGM3VANCNFSM4M7C5IGA>.

---------- https://www.mpimet.mpg.de/en/staff/bjorn-stevens/

[RobertPincus]

I’m thinking of a synthesis or blending of measurements from different instruments into a variable. Calling this a virtual instrument seems confusing. One possibility: each file contains one or more instrument variables - these have to be unique (Tobi’s instrument IDs) so there can be more than one per file. Each variable has campaign, [activity], and platform(s) attributes. Data variables have an instrument attribute that points to the instrument variable. Then each data variable is linked one or more instruments, each on a platform, participating in a campaign.

[bjorn-stevens]

Yes, but maybe it is better for synthetic variables to be sourced from variables, rather than instruments. I know this causes a potential confusion as many variables that we associate with instruments are actually sourced from other variables (voltages) that come from that same instrument, but I think the distinction would be clearer in that if you can source a variable to measurements from a physical instrument it is data, if you source it to variables from multiple instruments, it is something else… but maybe we don’t even have to say that, rather we can say that variables can come from other variables or from instruments, instruments can come from other instruments or platforms, platforms can come from other platforms or campaigns. I’m not for splitting campaigns because it is non-informative, actually it is anti-informative. But we would check for completeness, i.e., that following the various paths upwards one always ends at a campaign, and the same campaign along all paths. Cheers, Bjorn

…

On 13. May 2020, at 00:53, Robert Pincus ***@***.***> wrote: I’m thinking of a synthesis or blending of measurements from different instruments into a variable. Calling this a virtual instrument seems confusing. One possibility: each file contains one or more instrument variables - these have to be unique (Tobi’s instrument IDs) so there can be more than one per file. Each variable has campaign, [activity], and platform(s) attributes. Data variables have an instrument attribute that points to the instrument variable. Then each data variable is linked one or more instruments, each on a platform, participating in a campaign. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#10 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APKHP3M73SJOBYODPXCIWBLRRHHQDANCNFSM4M7C5IGA>.

---------- https://www.mpimet.mpg.de/en/staff/bjorn-stevens/

[RobertPincus]

It’s a good idea to have derived variables source from other, more fundamental variables. It may be, though, that some variables in the tree leading back to basic measurements are not present in the archive.

[d70-t]

I like the idea of sourcing derived variables from other variables. Also if that concept is taken further, it may be extended such that only metadata of derived variables needs to be stored, as this data would then be sufficient to recompute the variables. I would not say that this should be the default, but for variables which are hard to store but easy to compute, this seems to be an appropriate solution.

So we may have the following relations?:

variable   --(measured by)------> instrument (n:1)
variable   --(derived from)-----> variable (n:m)
instrument --(part of)----------> platform (n:1)
platform   --(part of)----------> platform (n:1)
platform   --(participated in)--> campaign (n:m)

Storing the relations in the things which potentially are lower in the hierarchy (e.g. even data files) seems to be a good choice, as this makes it easier to add more instruments and variables without the need to edit higher level files.

One thing to consider is that everything which is pointed to needs to have a unique id. Accordingly, if we want to point to variables, every variable needs to have an id which is unique across everything which we want to cover with our metadata. I think that this is a good thing, but that must then be conveyed to all data providers.

It may be a good idea to allow the addition of external variables, which are not fully covered by the metadata structure, but at least has a unique id and the possibility to add a description of how this data can be obtained or generated. That way, it would still be possible to formally point to such a variable and to give the user of a derived variable a chance for retracing how the derived variable has been created.

The part of relations may need to have at time frame associated to the relation, as potentially instruments and sub-platforms could be moved between platforms. This also would maybe contradict the (n:1) scheme for these relations.

For the version with re-computation the derived from relation must be augmented with a recipe. Just a detail: this might require a variable can have one derived fromrelation pointing to many variables in stead of having manyderived from` relations pointing to one variable each.

[bjorn-stevens]

I like this… could we move to some suggestions for key words and try to finalize a first form of the meta-data for release next week? My only modification would be that platform —> campaign is (n:1). There the reasoning is that what we call platforms are incarnations that come into being for a particular campaign, i.e., HALO in NARVAL is not HALO in EUREC4A. I realize that there is a tendency for platforms to think of themselves as campaigns in themselves, but this isn’t helpful or informative. The community building that different groups might want to do around their projects or sub-projects, or funding source, can perhaps be handled by a different part of the meta data, i.e., ‘project’ and that is where things like ISOTOPES, EUREC4_UK, ATOMIC, EUREC4A_OA, etc could find expression.

…

On 14. May 2020, at 11:04, d70-t ***@***.***> wrote: I like the idea of sourcing derived variables from other variables. Also if that concept is taken further, it may be extended such that only metadata of derived variables needs to be stored, as this data would then be sufficient to recompute the variables. I would not say that this should be the default, but for variables which are hard to store but easy to compute, this seems to be an appropriate solution. So we may have the following relations?: variable --(measured by)------> instrument (n:1) variable --(derived from)-----> variable (n:m) instrument --(part of)----------> platform (n:1) platform --(part of)----------> platform (n:1) platform --(participated in)--> campaign (n:m) Storing the relations in the things which potentially are lower in the hierarchy (e.g. even data files) seems to be a good choice, as this makes it easier to add more instruments and variables without the need to edit higher level files. One thing to consider is that everything which is pointed to needs to have a unique id. Accordingly, if we want to point to variables, every variable needs to have an id which is unique across everything which we want to cover with our metadata. I think that this is a good thing, but that must then be conveyed to all data providers. It may be a good idea to allow the addition of external variables, which are not fully covered by the metadata structure, but at least has a unique id and the possibility to add a description of how this data can be obtained or generated. That way, it would still be possible to formally point to such a variable and to give the user of a derived variable a chance for retracing how the derived variable has been created. The part of relations may need to have at time frame associated to the relation, as potentially instruments and sub-platforms could be moved between platforms. This also would maybe contradict the (n:1) scheme for these relations. For the version with re-computation the derived from relation must be augmented with a recipe. Just a detail: this might require a variable can have one derived fromrelation pointing to many variables in stead of having manyderived from` relations pointing to one variable each. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#10 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APKHP3KOBWGXSSWNBN66YS3RROXZHANCNFSM4M7C5IGA>.

---------- https://www.mpimet.mpg.de/en/staff/bjorn-stevens/

[RobertPincus]

@bjorn-stevens Agreed on the value of optional attributes for further community building. We’ve called this “activity” in previous iterations. “project” would also work. @d70-t I agree with the idea of as much metadata as possible being stored in the files rather than higher-level files. This way it follows the data around. We will want to harmonize/control/ensure consistency for some elements of the metadata. These are at least campaign, platform, activity. The existing EUREC4A.yml file could be considered the source for this information. How will we ensure the uniqueness of IDs? How do we think about unique IDs for, say, dropsondes or radiosondes? One per platform/type combination e.g. dropsondes from HALO? One per sonde?

[bjorn-stevens]

I had another thought on uniqueness…. maybe what has to be unique is the realization. i.e., if the platform ID is HALO that is not necessarily unique in the sense that HALO in a different campaign would be different, different meta-data, different configuration, different science team, so almost everything is different, so what is unique is the combination of the platform ID and the campaign ID. This relaxes the uniqueness constraint. Hence the dropsonde ID is not enough to make it unique, and we probably don’t need to specify disposable instrumentation in a way that allows us to find a second incarnation of it, but for reused instrumentation you would like to be able to find the Leipzig RPG radar and distinguish it from the Cologne or MPI one in future campaigns, whether this really needs a unique ID, rather than a unique combination, seems unclear to me.

…

On 14. May 2020, at 15:16, Robert Pincus ***@***.***> wrote: @bjorn-stevens Agreed on the value of optional attributes for further community building. We’ve called this “activity” in previous iterations. “project” would also work. @d70-t I agree with the idea of as much metadata as possible being stored in the files rather than higher-level files. This way it follows the data around. We will want to harmonize/control/ensure consistency for some elements of the metadata. These are at least campaign, platform, activity. The existing EUREC4A.yml file could be considered the source for this information. How will we ensure the uniqueness of IDs? How do we think about unique IDs for, say, dropsondes or radiosondes? One per platform/type combination e.g. dropsondes from HALO? One per sonde? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APKHP3O7L7AIDIYJP3P6RR3RRPVK7ANCNFSM4M7C5IGA>.

---------- https://www.mpimet.mpg.de/en/staff/bjorn-stevens/