Support units in Table Schema

[rufuspollock]

Move the units draft spec http://specs.okfnlabs.org/units/ back to FD specs

/cc @pwalsh

[pwalsh]

@rufuspollock before we move it here, it would be good to discuss how it is to be implemented. Is the proposal to implement this spec, as is, as part of Table Schema?

[rufuspollock]

@pwalsh open to suggestions. I was thinking of keeping it separate but adding support for referencing it from table schema but not sure what is best.

[Stephen-Gates]

I've drafted a pattern #607 and started a discussion on the forum https://discuss.okfn.org/t/table-schema-units-pattern/6573

[rufuspollock]

Notes from thread in the PR #607

@Stephen-Gates wrote:

@rufuspollock I put this up to stimulate conversation. I guess we need to resolve if we adopt the suggestion on the forum about using an existing specification UCUM, ISO 80000, BIPM.

@rufuspollock wrote back

@Stephen-Gates have you looked at any of those specs in detail? UCUM seems pretty heavy duty - is there a subset of that we could adopt / summarize?

We really just want something relatively simple that fits the 80/20 rule ...

Re ISO 80000 - is that open? If not that would be an issue for us ...

The BIPM stuff looks good though very physically oriented and fundamental.

/cc @dr-shorthair

@dr-shorthair wrote

UCUM may look heavy duty (though I don't think it is, really). But it provides an approach to build any unit-of-measure-symbol from the atomic elements.

This gets over the problem that any static set will come up short. A static list might look like 80-20, but what do you say to the people that need one of the 20 (which is still a lot of applications!).

[rufuspollock]

@Stephen-Gates (/cc @dr-shorthair) I think our aim is to see if we can extract a subset of UCUM that gives us 80/20 and then say if you want more go to UCUM.

I have to say i think this could / should go in 2 stages:

1. Move the spec back here (as it is - even if flawed)
2. Then start upgrading it ... (this can be a new issue ...)

This keeps things clean. wdyt? And if so would that mean we could merge #607 as it is?

[rufuspollock]

@Stephen-Gates any luck here on progressing this?

[Stephen-Gates]

Sorry been focussed a new release of Data Curator. Haven’t forgotten

[lwinfree]

Hi @Stephen-Gates (and also @rufuspollock + @pwalsh)! I wanted to let you know that @mbomhoff from Planet Microbe has been working with data packages for their oceanographic data and has been thinking about what units specs would work best for them. I wanted to tag Matt so he can keep updated on the specs units conversation, and also intro y'all in case you want to connect and discuss what units ideas Matt has. Thanks both 😄

[mbomhoff]

@lwinfree Thanks Lilly! Our data packages are in https://github.com/hurwitzlab/planet-microbe-datapackages. For the time being we added a custom property unitRdfType.

[Stephen-Gates]

@mbomhoff are you ok with the direction the draft PR was taking if we address the comments above?

[Stephen-Gates]

@rufuspollock do you have any concerns about using UCUM given its licence?

[dr-shorthair]

Item 2. in the license is problematic:

Users shall not modify the Licensed Materials and may not distribute modified versions of the UCUM table (regardless of format) or UCUM Specification. Users shall not modify any existing contents, fields, description, or comments of the Licensed Materials, and may not add any new contents to it.

Unfortunately UCUM now appears to be an infrastructure orphan - I've not been able to make contact with Guenther Schadow for a couple of years now. Possibly retired. I'll try again.

[mbomhoff]

@Stephen-Gates It looks like the draft spec is capable of describing all of the units that we use in our project, but I think our application falls under the case of using an existing spec. One of the goals of our project is to use ontologies to unify disparate datasets from various sources. To describe a field we supply an Environment Ontology (ENVO, http://environmentontology.org/) purl in the rdfType property and a Unit Ontology (UO, https://github.com/bio-ontology-research-group/unit-ontology) purl in a custom unitRdfType property. If the proposal is adopted we would add the unit property for consistency but most likely keep our existing rdfType and unitRdfType properties. For example, to describe the "depth" field we would use:

rdfType: "http://purl.obolibrary.org/obo/ENVO_3100031",
unitRdfType: "http://www.ontobee.org/ontology/UO?iri=http://purl.obolibrary.org/obo/UO_0000008",
unit: "m"

For us the UO purl provides stronger semantics and some additional info such as aliases (meter, metre) and a text description.

[rufuspollock]

@Stephen-Gates any chance to look at this further. It sounds like we have to steer around UCUM atm.

[dr-shorthair]

UCUM only provides the terminal symbols, and a grammar to combine them into any UoM. So it is a mistake to talk about 80:20 provide by UCUM with respect to some finite set. UCUM probably provides 95%+, but by utilising the grammar.

Meanwhile, I have now tracked down the owner of UCUM so it's not dead yet.

[Jacob-Barhak]

Lilly Winfree directed me to this discussion after I asked her how you handle units in your project in PyData Asutin. We have been dealing with Unit standardization for over a year and can connect you to some of unit specs - at least in the medical domain.
However, you will see that we did a lot of the work already in the following project:
https://clinicalunitmapping.com/
In the About page, you will see many publications that describe details. Currently we already bundle 4 unit specs/standards and have machine learning tools developed to address those - it is easy to add more specs. If you are interested in joining forces, please contact me by email outside this Github thread to see how it is possible. The project is currently not fully open, yet there is a desire by some stakeholders to keep some of its elements open. So I will appreciate offline communication before returning to this open thread and continuing public discussion.
In any case, please note that some open standards contain UCUM, so your fear of the UCUM license was already handled by others successfully, so it may be possible to resolve this issue with some effort. Yet I need to see your issues first and discuss how this can be resolved offline.
Looking forward for more communications.

[rufuspollock]

@Jacob-Barhak this is great info - if you could share your experience and links that would help esp any key pointers. Your tip re UCUM is also very helpful. We will look at https://clinicalunitmapping.com/

[Jacob-Barhak]

So @rufuspollock , all documentation associated with the project is available in the about page: https://clinicalunitmapping.com/about
You will find many publications already and some presentations, yet no important code is currently open source. If your problem is small, then you should have enough pointers there to resolve your unit issues - yet if you are interested in a global solution for the units problem, we should schedule a video chat and talk.
Again, despite expressed desire by many stakeholders to make it an open source project, it is currently not fully available to the public. If you have ideas on how to make it available, I am open to suggestions.

[cpina]

If adding support to units was done: frictionless-py could output the values with its units (maybe optionally) using https://pint.readthedocs.io/en/stable/ Obviously if the units used in the spec were available in the Pint library.

[jgunstone]

made a comment about units over here, acep-uaf/aetr-web-book-2024#40

but maybe this issue is a more appropriate place so I've copied below:

another units good standard: https://www.qudt.org/doc/DOC_VOCAB-UNITS.html

FYI in case its useful / interesting:
the https://github.com/annotated-types/annotated-types?tab=readme-ov-file#unit package has an implementation that allows you to save units as metadata only; and then in the docs has a demonstrattion of a BYO units package implementation... thus side-stepping any authority on units or the units packages / standards that should be used....

[Jacob-Barhak]

Are you still looking for solutions for units?

You may want to check advances in clinicalunitmapping.com
https://clinicalunitmapping.com/infer

There is now AI behind this that is pretty good already. This project is still in beta and there is still work to do and its use is limited to demonstrate feasibility, yet it is getting better.

You can find recent publications in:
https://clinicalunitmapping.com/about

What are your unit needs? Why exactly do you need them? Will be happy to talk via video.

[dr-shorthair]

Also see https://si-digital-framework.org/ from BIPM who are the authority on SI.

[pvgenuchten]

hi, what is the status here? unit is crucial for our users.

related to https://discuss.okfn.org/t/table-schema-units-pattern/6573

It doesn't make sense to add a reference to a unit in rdftype, because rdftype should refer to 'max-height'
it could be a way forward though, rdfType could point to a measurable property, which itself indicates its favourable unit

Another thing I wonder (which could also be an aspect of the rdfType) is if at this level a reference can be added to a procedure (for example a laboratory method) on how the value was estimated, and maybe an indication related to the accuracy of the value?

I noticed that unit is included in the camtrap-dp

[roll]

Hi @peterdesmet,

WDYT? This feature request is one of the most popular historically. Is it possible to land it in the standard?

[peterdesmet]

Yes, I think we can add unit as a field property to Table Schema. There are several ways to extend this property (e.g. linking out to controlled vocabularies through e.g. unitRdfType), but I would leave that to users and keep it generic in Data Package:

unit:
  title: Unit
  definition: The unit for this field.
  type: string

Example:

{
  "name": "speed",
  "type": "number",
  "unit": "m/s"
}

To keep it generic:

• I would not restrict values
• I would not restrict the property to certain field types, although it could be argued to restrict it to string, integer, number and perhaps any.
• I would not recommend any vocabularies in the definition, although it could be argued to recommend some (cf. how we do that for contributor roles).
• I would not recommend the use of unitRdfType but leave that to the users. Linking out to definitions is a larger discussion to be had.

What advantage does this have? Implementations will know to find the string representation for a field in the unit property.

[nichtich]

What's the advantage of unit if it can be any string? This is already possible:

{
  "name": "speed",
  "description": "speed in m/s",
  "type": "number"
}

I'd welcome a unit field with defined semantics and either UCUM or unit URIs (such as QUDT) or both seem to fit well:

Example with valid UCUM unit (m/s) and unit URI (redundant):

{
  "name": "speed",
  "unit": "m/s",
  "unitType": "https://qudt.org/vocab/unit/M-PER-SEC"
  "type": "number",
}

[nichtich]

Another thing I wonder (which could also be an aspect of the rdfType) is if at this level a reference can be added to a procedure (for example a laboratory method) on how the value was estimated, and maybe an indication related to the accuracy of the value?

In many cases that information is not fixed for every value of the field so its better put in a separate field. Even if it is known and fixed for a field, I don't know any established formal standard how to express it and we better should not try to invent our own standard without comprehensive evaluation how people give this information in actual data sharing.

[dr-shorthair]

QUDT is well maintained - https://github.com/qudt/qudt-public-repo/releases
It provides a URI for each unit, e.g. https://qudt.org/vocab/unit/M https://qudt.org/vocab/unit/DEG_C https://qudt.org/vocab/unit/KiloGM etc - the complete catalog is https://www.qudt.org/doc/DOC_VOCAB-UNITS.html

If you need a new unit, then it is usually turned around in a week or so - https://github.com/qudt/qudt-public-repo/wiki/Unit-Vocabulary-Submission-Guidelines .

[nichtich]

I know and I like QUDT and advocate for using it in RDF data, but what's the proposed use of it in table schema? If units are going to be specified with URIs then we should support any URI but recommend using QUDT URIs, is this what you propose?

recommended:

{
  "name": "max-height",
  "type": "number",
  "unitType": "https://qudt.org/vocab/unit/M"
}

also possible:

{
  "name": "max-height",
  "type": "number",
  "unitType": "http://purl.obolibrary.org/obo/UO_0000008"
}

[Jacob-Barhak]

Hi Guys, You are dealing with a much larger issue than you think - units are not standardized despite what people think and despite multiple standards. Ucum units are pretty limited. Not sure what is your scope, yet if you dive into this problem enough, you will end up doing what I am doing already for several years and it is far from trivial. I mentioned my work before in this thread. If you need a specific local solution, then it's easy, yet once you see the real scope and open to other scenarios like it seems you are starting to do, it will become much harder and a project on its own. So try to keep it simple. Good luck

…

On Wed, Nov 6, 2024, 16:07 Jakob Voß ***@***.***> wrote: What's the advantage of unit if it can be any string? This is already possible: { "name": "speed", "description": "speed in m/s", "type": "number" } I'd welcome a unit field with defined semantics and either UCUM or unit URIs (such as QUDT) or both seem to fit well: Example with valid UCUM unit (m/s) *and* unit URI (redundant): { "name": "speed", "unit": "m/s", "unitType": "https://qudt.org/vocab/unit/M-PER-SEC" "type": "number", } — Reply to this email directly, view it on GitHub <#1002 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAPE423NUGBLRMC7PZT7J73Z7IPCNAVCNFSM6AAAAABRILF67OVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMJWG4YTOOI> . You are receiving this because you were mentioned.Message ID: <frictionlessdata/datapackage/repo-discussions/1002/comments/11167179@ github.com>

[nichtich]

Above you wrote

What are your unit needs?

That's the crucial question. As far as I understand, the need is how to optionally express the unit of values in a field (such as field speed with plain number values) so the values can be calculated with (e.g. compare values given in m/s with other values given in miles per hour). The only practical solution I can think of is to allow referencing a unit from an existing standard such as UCUM code or QUDT URI.

[jgunstone]

Agree simple is good -
In the absense of a dominant standard I wonder if the best approach is to define the data that would be stored, e.g.

{
  "unit_symbol": "M",
  "unit_uri":  "https://qudt.org/vocab/unit/M",
}

I think that this gets you quite far quite quickly.
A user can then write the code to handle these units... and overtime perhaps support for common units systems like ucum or qudt get added to the core lib or added as an extension.

I think the most important thing is giving users a consistent container for storing unit information

[roll]

Currently, the recommended way for using existent dictionaries is like:

{
"key": "value",
"dct:key": "value"
}

Can this approach be used with units?

[Jacob-Barhak]

John and Roll, Not sure where you are heading, yet since there is no one standard, the unit and source should be good enough for now - in some cases you will not have URI - yet it's important to have some level of context. URI can change in time as well as standards that have versions. In some extreme cases a unit may mean something different - for example M can be understood as meter or a Mega like in Mb. So really there is plenty of confusion and whatever you do will not answer the entire question. How about using { "unit_symbol": "M", "unit_uri": "https://qudt.org/vocab/unit/M", "source": "qudt", } This will take you a long way and leave the real bad problems to someone else to deal with. Its not perfect, yet you really dont want to deal with the unit mess - if you do, please contact me - I have plenty of boring work I need help with. Also in this case if UI does not exist, you at least know the source so someone can trace it back. Yet you can also decide that the source is either a URI or plain text and continue this way. Its your system - you choose. Whatever you choose - good luck, units are not as trivial as everyone thinks and you better know the limitations , this discussion is actually important.

…

On Thu, Nov 7, 2024 at 3:07 AM roll ***@***.***> wrote: Currently, the recommended way for using existent dictionaries is like: {"key: "value","dct:key": "value" } Can this approach be used with units? — Reply to this email directly, view it on GitHub <#1002 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAPE4267S5S6GBQUPJAODBLZ7MUVXAVCNFSM6AAAAABRILF67OVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMJXGUZTQNI> . You are receiving this because you were mentioned.Message ID: <frictionlessdata/datapackage/repo-discussions/1002/comments/11175385@ github.com>

[tucotuco]

@Jacob-Barhak This solution of three terms is exactly the generic solution we found for terms expected to come from controlled vocabularies. For example, if the root term in question is "eventType", then it could be populated with a string literal, accompanied by the equivalent of your "source" in a term called "eventTypeVocabulary". The third term, "eventTypeIRI" allows a resolvable controlled vocabulary term to be declared directly. This covers all the situations we need, 1) I made up this value for eventType, 2) this value for eventType can be found in that vocabulary (but they don't have resolvable controlled values), and 3) the exact value I am talking about can be found at this IRI.

[Jacob-Barhak]

Ok John, If you figured it out already, you are on a good route. I really suggest not to mess with units at a deeper level unless you want your entire project to lose focus. If you have a simple solution, just go with it. You can follow my work to see when a better solution appears specifically for units. It's a slow process, yet it will get there eventually. Good luck

…

On Thu, Nov 7, 2024, 18:09 John Wieczorek ***@***.***> wrote: @Jacob-Barhak <https://github.com/Jacob-Barhak> This solution of three terms is exactly the generic solution we found for terms expected to come from controlled vocabularies. For example, if the root term in question is "eventType", then it could be populated with a string literal, accompanied by the equivalent of your "source" in a term called "eventTypeVocabulary". The third term, "eventTypeIRI" allows a resolvable controlled vocabulary term to be declared directly. This covers all the situations we need, 1) I made up this value for eventType, 2) this value for eventType can be found in that vocabulary (but they don't have resolvable controlled values), and 3) the exact value I am talking about can be found at this IRI. — Reply to this email directly, view it on GitHub <#1002 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAPE422ADY55GMQGDZ5WB5TZ7OGDNAVCNFSM6AAAAABRILF67OVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMJXHE3TGMQ> . You are receiving this because you were mentioned.Message ID: <frictionlessdata/datapackage/repo-discussions/1002/comments/11179732@ github.com>