You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
With the PyIceberg <> Iceberg-Rust integration I'm running into issues with the Decimal literal.
Problem
The decimal comes with:
Precision: is the number of digits in a number.
Scale: the number of digits to the right of the decimal point in a number.
However, internally it is stored as a i128, which is fine for the Precision, since everything fits in there. Unfortunately, you cannot use this primitive on its own since you don't know the scale.
Potential solution
My first instinct was to introduce a separate primitive that would maintain the decimal scale. Unfortunately, Rust does not have an internal type; therefore, we now use rust_decimal. However, I don't like the idea of exposing third party types in the public Iceberg-Rust API. As pointed out in issue #669, rust_decimal does not support full precision. Switching libraries would already entail breaking the API.
I wanted to start a discussion to see if more folks have run into this and if there are any solutions I've missed.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
-
With the PyIceberg <> Iceberg-Rust integration I'm running into issues with the Decimal literal.
Problem
The decimal comes with:
However, internally it is stored as a
i128, which is fine for the Precision, since everything fits in there. Unfortunately, you cannot use this primitive on its own since you don't know the scale.Potential solution
My first instinct was to introduce a separate primitive that would maintain the decimal scale. Unfortunately, Rust does not have an internal type; therefore, we now use
rust_decimal. However, I don't like the idea of exposing third party types in the public Iceberg-Rust API. As pointed out in issue #669,rust_decimaldoes not support full precision. Switching libraries would already entail breaking the API.I wanted to start a discussion to see if more folks have run into this and if there are any solutions I've missed.
Beta Was this translation helpful? Give feedback.
All reactions