Feat/enable native decimal support #282
Open
+5
−4
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Enables native Arrow decimal type support by removing unnecessary blocking condition in
ScanRow
and enabling native decimal support by default. Thedecimal128Container
infrastructure was already complete - these changes allow decimal columns to return native Arrowdecimal128
types instead of raw bytes.Changes
UseArrowNativeDecimal = true
inArrowConfig.WithDefaults()
ScanRow
error conditionWhy Native Arrow Decimal128?
This provides significant advantages over the previous raw bytes approach:
decimal128
types with precision/scale metadataarrow.DECIMAL128
instead ofarrow.STRING
Impact
Before:
UseArrowNativeDecimal = false
by default[]byte
) in Arrow records requiring manual parsingarrow.STRING
type for decimal columnsAfter:
UseArrowNativeDecimal = true
by defaultarrow.Decimal128
types in Arrow recordsarrow.DECIMAL128
with precision/scale metadatafloat64
available viadecimal128Value.ToFloat64(scale)
For Direct Arrow Usage
Users working with Arrow records directly now get:
Configuration Note
The
UseArrowNativeDecimal
option is currently internal and not exposed through the public API. Users cannot currently override this setting, but the new default behavior provides proper Arrow type semantics.Testing
✅ Updated test expectations to reflect new default behavior
✅ All tests pass locally
✅ Arrow records now contain native decimal128 types instead of strings
Breaking Change
This is a minor breaking change for users directly accessing Arrow records, as decimal columns now return
arrow.Decimal128Array
instead ofarrow.StringArray
. However, this provides the correct Arrow type semantics and eliminates the need for manual parsing.