Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FieldNode counting requires special handling with strings, for some reason #44

Open
ExpandingMan opened this issue Jun 25, 2019 · 0 comments

Comments

@ExpandingMan
Copy link
Owner

For most List objects, there are two FieldNode objects in the RecordBatchList: one for the List itself, and one for the underlying values. For some strange reason, the arrow creators decided that strings are special, so if you have an array of strings, there is only a single FieldNode representing the array itself.

The only reason I can think of to do this would be a tiny bit of compression of the metadata. Consistency of the format seems like a horrible price to pay for that.

Anyway, I'm keeping this issue open because I'm worried that this problem that the underlying data of arrays cannot be constructed from knowledge of the underlying data alone, but requires additional metadata, might be an ongoing issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant