Files containing binary data with >=8_388_855 bytes per row written with `arro3` can't be read with `pyarrow`

[jonded94]

Hello,

internally, we wrote an own library that wraps arrow-rs to make it useable from Python.
Such a thing also exists publicly available through arro3 which I used here for some minimal reproducible example:

import pyarrow.parquet
import arro3.io

data = [8388855, 8388924, 8388853, 8388880, 8388876, 8388879]

schema = pyarrow.schema([pyarrow.field("html", pyarrow.binary())])
data = [{"html": b"0" * d} for d in data]

t = pyarrow.Table.from_pylist(data, schema=schema)

path = "/tmp/foo.parquet"
with open(path, "wb") as file:
    for b in t.to_batches():
        arro3.io.write_parquet(b, file, max_row_group_size=len(data) - 3)

reader = pyarrow.parquet.ParquetFile(path)
for i in range(2):
    print(len(reader.read_row_group(i)))

This code writes a bit of dummy binary data through arrow-rs. Reading that with pyarrow results in

  File "pyarrow/_parquet.pyx", line 1655, in pyarrow._parquet.ParquetReader.read_row_group
  File "pyarrow/_parquet.pyx", line 1691, in pyarrow._parquet.ParquetReader.read_row_groups
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
OSError: Couldn't deserialize thrift: No more data to read.
Deserializing page header failed.

---

Observation

• Reading in the same file through arro3 or own internal library wrapping arrow-rs works just fine
• Reading in the same file through duckdb also works just fine
• Reducing the amount of binary data per row slightly leads to the error disappearing (8_388_855 per row or 25_166_565 per row group or more seems to be the problematic amount)
• Issue is reproducible with pyarrow version 18.1.0, 19.0.1 and 20.0.0

I'm highly convinced that the underlying issue is somewhere in either the arrow-rs or arrow codebase, but opened an issue here nevertheless since the only (easy) way to reproduce this issue with publicly available packages is arro3 right now.

If you know any way to find out how to more narrowly identify where this issue could be stemming from, please let me know! I opened other issues here and here.

[kylebarron]

Thanks for the report and for also creating the issues in pyarrow and arrow-rs.

To confirm, you initially found this out in your own Rust code that used arrow-rs directly? I just want to ensure that it's not somehow isolated to arro3. (I can't imagine how it would be isolated to arro3, the bulk of our parquet writing code is just a few lines https://github.com/kylebarron/arro3/blob/main/arro3-io/src/parquet.rs#L250-L260)

In any case I'm happy that arro3 leads to a very easy minimal reproducible example for other contributors (I don't know enough about the Parquet internals myself in either pyarrow or arrow-rs) 🙂

[jonded94]

To confirm, you initially found this out in your own Rust code that used arrow-rs directly?

We basically built an internal equivalent to arro3 (partly because we need some more features than arro3 currently provides and also just because we weren't aware of arro3s existance.. 😅).

So, it's also PyO3 Rust code wrapping arrow-rs code to make it available from Python, very very similar to arro3.

[jonded94]

Ideally, we'd somehow find out whether the file written through the arrow-rs is not conforming to the Thrift Parquet standard, then we'd be sure that the error is in the arrow-rs codebase.. 😬 But I'm really not aware of how to do this 😅

[kylebarron]

A priori I think it's equally likely that the bug is in pyarrow compared to arrow-rs.

partly because we need some more features than arro3 currently provides

You're welcome to create issues here for more features!

[jonded94]

A priori I think it's equally likely that the bug is in pyarrow compared to arrow-rs.

True!

You're welcome to create issues here for more features!

Your PR (#313) actually would implement a good chunk of the features (surely exceeding what we did in some aspects), although the stuff that we have is specifically about interfaces for (very) large Parquet datasets, where we do somewhat opinionated in-process caching and also writing of cache-specific sidecar files to speed up common operations, a bit similar to what Delta Lake does for example (but definitely not as polished).

I'm a bit unsure how well this would generalize for a package that is meant for the general public, but happy to have a talk about this eventually! As soon as we became aware of arro3 and especially the fact that it can be used without pyarrow, we thought a bit about how & which features we could bring upstream, but did not have any capacity for that yet.

[kylebarron]

#313 is essentially ready to go except for writing updated documentation; you're welcome to test out that branch and I'd be curious to see how it works for you.

#313 doesn't attempt to cover Parquet Datasets that have multiple physical files. I've done a small amount of prototyping for that but haven't come up with a PR here for it yet. If you're interested we could collaborate.

[jonded94]

The underlying issue apparently here is that the data page headers are getting larger than 16MiBs: apache/arrow#46404 (comment)

I actually tried to mitigate that with arro3 by setting max_statistics_size, but apparently this parameter is entirely unused: max_statistics_size

I think potentially making that thing actually do something could fix the underlying issue here? 😅 Or expose a way to entirely skip writing the min/max value headers? arrow-rs allows this even on column-level (https://arrow.apache.org/rust/parquet/file/properties/struct.WriterPropertiesBuilder.html#method.set_column_statistics_enabled).

EDIT: I think the more modern / canonical way to do this is set_statistics_truncate_length.

[alamb]

The underlying issue apparently here is that the data page headers are getting larger than 16MiBs: apache/arrow#46404 (comment)

I actually tried to mitigate that with arro3 by setting max_statistics_size, but apparently this parameter is entirely unused: max_statistics_size

I think potentially making that thing actually do something could fix the underlying issue here? 😅 Or expose a way to entirely skip writing the min/max value headers? arrow-rs allows this even on column-level (https://arrow.apache.org/rust/parquet/file/properties/struct.WriterPropertiesBuilder.html#method.set_column_statistics_enabled).

EDIT: I think the more modern / canonical way to do this is set_statistics_truncate_length.

Yes, I will file a ticket to consider truncating the statistics length by default as well

[jonded94]

@kylebarron if you're also already working on this: In our internal library, I wasn't able to produce pyarrow readable parquet files with page-level statistics, even with statistics_truncate_length and column_index_truncate_length set to a very low value.

apache/arrow-rs#7489 (comment)

Only if I explicitly disabled all page-level statistics entirely, the files were readable with pyarrow.

This is really not expected, right? I'm suspecting a bug somewhere in arrow-rs? @alamb

[kylebarron]

I'm not familiar with that myself; let's continue the discussion in apache/arrow-rs#7489 since it doesn't seem to be an arro3 issue.

[kylebarron]

Since this appears to solely be a bug in the underlying Rust libraries, I'll close this and it'll get automatically fixed when we upgrade to the next version of arrow