Remove pyarrow as required dependency, relying on Arrow PyCapsule Interface

[kylebarron]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

PyArrow is a massive dependency. Unpacked, it tends to be >100MB in size, and, until the latest versions (I think?) also required numpy as its own non-optional dependency.

It's also, in effect the only current dependency

datafusion-python/pyproject.toml

Line 46 in f0bbad7

dependencies = ["pyarrow>=11.0.0", "typing-extensions;python_version<'3.13'"]

It would be great if we could remove it, and that would greatly lessen the minimal environment size for datafusion python.

Many other Python Arrow libraries implement the PyCapsule Interface, so the user can use nanoarrow, arro3, Polars, DuckDB, etc, or pyarrow. Whatever is best for them.

Describe the solution you'd like

The Arrow PyCapsule Interface is a lightweight, decentralized protocol for sharing Arrow data between Python libraries. We already implement the PyCapsule Interface, so it's just a matter of removing places where we hard-code use of pyarrow.

Describe alternatives you've considered

Keep pyarrow dependency.

Additional context

[ion-elgreco]

I agree with removing pyarrow as a dependency. We have to consider that there are users out there that want to create small containers but do data operations in them, if you pull in pyarrow which is so heavyweight that complicates it a lot.

In delta-rs I've already done this, delta-io/delta-rs#3459 some time ago, which was very easy due to Arro3. (Thanks @kylebarron :D)

As I am thinking about it, @kylebarron a pointer might make more sense indeed since it's more to be interopable between other libs, if you want to analyze the data it can be done directly in datafusion.

[kylebarron]

We already have implementations of the PyCapsule interface, and we're improving it in #1222 (thanks to @kosiew ). It's just methods like to_batches that are just syntactic sugar for calling pa.table(datafusion_df). What should we do about them?

[timsaucer]

The more I think about it, I think this is a good idea. We just need someone who wants to dive in and take on the work. I have too many other things in the air right now to tackle myself.

[ion-elgreco]

Perhaps @H0TB0X420 would like to move this forward?

[H0TB0X420]

Sure! I can work on moving this forward. Just to confirm, we would like to remove places where pyarrow is explicitly used and implement the PyCapsule Interface?

[kylebarron]

First, let's establish a list of all the places where pyarrow is used. Where is it used as an input parameter and where is it used as a return type?

Wherever pyarrow is used as an input parameter is pretty trivially replaced by the PyCapsule Interface, because the interface can accept pyarrow objects without us needing to depend on pyarrow as a dependency.

Wherever it's used as a return type is a little more tricky, because we need to return something.

[ion-elgreco]

First, let's establish a list of all the places where pyarrow is used. Where is it used as an input parameter and where is it used as a return type?

Wherever pyarrow is used as an input parameter is pretty trivially replaced by the PyCapsule Interface, because the interface can accept pyarrow objects without us needing to depend on pyarrow as a dependency.

Wherever it's used as a return type is a little more tricky, because we need to return something.

Where we return a pyarrow type we either need to remove it and implement on the Datafusion type directly, for example for the schema we just need to have it be a Datafusion.schema that's how I mentioned it in the initial issue when I had time :p