Support for PyArrow parquet partitioning

[bsprenger]

Hi all! 😄 Is there support for loading parquet datasets with partitioning as described in the arrow docs?

Particularly, when I use load_dataset(), it currently seems not to be able to reconstruct the partition columns as described in the doc. For example, if I create a folder structure:

dataset_name/
  year=2007/
    month=01/
       data0.parquet
       data1.parquet
       ...
    month=02/
       data0.parquet
       data1.parquet
       ...
    month=03/
    ...
  year=2008/
    month=01/
    ...
  ...

using Pyarrow datasets I can load this dataset, specify the partition columns, and it will automatically add the year, month etc as columns, but I don't see a way to do this through load_dataset (which ignores the key=value folder structure). Does someone know if there's a way around this?

Thanks so much! 😄