Polaris as catalog for Apache Paimon?

[georgelza]

Hi guys...
Does Polaris support Apache Paimon a OTF... ?
G

[dimas-b]

@georgelza : could you de-abbreviate OTF, please 😅

[georgelza]

Open Table Format . aka Apache Paimon, Apache Fluss, Apache Iceberg.
Open File Format. Apache Parquet, Apache Avro, ORC.

;)

[dimas-b]

TIL 😅

[dimas-b]

ATM, Polaris as a catalog supports Apache Iceberg (implements the Iceberg REST Catalog API).

[georgelza]

thanks, Bummer, liked i and hoped that due to the REST interface it would/could cater for more, via a generic sort of interface.

[dimas-b]

There are some discussions about making a "wider" catalog API... I hope they will materialize in a proposal soon 😅

[dimas-b]

There's also the Generic Table API in Polaris, but it is very different from the Iceberg REST Catalog API. It is basically just an enumeration of table locations with properties ATM.

[dimas-b]

The Polaris Spark Client is able to work with other table formats (e.g. Delta), but it currently does not leverage the catalog for coordinating table changes (i.e. data changes are handled purely on the client side).

[georgelza]

From my view point catalogs is probably the biggest gap I see...
we have multiple table formats, that analyst want to use... a analyst should not care about the OTF.
he wants to interface with a single catalog that can reference/point to multiple data sources/OTF stores.
i.e: Iceberg, Fluss, Paimon, etc.
and be that through spark or Python/Numpy/Panda's log into the catalog and be able to query his back end store, with the catalog telling him what data he has where.

at the moment the lack of a unified catalog ecosystem is creating a new island/silo...

[jbonofre]

That's a good discussion. We are working on a proposal (that will be shared soon) about supporting "new format" in Polaris. I will update this discussion with the discussion on the dev@ mailing list.