[FEATURE] Support for Iceberg branches and tags

[quodlibetor]

Iceberg has branches as a core part of the data model¹ and the Nessie catalog implements them both through REST parameters², specialized sql syntax³, and via spark properties like spark.sql.catalog.*.ref⁴.

I've tried all of the above techniques, and I haven't been able to figure out a way to connect directly to a branch or query a branch from DuckDB. I tried to find issues or PRs relating to them, but the closest I could find was duckdb/duckdb#17986.

My company uses branches heavily, and it would be really nice to be able to interact with them easily via DuckDB.

(In case folks come here wanting the same thing, currently I've got a collection of horrible shell scripts that query the Nessie REST API and generate SQL statements that look like CREATE VIEW <branch>_<table> AS SELECT * FROM iceberg_scan('s3://path/to/metadata/file.json'), and it works okay for reading snapshots.)

Footnotes

1. https://iceberg.apache.org/docs/latest/branching/ ↩

2. https://projectnessie.org/guides/iceberg-rest/#example-iceberg-rest-uri-parameters ↩

3. https://projectnessie.org/guides/iceberg-rest/#time-travel ↩

4. https://projectnessie.org/iceberg/spark/#configuration ↩

[Tishj]

I imagine read support should be trivial, but write support isn't something I'm willing to commit to yet

[Tishj]

But you might be disappointed because I'm not implementing nessie-specific endpoints.
Branches/tags should be discoverably through returned table metadata, and Nessie doesn't do that

[quodlibetor]

I imagine read support should be trivial, but write support isn't something I'm willing to commit to yet

For me I'm mostly interested in duckdb for exploration, so this is fine. I have lots of long-lived branches with real data that I need to look at.

Branches/tags should be discoverably through returned table metadata, and Nessie doesn't do that

I'm not sure exactly what this means, but I do know exact branch names -- I don't need duckdb to discover them for me. So if I can provide a branch name as a constant (like SELECT * FROM table AS OF my_branch or according to these docs SELECT * FROM table.branch_my_branch) that is all I need.

[Tishj]

Yes, but Nessie deliberately doesn't give us that information:

To retain Nessie’s consistency and cross-branch/tag isolation guarantees, we have deliberately chosen to only return the state of a table or view as a single snapshot in Iceberg.

Their returned metadata is completely bare, which is why we have had to disable all time travel tests on the Nessie catalog in our CI.

They require these parameters to be passed to the REST endpoint in a way that doesn't conform to the spec.
I appreciate their commitment to catalog-wide transactions, but if they don't go through the proper channels to add this to the spec, I'm not going to support it.

[quodlibetor]

Oh I understand now, thank you that's completely reasonable.

[Ferdinanddb]

but write support isn't something I'm willing to commit to yet

@Tishj May I ask you why?

From what I know and can say: Iceberg is advertising for what they call the WAP (Write Audit Publish) workflow, where the point is to:

• Write to an audit branch,
• Audit the audit branch with data validation checks,
• Promote/Publish the audit branch to the main branch (via fast-forward).

I am using that WAP workflow everywhere now, as it is very convenient to validate changes before promoting them (let's say validate that new data are compliant with what I expect to receive), so being able to write to an audit branch (and read from it) via DuckDB would be awesome!

Edit: (FYI, I am not using Nessie on my end; I use the GCP Lakehouse catalog, which is compliant with the Iceberg REST catalog spec, from what I understand!)

[quodlibetor]

I have also been using Nessie via its support for the Iceberg REST catalog. I don't know how complete or correct Nessie's support for the spec is because I haven't used any other catalogs which support it -- I could just be using Nessie-isms everywhere.

If it'd help you to make progress on this I'd be happy to do some investigation if there's something you'd like to know.