updateTable using glue with a large schema fails

[vrecan]

Apache Iceberg version

main (development)

Please describe the bug 🐞

When using glue and a large message schema we get this error on every updateCatalog

Glue: UpdateTable, https response error StatusCode: 400, RequestID: 21d099dd-a972-488f-b3ab-7a6a2a1a4c35, InvalidInputException: Payload size of request exceeded limit

I believe it's because constructTableInput always includes all schema columns via schemasToGlueColumns(staged.Metadata()) on every UpdateTable call. We have roughly 3600 fields. This is by design because we use these records to write to elastic search with the "elastic common schema" format which has a lot of fields. A usual message will only have between 20-50 fields set but all fields exist in the schema.

Would it make sense to be able to avoid having the schema in updateTable? I believe it's only used as metadata used in athena, the real schema would still exist in the metadata and parquet files?

[laskoviymishka]

This is a valid issue. The root cause is that constructTableInput always serializes all schema columns (current + all historical) into StorageDescriptor.Columns on every create/update call. For schemas with thousands of fields this reliably hits Glue's request payload limit.

This has been discussed in the Java implementation before:

• AWS: provide option to hide old fields in Glue table iceberg#7584 — request to hide non-current fields in Glue (May 2023)
• Add optional Glue Schema configuration to exclude Non-Current Fields iceberg#11334 — PR attempting glue.non-current-fields-disabled option (Nov 2024, not merged)
• AWS: Add parameter of excluding non-current fields in Glue iceberg#12664 — follow-up PR (Jun 2025, closed due to inactivity)

I made fix #769 via a new glue.schema-columns property (default true) — setting it to false omits columns from the StorageDescriptor entirely, keeping the table readable since the true schema lives in the metadata JSON.

Worth noting: both pyiceberg and iceberg-rust have the same gap — neither exposes any property to opt out of writing columns to the StorageDescriptor. It would be worth reviving the stalled Java PRs and filing symmetric issues/PRs across all implementations to keep behavior consistent.