Support persisting TableMetadata in the metastore

[eric-maynard]

Description

This adds a new flag METADATA_CACHE_MAX_BYTES which allows the catalog to store table metadata in the metastore and vend it from there when loadTable is called.

Entries are cached based on the metadata location. Currently, the entire metadata.json content is cached.

---

Features not included in this PR:

1. Support for updating the cache when a table is updated
2. Support for invalidating cache entries in the background, rather than waiting for loadTable to be called
3. Structured storage for table metadata

There is partial support for (1) here and I want to extend it, but the goal is to structure things in a way that will allow us to implement (2) and (3) in the future as well.

Performance

I added a new benchmark here and collected results with and without caching enabled. For these results, I used a very narrow distribution that repeatedly hit the same few tables.

Without metadata persistence:

With metadata persistence:

How Has This Been Tested?

Existing tests vend table metadata correctly when caching is enabled.

Added a small test in BasePolarisCatalogTest to cover the basic semantics of caching

Manual testing with eclipselink -- I observed the entities getting created in Postgres and saw large metadata being cached:

db=# select length(internalproperties), substring(internalproperties, 1, 1000) from entities where id = 152;
...
 768691 | {"metadata_location":"file:/tmp/quickstart/ns/tn1731005976265/metadata/00000-e77a2576-5efa-4b7a-b948-121813d713f8.metadata.json","content":"{\"format ...

With MySQL, small metadata is persisted:

mysql> SELECT length(internalproperties), substring(internalproperties, 1, 1000) FROM ENTITIES WHERE id = (SELECT MAX(id) FROM ENTITIES WHERE typecode = 10);
. . .
8159 | {"metadata_location":"file:/tmp/quickstart/ns/t2/metadata/00000-64f975bd-c3a8-4069-bb56-f282003e9157.metadata.json","content":"{\"format-version\"

However large metadata may cause internalproperties to exceed the size limit and nothing will be cached. Calls still return safely.

[RussellSpitzer]

I think this is generally going in the right direction and I'm excited to see this kind of development. My one big ask is to split out the StringLoadingResponse from the actual caching here into another PR. At the moment it's a little bit large although I think both changes are going to be good for the project. It would at least be easier for my brain :)

[eric-maynard]

Sounds good @RussellSpitzer, thanks for taking a look. I will unwind the changes related to directly serving the metadata.json content from persistence for now.

[eric-maynard]

Per @RussellSpitzer's review I've removed references to StringLoadTableResponse and the MetadataJson type used to build it. I did leave some related code changes in place, such as a change to work with RESTResponse instead of LoadTableResponse, to make reapplying later easier.

[adnanhemani]

I was looking at updateTable in IcebergCatalogHandler.java - this also returns back LoadTableResponses and calls load table in CatalogHandlerUtils. Is this also a potential call that could be optimized using this additional persistence?

[dimas-b]

This looks like a huge change (conceptually). Could you open a dev ML thread on this? I did not review in full, but I believe some aspects would be nice to discuss by email (GH comments are not always convenient).

[flyrain]

Not a blocker: It's reasonable to turn it off by default. Do we have a recommended size so that users/admins could use it without additional evaluation work?

[pingtimeout]

I can see multiple occurrences of this pattern in the PR: LOGGER.debug(String.format(...)). That's going to be a problem given that, no matter whether debug logs are enabled, the log message will be computed and allocated on the heap. Could you switch to the placeholders pattern?

logger.debug("Will not cache metadata for {}; metadata above the limit of {} bytes", tableLikeEntity.getTableIdentifier(), maxBytesToCache);

[RussellSpitzer]

A Checkstyle rule we have on Apache Iceberg for this,

        <module name="RegexpSinglelineJava">
            <property name="format" value="(?i)log(ger)?\.(debug|info|warn|error)\(.*%[sd]"/>
            <property name="message" value="SLF4J loggers support '{}' style formatting."/>
            <property name="ignoreComments" value="true"/>
        </module>

[pingtimeout]

Shouldn't this place use the constant APPROXIMATE_BYTES_PER_CHAR instead of * 2?