Add a weigher to the EntityCache based on approximate entity size #490

eric-maynard · 2024-11-27T09:47:14Z

Description

The EntityCache currently uses a hard-coded limit of 100k entities, which is apparently intended to keep the cache around 100MB based on the assumption that entities are ~1KB.

If that assumption is not true, or if there is not even 100MB of free memory, we should not let the cache use up too much memory. This PR adds a weigher to constrain the cache not by a number of entities, but by the approximate number of bytes inserted into the cache. This value is configurable via a new option.

This PR further adds an option to allow the cache to use soft values to allow GC to clean up entries when memory pressure becomes too great.

snazy · 2024-11-27T19:38:24Z

polaris-core/src/main/java/org/apache/polaris/core/persistence/cache/EntityWeigher.java

+        + value.getEntity().getProperties().length()
+        + value.getEntity().getInternalProperties().length();


The assumption here is wrong. There is no guarantee that one character is only one byte, even with "compact strings" - it's likely off by factor 2.

It's an approximate size. Even if it's off by a factor of 10, that should be okay. A 1GB cache is greatly preferable to an unbounded cache.

Just to confirm my understanding: if the main concern is the difference between a string's length and its actual memory usage, could we address this by multiplying String.length() by an estimated factor, such as 2 or 3? And I agree that, regardless of the exact multiplier, this would help us "bound" the overall size of the cache

I'm totally fine adding some constant multiplier (2?) if that's what we want to do.

I think it won't be possible to choose a constant that's always correct, e.g. if we have an ASCII string, then each char really is 1 byte. So rather than try in futility to make this number the exact bytes in the entity, I wonder if we need to just accept that it's an approximate "size" limit rather than an exact bytes limit.

How about calibrating the weighter using a scientific method, eh?

For example: make a test that keeps adding stuff to the cache and measures actual heap usage. Plot that against the number of entries / number of properties and do statistical regression (linear).

Hey @dimas-b, just wanted to check in again here to see if you might have time this week to investigate this a little further. Let me know if I can help by writing or running any tests here.

Sorry about the delay.

The second run looks pretty good to me. You may want to call System.gc() before taking heap usage measurements.

Also, for this analysis is it preferable to plot Weight (X) against Heap (Y) as a scatter chart (not time series) because we want to establish whether a user can rely on the Weight to be an estimator of Heap usage.

If possible please run multiple scenarios for the objects in the cache (different properties, different chars, lengths, etc...) but put all data on the same chart (as dots).

For the purpose of cache size limiting, I'd say that if Weight is X and Heap is Y, then most of the dots should be close to , but under the Y = X line.

Thanks. Based on the above, here's what I collected. All of the heap size measurements below are taken with System.gc() being called beforehand. The same spreadsheet & branch linked above were used. No multiplier was applied.

This shows weight vs. heap usage for small & large objects (100 characters vs 10k characters):

As you can see, across all 4 trials heap usage stabilizes somewhere. So in that sense the weigher is working very well. It's notable that the exact amount of heap usage we stabilize at varies greatly by entity size. Each entity has a 1000 "weight unit" cost associated with it besides whatever weight it's assigned based on its properties and the multiplier, so it makes sense that when using smaller entities we'll stabilize a bit lower than expected.

The exact point at which we stabilize varies from ~200MB to ~60MB, suggesting a multiplier anywhere in the range of 0.5 and 2.0 could be effective. Indeed, when we perform the same test with a multiplier of 2 we see that all 4 trials stabilize below 100MB.

Focusing in on the time before the heap usage stabilizes, we see that adding a WU of data adds between 0.55b - 1.85b. This points us to the same conclusion -- a multiplier between 0.55 and 1.85 is needed to ensure that a weight target of 100M weight units keeps the cache below 100MB of heap usage.

I chose 3 rather than 2 in this PR as an exceedingly conservative value intended to always keep us below 100MB, as well as to account for transient memory usage. However I am happy to change this value to anything you prefer.

It's good to see a clear linear relationship between weight and heap usage 👍

Did you test include Grant Records inside cached entities? Do we expect those to be prevalent in the cache at all?

I didn't include grant records here, and I only added very simplistic entities with large properties. The 1000 WU overhead is intended to capture the storage needed for non-string fields like grant record foreign keys or a timestamp associated with a table entity.

snazy · 2024-11-27T19:39:04Z

polaris-core/src/main/java/org/apache/polaris/core/persistence/cache/EntityWeigher.java

+  public static final long WEIGHT_PER_MB = 1024 * 1024;
+
+  /* Represents the approximate size of an entity beyond the properties */
+  private static final int APPROXIMATE_ENTITY_OVERHEAD = 1000;


Where does the value 1000 come from?

See this old comment -- the apparent assumption was that entities are generally ~1KB.

This means in the worst case, we will not get a larger number of entries due to this change. It should strictly be smaller.

snazy · 2024-11-27T19:39:37Z

...e/src/testFixtures/java/org/apache/polaris/core/persistence/PolarisTestMetaStoreManager.java

@@ -828,7 +828,7 @@ void dropEntity(List<PolarisEntityCore> catalogPath, PolarisEntityCore entityToD
  }

  /** Grant a privilege to a catalog role */
-  void grantPrivilege(
+  public void grantPrivilege(


The changes here look unrelated?

They are not; the EntityCacheTest was moved into a new package and so these methods were no longer callable from it if they remained package-private

snazy · 2024-11-27T19:43:13Z

polaris-core/src/main/java/org/apache/polaris/core/persistence/cache/EntityCache.java

            .expireAfterAccess(1, TimeUnit.HOURS) // Expire entries after 1 hour of no access
            .removalListener(removalListener) // Set the removal listener
+            .softValues() // Account for memory pressure


Either there's a weigher or there are soft-references, the latter is IMHO a bad choice, because it can likely ruin the efficiency of the eviction.

The docs seem to say you can do both, WDYM?

As for soft values being a bad choice, in my experience this is quite a common practice and the degradation we're likely to see is far less than the degradation we'd see in a situation where heap space gets exhausted and we do not have soft values.

Assuming there is not significant memory pressure, my expectation is that GC should more or less ignore the SoftReference objects.

If there's a specific performance concern that can be borne out in a benchmark, we should address it.

GC behaviour with soft references is very GC impl.-specific, AFAIK. A substantial amount of soft-referenced objects can still cause huge GC overhead (in particular with Parallel GC in my experience). This can easily happen if the weighter is underestimating.

On the other hand, if the weighter is overestimating, that reduces the cache efficiency and wastes memory (allotted for cache, but not actually used for cache).

Yet, if the weighter is accurate and the cache stays within the allotted boundaries, why would we want to use soft-references?

It's true that a substantial amount of soft references can cause GC overhead -- and if you're able to observe that on this branch, I think sharing those results would be helpful. I did not see such a thing in my testing, but I didn't try various GC implementations or anything like that.

More importantly, I think this analysis rests on a flawed assumption here:

memory (allotted for cache

Right now, we allot essentially infinite memory to the cache which is clearly incorrect. We need to bound that memory footprint, and we should also provide a mechanism to release that memory when even that footprint causes memory pressure when combined with everything else running in the JVM.

I do not expect that ~100M is a sufficient memory so as to cause issues, but again if there's some test that shows significant performance degradation due to a cache of this size doing GC, let's reconsider whether soft values make sense.

snazy · 2024-11-27T19:45:39Z

polaris-core/src/main/java/org/apache/polaris/core/persistence/cache/EntityCache.java

    this.byId =
        Caffeine.newBuilder()
-            .maximumSize(100_000) // Set maximum size to 100,000 elements
+            .maximumWeight(100 * EntityWeigher.WEIGHT_PER_MB) // Goal is ~100MB


While we're here - why is this always hard coded and not configurable? Or at least determined based on the actual heap size (which is not as easy as it sounds)?

This is a good callout. My concern with making it configurable is that the limit is close to arbitrary.

As you noted in another comment, we have a "goal" of 100MB but that could really be 200MB or 1MB depending on how eviction happens with soft values and the weights. So while raising the limit definitely gets you a bigger cache, it's quite opaque as to what value you would actually want to set.

Also, the EntityCache is ideally very transparent and not something a typical end user would want to concern themselves with.

In light of the above I kept it hard-coded for now but if you feel strongly that we should make it a featureConfiguration I am comfortable doing so

I think making the maximum configurable is a good thing but may be out-of-scope of this PR. Probably we can do that as a follow-up?

snazy · 2024-12-04T08:23:56Z

Let's defer this change until the bigger CDI refactoring and potential follow-ups have been done.

eric-maynard · 2024-12-04T08:39:49Z

To me, this seems quite unrelated to the CDI refactoring. Clearly, we cannot pause all development on the project due to a potential conflict with another existing PR.

If there's an especially nasty conflict you're worried about here, can you highlight it? Maybe we can try to minimize the potential problem.

snazy · 2024-12-04T10:33:46Z

Sure, nobody says we shall pause everything. But how the cache is used is heavily influenced by the CDI work.

I still have concerns about the heap and CPU pressure in this PR and #433 and how it's persisted. Also, having a weigher meant to limit heap pressure but having it's calculation being largely off is IMO not good. It's also that the design has hard limits, meaning the cache cannot be as effective as it could be - and at the same time there can be an unbounded number of EntityCache instances. IMO the EntityCache needs a redesign. Because of all that, I'm not sure whether it makes sense to add more stuff to it at this point, especially since there's been no Polaris "GA" release yet, so there's no pressure to push things in.

eric-maynard · 2024-12-10T19:32:36Z

@snazy

But how the cache is used is heavily influenced by the CDI work.

Can you clarify this point? To me, this seems like a strict improvement over the current behavior.

Also, having a weigher meant to limit heap pressure but having it's calculation being largely off is IMO not good.

I disagree that it's "largely off" to a degree that matters, which I articulated in a comment above. Let's keep discussing in more detail there if you'd like. For this thread, I would just ask: Is the current behavior better?

It seems to me that cache that is bounded to an imprecise number of bytes is vastly better than a totally unbounded cache.

It's also that the design has hard limits, meaning the cache cannot be as effective as it could be

This is true today. I'm not intending to change this behavior in this PR, only to make a best-effort attempt to limit how large the cache will grow (in bytes). If you feel that making the cache size configurable is imperative I would be happy to review a PR to that effect or to piggyback that functionality here if you prefer.

and at the same time there can be an unbounded number of EntityCache instances.

Today, this is not the case. The number of instances is bounded.

there's been no Polaris "GA" release yet, so there's no pressure to push things in.

I don't fully understand this point. Can you clarify? I am eager to see our first release, but I don't think its delay means we should put off impactful work.

snazy · 2024-12-11T13:05:42Z

Also, having a weigher meant to limit heap pressure but having it's calculation being largely off is IMO not good.

I disagree that it's "largely off" to a degree that matters, which I articulated in a comment above. Let's keep discussing in more detail there if you'd like. For this thread, I would just ask: Is the current behavior better?

2x+ is largely off, no? String.length() does not return the number of bytes.

It seems to me that cache that is bounded to an imprecise number of bytes is vastly better than a totally unbounded cache.

It's not unbounded at this point - 100,000 * 1kB is ~ 1MB - that's acceptable. But the intent of this PR is to add really large and uncompressed object trees.

So for me it seems that the effort has the potential to make Polaris unstable and/or Caffeine's eviction algorithm way less efficient.

there's been no Polaris "GA" release yet, so there's no pressure to push things in.

I don't fully understand this point. Can you clarify? I am eager to see our first release, but I don't think its delay means we should put off impactful work.

It's related to pushing for this change - we still have time to design things properly in Apache Polaris.

eric-maynard · 2024-12-11T18:28:46Z

String.length() does not return the number of bytes.

Nothing in the PR claims that it does. The length of the string does scale linearly with the number of bytes, though, so a doubling of the length limit will ~double the number of bytes in the cache.

It's not unbounded at this point

This is not correct. The cache is currently bounded to 100k objects, but it does not account for their size. The cache's size, in bytes, is currently unbounded.

github-actions · 2025-01-11T01:59:20Z

This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

eric-maynard · 2025-03-14T04:10:15Z

@snazy & @HonahX -- I switched to no soft values by default and blocked the behavior to use them behind a flag. Does this look safe enough to merge now?

HonahX

LGTM! Thanks @eric-maynard

dimas-b · 2025-03-15T01:37:16Z

polaris-core/src/main/java/org/apache/polaris/core/persistence/cache/EntityCache.java

+            .removalListener(removalListener); // Set the removal listener
+
+    if (PolarisConfiguration.loadConfig(BehaviorChangeConfiguration.ENTITY_CACHE_SOFT_VALUES)) {
+      byIdBuilder.softValues();


Just to clarify: What is our expected benefit from using soft values?

In my testing, it improved performance and prevented situations where a large cache lead to OOMs.

Thanks for this feedback! I'm glad to see that it had positive impact in you env 👍

However, my experience with soft references in cache is not so bright and I'd advise caution, especially when Parallel GC is used. It may be able to recover from close-to-OOM conditions, but API response times are likely to suffer due to GC pauses.

Given that the default is off (not using soft values). I'm fine with this PR in general.

polaris-core/src/main/java/org/apache/polaris/core/config/FeatureConfiguration.java

dimas-b · 2025-03-17T16:09:31Z

polaris-core/src/main/java/org/apache/polaris/core/persistence/cache/EntityWeigher.java

+    return APPROXIMATE_ENTITY_OVERHEAD
+        + (value.getEntity().getName().length() * APPROXIMATE_BYTES_PER_CHAR)
+        + (value.getEntity().getProperties().length() * APPROXIMATE_BYTES_PER_CHAR)
+        + (value.getEntity().getInternalProperties().length() * APPROXIMATE_BYTES_PER_CHAR);


The constants in this expression do not "approximate" any actual sizes. I suggest: ENTITY_SIZE_BOUND_HEURISTIC, CALIBRATED_BYTES_PER_CHAR, plus add javadoc about how we arrived at these calibration parameters.

No, they do approximate actual sizes. The testing described above details that approximately APPROXIMATE_BYTES_PER_CHAR of heap pressure are applied for each character in the entity that's being stored.

I think we can leave this "as is" for now. Putting a more general comment in the main thread.

polaris-core/src/main/java/org/apache/polaris/core/persistence/cache/EntityWeigher.java

snazy · 2025-03-18T09:41:24Z

(Just commenting in general)

Background: Nessie's cache keeps serialized representations of the cached object to limit the amount of reachable (from GC roots) Java objects - it also helps to calculate the effective heap usage and literally use "number of bytes" in the weigher. As an optimization, we added an additional SoftReference to the Java object as an optimization. That optimization looked quite solid in tests - but it's been proven to cause issues in (longer running) production systems. This is a prominent example why an excessive amount of (reachable) Java objects is bad.

Generally, there are many factors that influence the behavior of a JVM/GC combination. Naming some:

age of objects / object generation
whether objects are referenced/reachable from only one thread or multiple threads
number of totally reachable objects
kind of reference (hard/soft/weak/phantom)
system load
kind of GC (know which GC is used in all relevant environments) - G1/Shenandoah/Parallel/Z/Serial/...
GC configuration (pro tip: do NOT tune GC unless you really really have to and proven that it works)
number of CPUs / NUMA
hardware (of course)

About testing:

Do not call Runtime.gc() during tests - you don't know what it's really doing
Use tools like jmc, IntelliJ's profiler or other tools to monitor heap usage
Use multi-threaded tests
Push it to the limits / (try to) break it intentionally
Small/concise tests should be JMH benchmarks (it's got a lot of profilers)
Gatling for longer running tests, JVM stats monitored by external tools

dimas-b

I think we should be ok to merge this change. It enables users to limit cache growth on the heap and adjust the limit (if required) based on observed runtime behaviour.

I'm not sure we can achieve a more accurate weigher with the current approach of caching rather complex java objects. Switching to caching serialized objects would allow for a more accurate weigher, but that is probably beyond the scope of this PR.

polaris-core/src/test/java/org/apache/polaris/core/persistence/cache/EntityCacheTest.java

polaris-core/src/main/java/org/apache/polaris/core/config/FeatureConfiguration.java

dimas-b · 2025-04-08T13:48:04Z

polaris-core/src/main/java/org/apache/polaris/core/persistence/cache/EntityWeigher.java

+public class EntityWeigher implements Weigher<Long, ResolvedPolarisEntity> {
+
+  /** The amount of weight that is expected to roughly equate to 1MB of memory usage */
+  public static final long WEIGHT_PER_MB = 1024 * 1024;


Do we need this constant anymore? Why not inline its value in the config default?

To me the constant seems somewhat coupled to the weigher -- i.e. another weigher implementation could treat 1 string character as 1000 weight units, and would have a very different WEIGHT_PER_MB.

So this constant is specifically mean for the config default?

Pretty much yep. I also thought it was a helpful constant to have for anyone trying to read this code rather than us just setting the default to e.g. 104857600 and leaving its interpretation as an exercise for the reader.

On the other hand I can see how it's potentially misleading given the extended discussion around the amount of heap we expect a given weight to use, so I'm OK with yanking the constant if you think it's problematic.

edit: We could also consider renaming EntityWeigher to ApproximateStringSizeEntityWeigher or something more idiomatic. That would make this ApproximateStringSizeEntityWeigher.WEIGHT_PER_MB which would be more explicitly "approximate" and coupled to the weigher implementation.

EntityWeigher SGTM... I believe approximation is implied in its name and API :)

Re: constant, how about renaming it to WEIGHT_LIMIT_DEFAULT and using without extra modifications in the config variable definition?

dimas-b · 2025-04-08T13:52:29Z

polaris-core/src/main/java/org/apache/polaris/core/persistence/cache/EntityWeigher.java

+        + value.getEntity().getProperties().length()
+        + value.getEntity().getInternalProperties().length();


It's good to see a clear linear relationship between weight and heap usage 👍

Did you test include Grant Records inside cached entities? Do we expect those to be prevalent in the cache at all?

dimas-b · 2025-04-08T13:53:35Z

polaris-core/src/main/java/org/apache/polaris/core/persistence/cache/EntityWeigher.java

+    return APPROXIMATE_ENTITY_OVERHEAD
+        + (value.getEntity().getName().length() * APPROXIMATE_BYTES_PER_CHAR)
+        + (value.getEntity().getProperties().length() * APPROXIMATE_BYTES_PER_CHAR)
+        + (value.getEntity().getInternalProperties().length() * APPROXIMATE_BYTES_PER_CHAR);


I think we can leave this "as is" for now. Putting a more general comment in the main thread.

dimas-b · 2025-04-08T23:51:19Z

@snazy : Would you ok with merging this PR given my rationale in the comment above? (Future refactoring is not ruled out, of course).

eric-maynard · 2025-04-11T18:04:20Z

Thanks all! Merging for now, but I think we should revisit the weigher if/when the cache is updated to just take bytes[] rather than a complex Java object, as Dmitri noted above.

eric-maynard added 2 commits November 27, 2024 01:43

initial commit

f0ca27b

autolint

e28c519

eric-maynard requested review from jbonofre, ashvina, RussellSpitzer, snazy, vvcephei, takidau, jackye1995, flyrain, collado-mike, ebyhr and adutra as code owners November 27, 2024 09:47

eric-maynard mentioned this pull request Nov 27, 2024

Support persisting TableMetadata in the metastore #433

Open

5 tasks

snazy reviewed Nov 27, 2024

View reviewed changes

eric-maynard requested a review from snazy December 2, 2024 17:48

github-actions bot added the Stale label Jan 11, 2025

pull main

a7556b9

eric-maynard requested review from dennishuo and dimas-b as code owners January 23, 2025 20:47

resolve conflicts

fd351a1

eric-maynard removed the Stale label Feb 17, 2025

Merge branch 'main' of github.com:apache/polaris into cache-soft-values

d98be9c

eric-maynard added 2 commits March 13, 2025 21:09

switch to behavior change config

78725c5

autolint

93a5de1

HonahX approved these changes Mar 14, 2025

View reviewed changes

eric-maynard added 7 commits March 14, 2025 11:26

Merge branch 'main' of github.com:apache/polaris into cache-soft-values

5efb1c6

pull main

4e9d192

Add multiplier

ce7b2e5

account for name, too

ed42b8c

adjust multiplier

94c3fe3

add config

1a26018

autolint

230c95e

dimas-b reviewed Mar 15, 2025

View reviewed changes

dimas-b reviewed Mar 17, 2025

View reviewed changes

remove old cast

eb0a817

eric-maynard changed the title ~~Use soft values in the EntityCache; approximate entity size~~ Add a weigher to the EntityCache based on approximate entity size Mar 18, 2025

eric-maynard requested a review from dimas-b March 21, 2025 03:59

rebase

8782774

dimas-b reviewed Apr 8, 2025

View reviewed changes

eric-maynard added 3 commits April 8, 2025 11:01

more tests, fixes per review

fc9daee

add precise weight test

f399043

autolint

ca620aa

dimas-b approved these changes Apr 8, 2025

View reviewed changes

github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Apr 8, 2025

sfc-gh-jojiang approved these changes Apr 11, 2025

View reviewed changes

HonahX approved these changes Apr 11, 2025

View reviewed changes

eric-maynard merged commit 510bd72 into apache:main Apr 11, 2025
5 checks passed

github-project-automation bot moved this from Ready to merge to Done in Basic Kanban Board Apr 11, 2025

		+ value.getEntity().getProperties().length()
		+ value.getEntity().getInternalProperties().length();

Add a weigher to the EntityCache based on approximate entity size #490

Add a weigher to the EntityCache based on approximate entity size #490

Conversation

eric-maynard commented Nov 27, 2024 • edited Loading

Description

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eric-maynard Apr 7, 2025 • edited Loading

Choose a reason for hiding this comment

dimas-b Apr 8, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eric-maynard Nov 27, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eric-maynard Nov 27, 2024 • edited Loading

Choose a reason for hiding this comment

dimas-b Feb 21, 2025 • edited Loading

Choose a reason for hiding this comment

eric-maynard Feb 21, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

snazy commented Dec 4, 2024

eric-maynard commented Dec 4, 2024

snazy commented Dec 4, 2024

eric-maynard commented Dec 10, 2024 • edited Loading

snazy commented Dec 11, 2024

eric-maynard commented Dec 11, 2024

github-actions bot commented Jan 11, 2025

eric-maynard commented Mar 14, 2025

HonahX left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

snazy commented Mar 18, 2025

dimas-b left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eric-maynard Apr 9, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dimas-b Apr 8, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dimas-b commented Apr 8, 2025

eric-maynard commented Apr 11, 2025

eric-maynard commented Nov 27, 2024 •

edited

Loading

eric-maynard Apr 7, 2025 •

edited

Loading

dimas-b Apr 8, 2025 •

edited

Loading

eric-maynard Nov 27, 2024 •

edited

Loading

eric-maynard Nov 27, 2024 •

edited

Loading

dimas-b Feb 21, 2025 •

edited

Loading

eric-maynard Feb 21, 2025 •

edited

Loading

eric-maynard commented Dec 10, 2024 •

edited

Loading

eric-maynard Apr 9, 2025 •

edited

Loading

dimas-b Apr 8, 2025 •

edited

Loading