Add process flag 'heap_growth'

[sverker]

New process flag heap_growth to enable more memory conservative garbage collections.

• normal is the default old behavior. When allocating heap capacity during GC it takes into account the amount of live terms that survived the last GC. A process continuously generating live data will thus grow its heap capacity over time and get less frequent but larger GCs.
• low is the new memory conservative option. The GC maintains a bounded small heap capacity which results in more frequent but smaller GCs.

In both cases, terms that survive two GCs will be moved to the old-heap whose size calculation is not affected by this PR.

heap_growth can be set/read per process with

• process_flag(heap_growth, low | normal)
• process_info(Pid, heap_growth)
• spawn_opt(_, [{heap_growth, low | normal}])

and node wise with

• system_flag(heap_growth, low | normal)
• system_info(heap_growth)
• erl -hg low

[KennethL]

Should this:
spawn_opt(_, [{heap_growth}])
be
spawn_opt(_, [{heap_growth, low | normal}])
??

[lhoguin]

Can we have some context as to why you are adding this feature? In what scenarios will it provide benefits?

[github-actions]

CT Test Results

       3 files     125 suites   37m 20s ⏱️
1 434 tests 1 392 ✔️ 42 💤 0 ❌
1 732 runs  1 673 ✔️ 59 💤 0 ❌

Results for commit 575e9ab.

♻️ This comment has been updated with latest results.

To speed up review, make sure that you have read Contributing to Erlang/OTP and that all checks pass.

See the TESTING and DEVELOPMENT HowTo guides for details about how to run test locally.

Artifacts

• Complete CT logs (Download Logs)
• HTML Documentation (Download HTML Docs)
• Windows Installer

// Erlang/OTP Github Action Bot

[starbelly]

Hey @sverker , this seems terribly useful. If I'm grokking correctly, this is terribly useful in the context of long lived processes which may have to consume lots of messages but do not hold them in state (i.e., proc heap). Is that right?

[sverker]

The idea of this PR was to optimize process heap memory consumption at the expense of CPU load caused by more frequent GCs when a process creates lots of live data.

With the current (normal) GC strategy, we exponentially grow the size of the new-heap based on the amount of terms that survived the last GC. This is done before we know how much data will survive this GC. We assume the process will continue to create lots of live data and we want to give it more heap capacity to do that. A larger heap will prolong the time until next GC and that will also give the data more time to die.

If the process then stops creating live data and enters a more passive period, then it might get "stuck" with a large inflated new-heap with lots of capacity that it does not use.

The low heap growth option introduced by this PR will disable that exponential growth for anticipated heap use and instead lets the process work with a more or less constant sized new-heap.

We had a real use case where memory consumption was a high priority concern. However, they gained almost nothing from this PR when tested. That's why it has not been merged.

[starbelly]

The idea of this PR was to optimize process heap memory consumption at the expense of CPU load caused by more frequent GCs when a process creates lots of live data.

With the current (normal) GC strategy, we exponentially grow the size of the new-heap based on the amount of terms that survived the last GC. This is done before we know how much data will survive this GC. We assume the process will continue to create lots of live data and we want to give it more heap capacity to do that. A larger heap will prolong the time until next GC and that will also give the data more time to die.

If the process then stops creating live data and enters a more passive period, then it might get "stuck" with a large inflated new-heap with lots of capacity that it does not use.

The low heap growth option introduced by this PR will disable that exponential growth for anticipated heap use and instead lets the process work with a more or less constant sized new-heap.

Precisely. We have processes in our system at work that suffer because of this, such that we have to set fullsweep_after to a very low value (or zero in some cases), in addition to some more expensive work arounds. This is mainly per passing complex terms to processes and then putting them in ets. We are actually going to make changes to one such set of procs that gets around all this via passing and storing serialized binaries via term_to_binary/1 and friends.

That said, we have some other procs that could maybe benefit from this.

We had a real use case where memory consumption was a high priority concern. However, they gained almost nothing from this PR when tested. That's why it has not been merged.

I'm happy to try a patch out if there's still interest in this.

[sverker]

I'm happy to try a patch out if there's still interest in this.

This branch is currently based on OTP-24.3, what do you need? You can always try merge forward to newer versions. I don't think the merge conflicts you might get will be particularly hard to solve.

[starbelly]

I'm happy to try a patch out if there's still interest in this.

This branch is currently based on OTP-24.3, what do you need? You can always try merge forward to newer versions. I don't think the merge conflicts you might get will be particularly hard to solve.

I can merge it forward I'm quite sure, I'll report back soon.

[starbelly]

Hey @sverker it took me a minute on this, but I can say I saw no substantial improvement.