Skip to content

Commit

Permalink
feat(java): avoid big object graph cause buffer take up too much memo…
Browse files Browse the repository at this point in the history
…ry (#1397)

There has been cases where Fury use too much memory:

![image](https://github.com/apache/incubator-fury/assets/12445254/b5e9d0cb-ea50-4615-a851-d4cf0ca9e594)

This is caused by Fury held buffer. Fury holds a MemoryBuffer and write
to it when users doesn't provide its own buffer. This held buffer can
save memory copy since we don't have to grow buffer every time. But Fury
never clear this buffer. If a big object tree is passed, the held buffer
will take much memory even later object tree is small.

This PR fix it by reset buffer when it's lager than 128Kb.
  • Loading branch information
chaokunyang authored Mar 6, 2024
1 parent 0c4bdef commit 01f1b66
Showing 1 changed file with 9 additions and 0 deletions.
9 changes: 9 additions & 0 deletions java/fury-core/src/main/java/org/apache/fury/Fury.java
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,7 @@ public final class Fury implements BaseFury {
private static final byte isCrossLanguageFlag = 1 << 2;
private static final byte isOutOfBandFlag = 1 << 3;
private static final boolean isLittleEndian = ByteOrder.nativeOrder() == ByteOrder.LITTLE_ENDIAN;
private static final int BUFFER_SIZE_LIMIT = 128 * 1024;

private final Config config;
private final boolean refTracking;
Expand Down Expand Up @@ -1217,6 +1218,10 @@ public void resetWrite() {
nativeObjects.clear();
bufferCallback = null;
depth = 0;
MemoryBuffer buf = buffer;
if (buf != null && buf.size() > BUFFER_SIZE_LIMIT) {
buffer = MemoryBuffer.newHeapBuffer(BUFFER_SIZE_LIMIT);
}
}

public void resetRead() {
Expand All @@ -1227,6 +1232,10 @@ public void resetRead() {
nativeObjects.clear();
peerOutOfBandEnabled = false;
depth = 0;
MemoryBuffer buf = buffer;
if (buf != null && buf.size() > BUFFER_SIZE_LIMIT) {
buffer = MemoryBuffer.newHeapBuffer(BUFFER_SIZE_LIMIT);
}
}

private void checkDepthForSerialization() {
Expand Down

0 comments on commit 01f1b66

Please sign in to comment.