Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: more backend capabilities for heapdump #233

Closed
wants to merge 4 commits into from
Closed

feat: more backend capabilities for heapdump #233

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
2c09ee2
feat: more backend capabilities for heapdump
y1yang0 Sep 13, 2023
a57f066
review from Jason
y1yang0 Sep 13, 2023
781b3c7
fix: extra whitespace
y1yang0 Sep 22, 2023
4dddb87
fix test
y1yang0 Jan 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions analysis/heap-dump/api/src/main/java/org/eclipse/jifa/hda/api/HeapDumpAnalyzer.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,8 @@ public interface HeapDumpAnalyzer {
@ApiMeta(aliases = "inspector.value")
String getObjectValue(int objectId);

@ApiMeta(aliases = "vmBoard.summary")
VMBoard.Summary getSummaryOfVMBoard();
@ApiMeta(aliases = "classLoaderExplorer.summary")
ClassLoader.Summary getSummaryOfClassLoaders();

Expand All @@ -71,10 +73,13 @@ PageView<ClassLoader.Item> getChildrenOfClassLoader(int classLoaderId,
PageView<UnreachableObject.Item> getUnreachableObjects(int page, int pageSize);

@ApiMeta(aliases = "directByteBuffer.summary")
DirectByteBuffer.Summary getSummaryOfDirectByteBuffers();
DirectByteBuffer.Summary getSummaryOfDirectByteBuffers(String mode);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about making 'mode' optional?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See queryDirectByteBufferData, default to query all byte buffers.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

optional here means that front-end doesn't have to pass the parameter.
see @ApiParameterMeta.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about making mode as an Enum type like 'grouping'?


@ApiMeta(aliases = "directByteBuffer.records")
PageView<DirectByteBuffer.Item> getDirectByteBuffers(int page, int pageSize);
PageView<DirectByteBuffer.Item> getDirectByteBuffers(String mode, int page, int pageSize);

@ApiMeta(aliases = "listObjects")
PageView<JavaObject> getListObjects(String objectLabel, int objectId, int type, int page, int pageSize);

@ApiMeta(aliases = "outbounds")
PageView<JavaObject> getOutboundOfObject(int objectId, int page, int pageSize);
Expand Down
9 changes: 9 additions & 0 deletions analysis/heap-dump/api/src/main/java/org/eclipse/jifa/hda/api/Model.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -537,6 +537,15 @@ public TreeResult(PageView<JavaObject> pv) {

}

interface VMBoard {
@Data
class Summary{
public String requestId;
public List<String> vmOptions;
public List<String> gc;
}
}

interface GCRootPath {

List<String> EXCLUDES = Arrays.asList("java.lang.ref.WeakReference:referent",
Expand Down
28 changes: 17 additions & 11 deletions analysis/heap-dump/impl/src/main/java/org/eclipse/jifa/hda/impl/AnalysisContext.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,7 @@
import org.eclipse.mat.snapshot.ISnapshot;

import java.lang.ref.SoftReference;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.*;

public class AnalysisContext {

Expand Down Expand Up @@ -54,14 +51,23 @@ static class ClassLoaderExplorerData {
}

static class DirectByteBufferData {
static final String OQL =
"SELECT s.@displayName as label, s.position as position, s.limit as limit, s.capacity as " +
"capacity FROM java.nio.DirectByteBuffer s where s.cleaner != null";
static final String JDK_MANAGED_BUFFER_OQL =
"SELECT s.@displayName as label, s.position as position, s.limit as limit, s.capacity as " +
"capacity FROM java.nio.DirectByteBuffer s where s.cleaner != null";
static final String JNI_ALLOCATED_BUFFER_OQL =
"SELECT s.@displayName as label, s.position as position, s.limit as limit, s.capacity as " +
"capacity FROM java.nio.DirectByteBuffer s where s.cleaner = null and s.att = null";
static final String ALL_BUFFER_OQL =
"SELECT s.@displayName as label, s.position as position, s.limit as limit, s.capacity as " +
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this break the existing default behaviour now?

Previously the default was equivalent to JDK_MANAGED_BUFFER_OQL, but below we default to ALL. I'm not sure which is intended.

Copy link
Contributor Author

@y1yang0 y1yang0 Sep 13, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The frontend will be synced later by @D-D-H

The existing default behavior is problematic because actual direct buffer(and related off-heap memory) is less than presented, that why we provide these fine-grained options to inspect direct buffer.

"capacity FROM java.nio.DirectByteBuffer s";

static final Map<String, Object> JDK_MANAGED_BUFFER_ARGS =
Collections.singletonMap("queryString", JDK_MANAGED_BUFFER_OQL);
static final Map<String, Object> JNI_ALLOC_BUFFER_ARGS =
Collections.singletonMap("queryString", JNI_ALLOCATED_BUFFER_OQL);
static final Map<String, Object> ALL_BUFFER_ARGS =
Collections.singletonMap("queryString", ALL_BUFFER_OQL);

static final Map<String, Object> ARGS = new HashMap<>(1);
static {
ARGS.put("queryString", OQL);
}

RefinedTable resultContext;

Expand Down
176 changes: 136 additions & 40 deletions analysis/heap-dump/impl/src/main/java/org/eclipse/jifa/hda/impl/HeapDumpAnalyzerImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -496,6 +496,40 @@ private ClassLoaderExplorerData queryClassLoader(AnalysisContext context) throws
}
}

@Override
public Model.VMBoard.Summary getSummaryOfVMBoard() {
Map<String, Object> queryVMOptions = new HashMap<>();
Map<String, Object> queryGc = new HashMap<>();
queryVMOptions.put("queryString", "SELECT toString(x) AS VMOptions FROM OBJECTS (SELECT OBJECTS s.vmArgs.list.a[0:-1] FROM sun.management.VMManagementImpl s) x");
queryGc.put("queryString", "SELECT toString(g.name) AS GC FROM sun.management.GarbageCollectorImpl g");
return $(() -> {
List<String> options = new ArrayList<>();
List<String> gc = new ArrayList<>();
IResultTable result = null;
IResult r = queryByCommand(context, "oql", queryVMOptions);
if (r instanceof IResultTable) {
// In case we get IResultText, i.e. empty result
result = (IResultTable) r;
for (int i = 0; i < result.getRowCount(); i++) {
Object row = result.getRow(i);
options.add((String) result.getColumnValue(row, 0));
}
}
r = queryByCommand(context, "oql", queryGc);
if (r instanceof IResultTable) {
result = (IResultTable) r;
for (int i = 0; i < result.getRowCount(); i++) {
Object row = result.getRow(i);
gc.add((String) result.getColumnValue(row, 0));
}
}
Model.VMBoard.Summary vmBoard = new Model.VMBoard.Summary();
vmBoard.setVmOptions(options);
vmBoard.setGc(gc);
return vmBoard;
});
}

@Override
public Model.ClassLoader.Summary getSummaryOfClassLoaders() {
return $(() -> {
Expand Down Expand Up @@ -603,56 +637,55 @@ public PageView<UnreachableObject.Item> getUnreachableObjects(int page, int page
}

private DirectByteBufferData queryDirectByteBufferData(
AnalysisContext context) throws SnapshotException {
DirectByteBufferData data = context.directByteBufferData.get();
if (data != null) {
return data;
AnalysisContext context, String mode) throws SnapshotException {
DirectByteBufferData data = new DirectByteBufferData();
final Map<String, Object> queryArg;
switch (mode) {
case "jniAlloc":
queryArg = DirectByteBufferData.JNI_ALLOC_BUFFER_ARGS;
break;
case "jvmManaged":
queryArg = DirectByteBufferData.JDK_MANAGED_BUFFER_ARGS;
break;
case "all":
default:
queryArg = DirectByteBufferData.ALL_BUFFER_ARGS;
break;
}

//noinspection SynchronizationOnLocalVariableOrMethodParameter
synchronized (context) {
data = context.directByteBufferData.get();
if (data != null) {
return data;
IResult result = queryByCommand(context, "oql", queryArg);
IResultTable table;
if (result instanceof IResultTable) {
table = (IResultTable) result;

RefinedResultBuilder builder =
new RefinedResultBuilder(new SnapshotQueryContext(context.snapshot), table);
builder.setSortOrder(3, Column.SortDirection.DESC);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is sort order important for summary?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we want to sort by retaiend heap

data.resultContext = (RefinedTable) builder.build();
DirectByteBuffer.Summary summary = new DirectByteBuffer.Summary();
summary.totalSize = data.resultContext.getRowCount();

for (int i = 0; i < summary.totalSize; i++) {
Object row = data.resultContext.getRow(i);
summary.position += data.position(row);
summary.limit += data.limit(row);
summary.capacity += data.capacity(row);
}

data = new DirectByteBufferData();
IResult result = queryByCommand(context, "oql", DirectByteBufferData.ARGS);
IResultTable table;
if (result instanceof IResultTable) {
table = (IResultTable) result;

RefinedResultBuilder builder =
new RefinedResultBuilder(new SnapshotQueryContext(context.snapshot), table);
builder.setSortOrder(3, Column.SortDirection.DESC);
data.resultContext = (RefinedTable) builder.build();
DirectByteBuffer.Summary summary = new DirectByteBuffer.Summary();
summary.totalSize = data.resultContext.getRowCount();

for (int i = 0; i < summary.totalSize; i++) {
Object row = data.resultContext.getRow(i);
summary.position += data.position(row);
summary.limit += data.limit(row);
summary.capacity += data.capacity(row);
}
data.summary = summary;
} else {
data.summary = new DirectByteBuffer.Summary();
}
context.directByteBufferData = new SoftReference<>(data);
return data;
data.summary = summary;
} else {
data.summary = new DirectByteBuffer.Summary();
}
return data;
}

@Override
public DirectByteBuffer.Summary getSummaryOfDirectByteBuffers() {
return $(() -> queryDirectByteBufferData(context).summary);
public DirectByteBuffer.Summary getSummaryOfDirectByteBuffers(String mode) {
return $(() -> queryDirectByteBufferData(context, mode).summary);
}

@Override
public PageView<DirectByteBuffer.Item> getDirectByteBuffers(int page, int pageSize) {
public PageView<DirectByteBuffer.Item> getDirectByteBuffers(String mode, int page, int pageSize) {
return $(() -> {
DirectByteBufferData data = queryDirectByteBufferData(context);
DirectByteBufferData data = queryDirectByteBufferData(context, mode);
RefinedTable resultContext = data.resultContext;
return PageViewBuilder.build(new PageViewBuilder.Callback<Object>() {
@Override
Expand Down Expand Up @@ -702,6 +735,69 @@ private PageView<JavaObject> queryIOBoundsOfObject(AnalysisContext context, int
});
}

@Override
public PageView<JavaObject> getListObjects(String objectLabel, int objectId, int type, int page, int pageSize) {
Map<String, Object> args = new HashMap<>();
if (type == JavaObject.ARRAY_TYPE || type == JavaObject.NORMAL_TYPE) {
objectLabel = String.valueOf(objectId);
}
String query = "list_objects " + objectLabel;
if (type == Model.Histogram.ItemType.SUPER_CLASS) {
query += " -include_subclasses";
}
if (type == Model.Histogram.ItemType.CLASS_LOADER) {
query = "oql";
String oql = "SELECT * FROM (SELECT * FROM java.lang.Class c WHERE c implements org.eclipse.mat.snapshot.model.IClass and c.@classLoaderId = " + objectId + ")";
args.put("queryString", oql);
}
String finalQuery = query;

return $(() -> {
if(type == Model.Histogram.ItemType.PACKAGE){
IResult result = queryByCommand(context, "histogram -groupBy BY_PACKAGE");
Histogram.PackageTree pt = (Histogram.PackageTree) result;
Object targetParentNode = new ExoticTreeFinder(pt)
.setGetChildrenCallback(node -> {
Map<String, ?> subPackages = ReflectionUtil.getFieldValueOrNull(node, "subPackages");
if (subPackages != null) {
return new ArrayList<>(subPackages.values());
} else {
return null;
}
})
.setPredicate((theTree, theNode) -> {
if (!(theNode instanceof XClassHistogramRecord)) {
try {
java.lang.reflect.Field
field = theNode.getClass().getSuperclass().getDeclaredField("label");
field.setAccessible(true);
String labelName = (String) field.get(theNode);
return labelName.hashCode();
} catch (Throwable e) {
e.printStackTrace();
}
}
return null;
})
.findTargetParentNode(objectId);
IContextObject c = pt.getContext(targetParentNode);
if (c instanceof IContextObjectSet) {
int[] objectIds = ((IContextObjectSet) c).getObjectIds();
return PageViewBuilder.build(objectIds, new PagingRequest(page, pageSize), this::getObjectInfo);
}
return PageView.empty();
}else{
IResultTree tree = queryByCommand(context, finalQuery, args);
List<?> objectIds = tree.getElements();

return PageViewBuilder.build(objectIds, new PagingRequest(page, pageSize), node -> {
int id = tree.getContext(node).getObjectId();
return getObjectInfo(id);
});
}
});
}

@Override
public PageView<JavaObject> getOutboundOfObject(int objectId, int page, int pageSize) {
return $(() -> queryIOBoundsOfObject(context, objectId, page, pageSize, true));
Expand Down
4 changes: 2 additions & 2 deletions ...ysis/heap-dump/impl/src/test/java/org/eclipse/jifa/hda/impl/TestHeapDumpAnalyzerImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -134,12 +134,12 @@ public void testGetUnreachableObjects() {

@Test
public void testGetSummaryOfDirectByteBuffers() {
ANALYZER.getSummaryOfDirectByteBuffers();
ANALYZER.getSummaryOfDirectByteBuffers("all");
}

@Test
public void testGetDirectByteBuffers() {
ANALYZER.getDirectByteBuffers(1, 10);
ANALYZER.getDirectByteBuffers("all", 1, 10);
}

@Test
Expand Down