Skip to content

Commit

Permalink
Simplify profile stack trace representation
Browse files Browse the repository at this point in the history
- Introduce a first-class Stack message type and lookup table.
- Replace location index range based stack trace encoding on Sample with
  a single stack_index reference.
- Remove the location_indices lookup table.

The primary motivation is laying the ground work for [timestamp based
profiling][timestamp proposal] where the same stack trace needs to be
referenced much more frequently compared to aggregation based on low
cardinality attributes.

Timestamp based profiling is also expected to be used with the the
upcoming [Off-CPU profiling][off-cpu pr] feature in the eBPF profiler.
Off-CPU stack traces have a different distribution compared to CPU
samples. In particular stack traces are much more repetitive because
they only occur at call sites such as syscalls. For the same reason it
is also uncommon to see a stack trace are a root-prefix of a previously
observed stack trace.

We might need to revisit the previous [previous benchmarks][benchmarks]
to confirm these claims.

The secondary motivation is simplicitly. Arguably the proposed change
here will make it easier to write exporters, processors as well as
receivers.

It seems like we had rough consensus around this change in previous SIG
meetings, and it seems like a good incremental step to make progress on
the timestamp proposal.

[timestamp proposal]: #594
[off-cpu pr]: open-telemetry/opentelemetry-ebpf-profiler#196
[benchmarks]: https://docs.google.com/spreadsheets/d/1Q-6MlegV8xLYdz5WD5iPxQU2tsfodX1-CDV1WeGzyQ0/edit?gid=2069300294#gid=2069300294
  • Loading branch information
felixge committed Jan 9, 2025
1 parent c4214b8 commit 849dc40
Showing 1 changed file with 18 additions and 15 deletions.
33 changes: 18 additions & 15 deletions opentelemetry/proto/profiles/v1development/profiles.proto
Original file line number Diff line number Diff line change
Expand Up @@ -163,24 +163,24 @@ message ScopeProfiles {
// - On-disk, the serialized proto must be gzip-compressed.
//
// - The profile is represented as a set of samples, where each sample
// references a sequence of locations, and where each location belongs
// references a stack trace which is a list of locations, each belonging
// to a mapping.
// - There is a N->1 relationship from sample.location_id entries to
// locations. For every sample.location_id entry there must be a
// - There is a N->1 relationship from Stack.location_indices entries to
// locations. For every Stack.location_indices entry there must be a
// unique Location with that index.
// - There is an optional N->1 relationship from locations to
// mappings. For every nonzero Location.mapping_id there must be a
// unique Mapping with that index.

// Represents a complete profile, including sample types, samples,
// mappings to binaries, locations, functions, string table, and additional metadata.
// It modifies and annotates pprof Profile with OpenTelemetry specific fields.
// Represents a complete profile, including sample types, samples, mappings to
// binaries, stacks, locations, functions, string table, and additional
// metadata. It modifies and annotates pprof Profile with OpenTelemetry
// specific fields.
//
// Note that whilst fields in this message retain the name and field id from pprof in most cases
// for ease of understanding data migration, it is not intended that pprof:Profile and
// OpenTelemetry:Profile encoding be wire compatible.
message Profile {

// A description of the samples associated with each Sample.value.
// For a cpu profile this might be:
// [["cpu","nanoseconds"]] or [["wall","seconds"]] or [["syscall","count"]]
Expand All @@ -197,10 +197,10 @@ message Profile {
// If multiple binaries contribute to the Profile and no main
// binary can be identified, mapping[0] has no special meaning.
repeated Mapping mapping_table = 3;
// Locations referenced by samples via location_indices.
// Locations referenced by stacks via location_indices.
repeated Location location_table = 4;
// Array of locations referenced by samples.
repeated int32 location_indices = 5;
// Stacks referenced by samples via stack_index.
repeated Stack stack_table = 5;
// Functions referenced by locations.
repeated Function function_table = 6;
// Lookup table for attributes.
Expand Down Expand Up @@ -374,11 +374,8 @@ message ValueType {
// augmented with auxiliary information like the thread-id, some
// indicator of a higher level request being handled etc.
message Sample {
// locations_start_index along with locations_length refers to to a slice of locations in Profile.location_indices.
int32 locations_start_index = 1;
// locations_length along with locations_start_index refers to a slice of locations in Profile.location_indices.
// Supersedes location_index.
int32 locations_length = 2;
// Reference to stack in Profile.stack_table.
int32 stack_index = 1;
// The type and unit of each value is defined by the corresponding
// entry in Profile.sample_type. All samples must have the same
// number of values, the same as the length of Profile.sample_type.
Expand All @@ -397,6 +394,12 @@ message Sample {
repeated uint64 timestamps_unix_nano = 6;
}

// A Stack represents a stack trace as a list of locations. The first location
// is the leaf frame.
message Stack {
repeated int32 location_indices = 1;
}

// Describes the mapping of a binary in memory, including its address range,
// file offset, and metadata like build ID
message Mapping {
Expand Down

0 comments on commit 849dc40

Please sign in to comment.