Comments (17)
That should be most CPU profiles, right?
Most of my experiments are with CPU profiles.
For profiles that focus on lock contention or memory allocation, the situation might be slightly different. I can imagine that for such profiles the leaf frame is similar more often. But the described problem should also in this case be the same, if memory allocations or lock acquisition happens in a stack with a high number of frames.
from oteps.
Yeah, this is definitely tree-ish: we're essentially trying to encode a flamegraph tree efficiently. For optimal density we'd probably want some sort of prefix tree structure. That being said, I'm not sure whether we're willing to pay the compute price of maintaining one in the profiler.
The algorithm that I had in mind for use with the repeated
fields falls more into the "simple and hopefully good enough" category: define some chunk size, split traces by that size and then keep a hash-LRU of chunks that we've seen previously. Should provide a good amount of dedup at very little compute / memory overhead. Implementations that wish to roll something fancier can do that as well.
from oteps.
cc @open-telemetry/profiling-maintainers @open-telemetry/profiling-approvers
from oteps.
Comment from the OTel maintainer meeting: could / should this be moved to a comment on the current Profiling PR in the OTLP repository?
from oteps.
This issue is linked in https://github.com/open-telemetry/opentelemetry-proto/pull/534/files#r1561128746. As this particular issue is relevant to the specification, I did open the issue in this repository.
from oteps.
Thanks for raising this.
In particular for deep stack traces with a high number of similar frames and where only leaf frames are different,
That should be most CPU profiles, right? @petethepig IIRC you had some benchmarks that showed the efficiency of this new encoding of stack traces. Did you use realistic CPU profiling data?
If this new approach is not a clear win in the majority of situations, we should remove it.
from oteps.
We could simply make both locations_start_index
and locations_length
repeated
fields: this would allow implementations to de-duplicate prefixes and should be even more efficient than just listing all indices all the time.
For example if you had two traces that only vary in the leaf:
trace_foo:
0) libc_entry_point
1) main
2) run_my_app
3) do_fancy_stuff
4) do_boring_stuff
5) strcpy
6) strlen
trace_bar:
0) libc_entry_point
1) main
2) run_my_app
3) do_fancy_stuff
4) do_boring_stuff
5) strcpy
6) memcpy
Then you could create locations like so:
locations:
0) libc_entry_point
1) main
2) run_my_app
3) do_fancy_stuff
4) do_boring_stuff
5) strcpy
6) strlen
7) memcpy
And then encode the reference like this:
trace_foo:
locations_start_index: [0]
locations_length: [7]
trace_bar:
locations_start_index: [0, 7]
locations_length: [6, 1]
from oteps.
@athre0z interesting idea! Do you have an algorithm in mind for encoding the data in this way?
A bit of a meta comment: I think it's difficult to evaluate different stack trace encoding schemas without some alignment on how we value encoding vs decoding efficiency, compression, as well as overall complexity. Additionally I suspect that we're reinventing well-known tree encoding formats here (the above looks trie-ish?), and that there is a lot more prior art that we could explore.
from oteps.
Your algorithm sounds like it could work nicely. That being said, I see two paths forward:
- Try out new ideas like yours and make sure we get the evaluation right this time (it seems like we didn't the first time around).
- Go back to pprof's encoding, which is not ideal, but is simpler to encode/decode and keeps us more compatible with pprof.
What do you think?
from oteps.
I don't really have a strong opinion on this. Intuitively I'd guess that this location list may make up a significant portion of message size, but these things tend to be hard to guess. Makes me wish for a protobuf message size profiler that attributes size consumed to message fields. Bonus points if it could also do it for compressed message size!
Whether "keeping more compatible with pprof" is a priority, IMHO, depends on whether Google decides to donate the format to OTel or not. If pprof keeps evolving independently, then we'll find ourselves in a C/C++ kind of situation where newer C versions gained features that C++ doesn't have, and it'll just be pure pain to somehow keep things compatible. In that case I'd prefer to intentionally break compatibility to avoid the misconception of interoperability without transpiling.
from oteps.
Additionally I suspect that we're reinventing well-known tree encoding formats here (the above looks trie-ish?), and that there is a lot more prior art that we could explore.
One approach I used a couple of times for encoding a tree is with two arrays where one array represents index into the parent node and another array contains the reference to the actual payload. In case of pprof profile this would look something like
message Profile {
...
// ID of the location corresponding to this calltree node.
// Note that len(calltree_location_ids) == len(calltree_parent_indices).
repeated uint64 calltree_location_ids = <id>;
// Index of the parent calltree node or -1 if root.
repeated int64 calltree_parent_indices = <id+1>;
}
...
message Sample {
// Index of the calltree node as defined by
// Profile.{calltree_location_ids,calltree_parent_indices} or -1 if no call stack.
int64 calltree_index = <id>;
}
Note that the structure-of-arrays approach is used here to minimize allocations at protobuf serialization / deserialization level. Alternative array-of-structures approach would contain a single array of a special Calltree
message type in the Profile message, but that would mean allocating a fair amount of small objects which is what the proposed approach tries to avoid.
One additional nice thing about this representation is that many data processing algorithms for profiling data operate on call trees and storing the data in this encoding even in memory is convenient enough. For example, something like focus
filter in pprof which filters the flame graph to stacks where any frame matches a specified regexp can be implemented as an efficient two-pass algorithm where first proper calltree nodes are selected and then tree is traversed to drop nodes that are filtered out at the first pass or are a descendent of such node.
Just as a thought.
from oteps.
Related Issues (20)
- Proposal: Reduce clock-skew issues in mobile and other client-side trace sources HOT 5
- Proposal: Supporting Real User Monitoring Events in OpenTelemetry HOT 27
- Proposal: Add Sensitive Data Labels HOT 15
- Proposal: Remote Sampling
- Proposal: "Plugable backend" Tracing Client/Query library HOT 16
- Proposal: Add support for Elastic Common Schema (ECS) in OpenTelemetry HOT 6
- Proposal: specify how opentelemetry will deal with idle metrics no longer being reported
- Add link to opentelemetry.io under About section HOT 1
- Proposal: Support ML Monitoring HOT 6
- Proposal: Dynamic configuration of metrics HOT 2
- Proposal: clarify behavior when retrieving non-existent currently active span HOT 13
- Proposal: The OpenTelemetry Spec should allow SDKs to export all the spans regardless of their sampled flag
- Proposal: OpenTelemetry Sandbox HOT 12
- Proposal: TraceID just being hex value doesnt help with decision making at tracing backend. Is there a possibility to encode additional metadata such as timestamp, region etc. HOT 4
- Proposal: Service renaming HOT 4
- Add a "not implemented" stage to maturity levels HOT 3
- Group maturity status into experimental and stable HOT 2
- How to convert Java Flight Recorder (JFR) file to Profiling Data Model v2 HOT 5
- profiles/follow up: consistent time format HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from oteps.