-
Notifications
You must be signed in to change notification settings - Fork 3.3k
Fix rebatching when the id is in the current batch #46004
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Fix rebatching when the id is in the current batch #46004
Conversation
080babe to
b1e7258
Compare
There is a bug in MoveToEarlierBatch where if the trace id is still in the current batch it will not be properly removed from the current batch causing a decision to be made for the trace twice. This does not break any functionality but will cause the trace dropped too early metric to be improperly incremented if a decision cache is used. Signed-off-by: Chris Marchbanks <[email protected]>
b1e7258 to
a4ca32c
Compare
| // processed before the proposed batch it will do nothing. Returns the | ||
| // batch that the trace will now be a part of (which may stay the same). | ||
| MoveToEarlierBatch(id pcommon.TraceID, currentBatch, batchesFromNow uint64) uint64 | ||
| MoveToEarlierBatch(id pcommon.TraceID, traceCurrentBatch, batchesFromNow uint64) uint64 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This rename is to avoid confusion withcurrentBatch in the batcher struct.
| } | ||
|
|
||
| func (b *batcher) MoveToEarlierBatch(id pcommon.TraceID, currentBatch, batchesFromNow uint64) uint64 { | ||
| func (b *batcher) MoveToEarlierBatch(id pcommon.TraceID, traceCurrentBatch, batchesFromNow uint64) uint64 { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why rename that variable?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wanted to differentiate between the batch the trace is currently in and b.currentBatch/currentBatchID.
Signed-off-by: Chris Marchbanks <[email protected]>
Description
There is a bug in
(*batcher).MoveToEarlierBatchwhere if the trace id is still in the current batch it will not be properly removed from the current batch causing a decision to be made for the trace twice. This does not break any functionality but will cause the trace dropped too early metric to be improperly incremented if a decision cache is used while also using the newdecision_wait_after_root_receivedoption.I also noticed that we aren't updating the batch id on
actualDataafter moving a trace so fixed that too. It shouldn't make too much of a difference as it shouldn't be possible to receive two root spans, but better to be safe.Testing
New tests added that reproduce the failure and cover 100% of code paths in
id_batcher.go. I have also deployed this to a couple of internal environments and verified that traces dropped too early metric is no longer incremented.