Skip to content

Make DedupeIntegration more memory efficient. #4446

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 10 commits into
base: master
Choose a base branch
from

Conversation

antonpirker
Copy link
Member

@antonpirker antonpirker commented Jun 5, 2025

Users reported that the DedupeIntegration can use up a lot of memory, because it keeps a full exception in memory for checking if it has seen this exception already.

Depending on the users code those exception objects can be big because they also include the traceback and local variables (which can be huge).

Idea is now to not save the whole exception but just a hash of the important parts of the exception to decide if we have seen this exception again.

fixes #3165
fixes #4327

Copy link

codecov bot commented Jun 5, 2025

❌ 33 Tests Failed:

Tests completed Failed Passed Skipped
24121 33 24088 5796
View the top 3 failed test(s) by shortest run time
tests.test_basics::test_option_before_breadcrumb
Stack Traces | 0.067s run time
tests/test_basics.py:240: in test_option_before_breadcrumb
    assert not reports
E   AssertionError: assert not [('event_processor', 'error')]
tests.test_basics::test_option_before_breadcrumb
Stack Traces | 0.071s run time
tests/test_basics.py:240: in test_option_before_breadcrumb
    assert not reports
E   AssertionError: assert not [('event_processor', 'error')]
tests.test_basics::test_breadcrumbs
Stack Traces | 0.072s run time
tests/test_basics.py:396: in test_breadcrumbs
    (event,) = events
E   ValueError: not enough values to unpack (expected 1, got 0)

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@antonpirker antonpirker changed the title Store a hash of the exc instead of full exc in DedupeIntegration Make DedupeIntegration more memory efficient. Jun 5, 2025
@sl0thentr0py
Copy link
Member

sl0thentr0py commented Jun 5, 2025

@antonpirker can't you just id(exc)?

@antonpirker
Copy link
Member Author

i thought about that too. but garbage collection moves objects around. so id(exc) could be different when we safe it and when we compare it...

@antonpirker
Copy link
Member Author

The id() of an object is guaranteed to never change during the life cycle of the object. So taking it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Memory leak during exception logging Memory leak in AWS Fargate
2 participants