feat(integrations): Add tracing to DramatiqIntegration

[Igreh]

Adds tracing support to DramatiqIntegration #3454

[szokeasaurusrex]

Hi @Igreh, I took a quick glance at your PR – at a high level this seems reasonable; however, before I do a full review, I would appreciate if you could please add some tests for these changes, since we need tests before we can consider merging this.

[szokeasaurusrex]

Why are we using a ContextVar here, instead of just a simple instance variable on the SentryMiddleware?

[Igreh]

To isolate variable between threads:

1. Dramatiq can be started in multithreading mode:
   dramatiq --processes 1 --threads 5 ...
2. And I was inspired by official middleware:
   CurrentMessageMiddleware

I will try recheck if it's actually needed

[Igreh]

Confirmed.
One SentryMiddleware instance will be shared between threads. So, isolating is necessary.
Previous (actually current) realisation adds _scope_manager to message, but I think ContextVar is much cleaner solution

[codecov]

Codecov Report

Attention: Patch coverage is 97.05882% with 1 line in your changes missing coverage. Please review.

Project coverage is 80.73%. Comparing base (2ccab61) to head (863d16b).
Report is 12 commits behind head on master.

Files with missing lines	Patch %	Lines
sentry_sdk/integrations/dramatiq.py	96.96%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4571      +/-   ##
==========================================
+ Coverage   80.71%   80.73%   +0.02%     
==========================================
  Files         156      156              
  Lines       16507    16522      +15     
  Branches     2806     2805       -1     
==========================================
+ Hits        13323    13339      +16     
- Misses       2300     2301       +1     
+ Partials      884      882       -2     

Files with missing lines	Coverage Δ
sentry_sdk/consts.py	94.64% <100.00%> (+0.01%)	⬆️
sentry_sdk/integrations/dramatiq.py	84.53% <96.96%> (+3.81%)	⬆️

... and 4 files with indirect coverage changes

[Igreh]

Hi @Igreh, I took a quick glance at your PR – at a high level this seems reasonable; however, before I do a full review, I would appreciate if you could please add some tests for these changes, since we need tests before we can consider merging this.

Hi!
Thanks for reply. I'm not very familiar about what should be tested exactly for now (transaction was successfully started? or what objects/props must be created? and etc ) but I will take a look and try to do my best =)
If you have any advice - would be very helpful

Edit:
Or I just need to cover one missed line?

[sentrivana]

Hey @Igreh, it'd be good to have "end-to-end" (i.e., not unit tests) for the different tracing code paths. Stuff like:

• the happy path (if everything works fine, you get the transaction and it has the data you expect it to have)
• how it behaves if there's an error (a transaction still gets sent, and it has the correct status, etc.; maybe also check if the user's program still behaves as expected -- the exception bubbles up if it should bubble up, etc.)
• trace propagation (i.e., the transaction picks up propagation headers/vars if present) -- I think this might still be missing in your PR

...and/or whatever you consider important so that when the test suite is run and ends up green, you'd have a reasonable amount of confidence that tracing in the Dramatiq integration is working.

You can have a look at some of these similar integrations and their test suites (e.g. arq, huey) for inspiration.

Thanks for working on this! Let us know if you need any help.

[Igreh]

Hi @sentrivana !
Thanks for your reply. It was very helpful.
I believe I was able to solve propagation issue (see last commit)
Also, I've changed the way dramatiq_task_id is stored - now as tag (like in celery). It's actually very convenient to find transaction by ID.
New tests were added (huey integration I used for inspiration).
One thing I can't understand for now: mechanism.type of error event is "logging" instead of "dramatiq". https://github.com/getsentry/sentry-python/pull/4571/files#diff-ed170393871d90a02937ffecacd1b736805e6ea3c43e5549fd0df9bab883709eR82
Disabling LoggingIntegration fixes it (and actually it's the way we use sentry) but I suppose the general solution must work on defaults)
Probably, I have to dig deeper to dramatiq source code to understand what's happening... Not sure but I will keep investigating

[Igreh]

Hi @sentrivana !
Things are more complicated than I expected... =)

I'm using (and testing) my current integration on real project and
we're experiencing different side effects, but I'm curious about one thing for now:
Retried transactions must be connected by the same trace? Is any sentry agreement for such issue?
Retrying in terms of Dramatiq means requeuing message and it means baggage is propagated also. Bug or feature ? )

Failed task with 3 retries might look like this:

[sentrivana]

Hey again 👋🏻 Thanks for your time on this!

One thing I can't understand for now: mechanism.type of error event is "logging" instead of "dramatiq". https://github.com/getsentry/sentry-python/pull/4571/files#diff-ed170393871d90a02937ffecacd1b736805e6ea3c43e5549fd0df9bab883709eR82
Disabling LoggingIntegration fixes it (and actually it's the way we use sentry) but I suppose the general solution must work on defaults)

What you're seeing is likely dramatiq logging the error when it happens, and our LoggingIntegration picking it up. In case that happens, our deduplication logic (in DedupeIntegration) drops the second (in this case better) event coming directly from the integration.

What you can do is add an ignore_logger("<logger-name>") to the integration, see for example here where we were solving the same issue for strawberry-graphql. This will tell the LoggingIntegration to skip capturing events from that specific logger, so the actual events coming from the specialized integration should get through.

(I know you're not working on the error part of the integration in this PR but it's still a good improvement so feel free to include it.)

I'm using (and testing) my current integration on real project and
we're experiencing different side effects, but I'm curious about one thing for now:
Retried transactions must be connected by the same trace? Is any sentry agreement for such issue?
Retrying in terms of Dramatiq means requeuing message and it means baggage is propagated also. Bug or feature ? )

Not sure off the top of my head how this works in our other task queue integrations, but my hunch is that retries should be under one trace.