feat(active_job): support for tracing (#2947)

* feat(rails): add messaging span data to ActiveJob consumer transaction

Sets messaging.message.id, messaging.destination.name,
messaging.message.retry.count, and messaging.message.receive.latency
on the consumer transaction, mirroring sentry-sidekiq's middleware.

Adds an opt-in shared example that adapters can include to verify
the data fields are populated correctly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(rails): emit producer span when enqueueing ActiveJob

Wraps ActiveJob enqueue with a `queue.publish` child span when an
active parent transaction exists, mirroring sentry-sidekiq's client
middleware. Uses the public `around_enqueue` callback so no new
ActiveJob monkey-patching is introduced.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(rails): propagate trace context through ActiveJob payload

Mirrors the OpenTelemetry pattern (the only documented way to add
metadata to an ActiveJob payload — Rails has no public extension hook
for serialize/deserialize): prepends the existing ActiveJobExtensions
module with serialize/deserialize overrides that inject and recover
sentry-trace and baggage headers under a namespaced "_sentry" key,
wrapped in rescue blocks so a Sentry bug never breaks job execution.

Threads the deserialized headers into SentryReporter.record, which now
uses Sentry.continue_trace when present so the consumer transaction
shares the producer's trace_id and chains under the producer
queue.publish span.

Guards the around_enqueue producer-span registration against duplicate
registration (each Test::Application.define re-runs the railtie and
without idempotency this stacks dozens of nested queue.publish spans).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fixup(rails): account for AJ producer span in active_storage subscriber spec

The producer-span change makes ActiveStorage's internally-enqueued
AnalyzeJob emit an extra queue.publish span on the request transaction,
which the previous index-based span lookups did not anticipate.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(active_job): propagate allowed user context

When config.send_default_pii is true, the producer-side serialize
override now copies a whitelisted set of user fields (id, email,
username) into the _sentry payload block. The consumer-side
deserialize stashes them and SentryReporter.record applies them to the
new scope so that the consumer transaction (and any error event
captured during perform) carries the originating user without leaking
ip_address, segment, or other fields back into the queue.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(rails): isolate Sentry hub per worker thread for ActiveJob

Calls Sentry.clone_hub_to_current_thread before opening the consumer
scope when perform_now runs on a non-main thread (mirrors the
sentry-sidekiq server middleware). This ensures that worker-side state
captured during job execution lives on a thread-local hub clone and
cannot leak back into the main process hub.

Adds a behaviour-driven shared example: two concurrent jobs in
separate worker threads do not cross-pollute each other's tags, and
the calling thread's scope is unchanged after both complete.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor(rails): bundle ActiveJob tracing examples into a distributed_tracing meta

Replaces the five individual it_behaves_like opt-ins in
test_adapter_spec.rb with one composite shared example so future AJ
adapter specs can opt in with a single line.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fixup(rails): widen latency tolerance on Rails < 7 in messaging_span_data spec

Rails 6.x's ActiveSupport::Testing::TimeHelpers#travel does not accept
the with_usec: option, so it truncates Time.now to whole seconds and
the measured latency can land up to ~999ms below the travel delta. Use
a 1100ms tolerance on Rails < 7.0 and the original 50ms tolerance on
7.0+ where with_usec: true is available.

Verified against Rails 6.1, 7.1, and 8.1 via ./bin/test --version.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor(rails): introduce worker_thread harness hook for the hub-isolation example

The worker_hub_isolation shared example previously hard-coded
Thread.new for the two concurrent jobs it spawns. That works fine on
adapters that keep their queue state in-process (:test, :inline) but
some real adapters need per-thread setup (e.g. :solid_queue on SQLite
needs an isolated database per worker thread to avoid
SQLite3::BusyException).

Adds a `worker_thread(&block)` hook on the harness, defaulting to
`Thread.new(&block)`, and switches the shared example to call it.
Adapters that need extra worker setup override the hook.

Behaviour on :test (the only adapter on this branch) is unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(rails): no with_usec in 7.0

* chore(rails): patch AJ test adapter for 5.2

* fix(active_job): always emit retry count on the consumer transaction

Set messaging.message.retry.count unconditionally, derived from
job.executions, to mirror sentry-sidekiq which pulls the count
directly from the job hash. The previous gate of rescue_handlers.any?
was imprecise: that list also includes plain rescue_from declarations,
which do not trigger retries. A job declaring only rescue_from would
still emit retry.count, while the absence of any handler would suppress
it even though executions is still a meaningful signal. Removing the
gate makes the attribute consistent and unambiguous across adapters.

Co-Authored-By: github-copilot <noreply@example.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(active_job): add active_job_propagate_traces config option

Mirrors Sidekiq's propagate_traces flag. When set to false, trace
propagation headers are not injected into the serialized job payload
and the consumer starts a new unconnected transaction.

Defaults to true (existing behaviour is preserved).

Co-Authored-By: github-copilot <noreply@example.com>

* feat(active_job): set scope tags and context on consumer like Sidekiq

Mirror Sidekiq's scope enrichment: set a 'queue' scope tag and an
'active_job' context block (job_class, job_id, queue, provider_job_id)
on every event captured within the consumer scope, including the
transaction and any captured errors.

Co-Authored-By: github-copilot <noreply@example.com>

* fix(active_job): avoid shared queue race in jruby

Co-Authored-By: github-copilot <noreply@example.com>

* fix(active_job): save and restore hub around job execution

This correctly handles all execution modes:
- Dedicated async workers (new thread, nil hub): clone -> restore nil
- Inline inside a Rack request (rack hub on thread): clone -> restore
  rack hub so the HTTP response completes normally
- Thread-pool workers (recycled thread, stale hub): clone -> restore
  stale hub (irrelevant; next job will clone again)

Co-Authored-By: github-copilot <noreply@example.com>

* fix(active_job): better specs for thread isolation

Co-Authored-By: github-copilot <noreply@example.com>

* fix(active_job): correct retry counter

* fix(active_job): widen manual flushing to rails < 6.1

* refactor(active_job): make the spec harness adapter-agnostic

The harness embedded :test-adapter specifics — the Rails 5.2 payload-
preservation shim, the drain loop, and the enqueued-payload accessor.
It also reached past ActiveJob::TestHelper to set
ActiveJob::Base.queue_adapter directly, which conflicts with TestHelper's
own _test_adapter slot (TestHelper's before_setup runs outside our around
hook, so any direct assignment is silently shadowed).

Switch the harness to ActiveJob's official queue_adapter_for_test hook
and a small set of abstract methods (queue_adapter_for_test,
with_adapter_active, drain, last_enqueued_payload, boot_adapter,
reset_adapter) that adapter contexts implement. The :test-adapter
shared context now owns everything specific to TestAdapter — including
the Rails52FullPayloadTestAdapter shim and the drain loop. Subsequent
adapter backends (e.g. Sidekiq) can compose with the harness without
fighting it.

Generalises the one shared-example line that reached into the
TestAdapter shape (trace_propagation) via last_enqueued_payload.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(active_job): verify the AJ tracing suite passes on the :sidekiq adapter

Runs the common ActiveJob spec suite end-to-end against
ActiveJob::QueueAdapters::SidekiqAdapter, driven by
Sidekiq::Testing.fake! (block form, public API) and Sidekiq::Job.drain_all.
Validates that the AJ tracing extension works as a generic, adapter-
agnostic instrumentation — independent of sentry-sidekiq's native
middleware.

The :sidekiq context plugs into the harness via queue_adapter_for_test
(installing a SidekiqAdapter instance through ActiveJob::TestHelper) and
with_adapter_active (wrapping example.run in Sidekiq::Testing.fake! so
fake mode is scoped per-example without touching global state).

The context deliberately does not load sentry-sidekiq: loading it would
install Sidekiq's client/server middleware globally and register
SidekiqAdapter in skippable_job_adapters, both of which would
short-circuit the AJ extension we're exercising.

Sidekiq becomes a sentry-rails dev dependency, gated on Rails version
(Sidekiq 7+ doesn't support Rails 5.2). The spec file and support file
no-op cleanly on older matrices where the gem isn't bundled.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(e2e): end-to-end ActiveJob distributed-tracing spec

Drives the svelte-mini app to click a new "Trigger Job" button, which
fetches POST /jobs/sample on the rails-mini app. The browser SDK
propagates sentry-trace + baggage to the Rails request; the AJ
extension this branch adds emits a queue.publish span on the
http.server transaction at enqueue, and a queue.active_job consumer
transaction when the :async pool runs the job. The spec asserts all
three rails-side artifacts share one trace and are correctly linked
(sentry-trace header on the controller request, parent_span_id on the
consumer transaction, and matching messaging.* data on the producer
and consumer ends).

Polls the shared envelope log because :async runs the job on a
separate thread, so the HTTP response returns before the consumer
transaction is recorded.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* perf(active_job): boot the dummy app once per spec group

The harness was calling make_basic_app in its around-each block,
which creates a fresh Rails::Application subclass and runs every
initializer on each example. With 98 AJ examples that overhead dwarfed
the actual test work — and worse, it left behind state (Sidekiq's
@config_blocks list, accumulated routes, lingering Rails::Application
subclasses) that made each subsequent make_basic_app a little
slower. Under Ruby 3.4 + Rails 8.1.3 the per-example time grew 3× over
the run, pushing the full sentry-rails CI past the 15-min timeout.

Hoist make_basic_app to before(:all) and replicate the per-example
bits of Sentry::Rails::Railtie's after_initialize hook in the around
block — re-init Sentry, re-activate tracing / structured logging,
re-register the AJ event handlers. The one-time extensions
(controller methods, streaming reporter, backtrace cleanup, etc.) were
already installed by the initial make_basic_app and persist for the
group.

Also memoize the SidekiqAdapter instance in the :sidekiq context.
Each SidekiqAdapter.new appended to Sidekiq's internal @config_blocks
list and added an on(:quiet) callback; creating a fresh adapter per
example was unnecessary global churn.

Result: spec/active_job goes from 33s → 0.8s, and the full
sentry-rails spec task (Ruby 3.4 + Rails 8.1.3) goes from 9:22 to
2:31 — well under the CI limit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* tests(active_job): add basic scenario for active_job_propagate_traces

* refa(active_job): log more details when adding sentry payload fails

* refa(active_job): store our stuff under single _sentry ivar

* refa(active_job): better thread isolation spec

* tests(active_job): request-based hub isolation coverage

* fix(active_job): calc latency to reduce flakiness

* feat(rails): add dj/resque adapter specs

* refactor(active_job): ensure enqueue won't crash due to sentry crashing

* fix(active_job): symbolize user keys for proper scope handling

Scope reads allowed user keys as symbols, so we gotta symbolize users that
we receive as plain data from the job payload.

* tests(rails): ensure our instrumentation failures do not break AJ backends

* tests(rails): ensure queue.publish is emitted regardless of propagate traces setting

* refactor(rails): remove dead defaulting to {} for user

* refactor(rails): use consumer_transaction helper

* fix(rails): relax spying to make the spec pass under old rails

* fix(active_job): use `background: true` like integration gems

* chore(deps): bump locks

* refactor(active_job): simpler allowlist calc

* refactor(active_job): streamline test job defs

* refactor(active_job): shorten comment about spec skipping

* refactor(active_job): shorten spec desc

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: github-copilot <noreply@example.com>

Author: 3 people

.devcontainer/.env.example

@@ -12,5 +12,11 @@ SENTRY_E2E_SVELTE_APP_PORT=4001
 SENTRY_E2E_RAILS_APP_URL="http://localhost:4000"
 SENTRY_E2E_SVELTE_APP_URL="http://localhost:4001"
 
+# ActiveJob queue adapter under test: async | inline | sidekiq | resque | delayed_job
+SENTRY_E2E_ACTIVE_JOB_ADAPTER="async"
+
+# Redis for the sidekiq/resque adapters (the Compose service is named "redis")
+REDIS_URL="redis://redis:6379"
+
 # Faster builds with compose
 COMPOSE_BAKE=true

.devcontainer/docker-compose.yml

@@ -25,6 +25,9 @@ services:
     command: ["mise", "run", "e2e:serve"]
     environment:
       BUNDLE_PATH: /home/sentry/bundle
+    depends_on:
+      redis:
+        condition: service_healthy
     volumes:
       - ..:/workspace/sentry:cached
       - bundle-gems:/home/sentry/bundle
@@ -38,6 +41,11 @@ services:
       - ALLOW_EMPTY_PASSWORD=yes
     ports:
       - "6379:6379"
+    healthcheck:
+      test: ["CMD", "redis-cli", "ping"]
+      interval: 2s
+      timeout: 3s
+      retries: 10
 
 volumes:
   bundle-gems:

.github/workflows/e2e_tests.yml

@@ -17,18 +17,24 @@ concurrency:
 
 jobs:
   e2e-tests:
-    name: e2e tests
+    name: e2e tests (ruby ${{ matrix.ruby.flavor }}, ${{ matrix.adapter }})
     runs-on: ubuntu-latest
-    timeout-minutes: 5
+    timeout-minutes: 8
 
     strategy:
       fail-fast: false
       matrix:
-        include:
-          - ruby_version: "3.4.9"
+        ruby:
+          - version: "3.4.9"
             flavor: "3.4"
-          - ruby_version: "4.0.3"
+          - version: "4.0.3"
             flavor: "4.0"
+        adapter:
+          - async
+          - inline
+          - sidekiq
+          - resque
+          - delayed_job
 
     steps:
       - name: Checkout code
@@ -42,9 +48,10 @@ jobs:
         run: |
           cd .devcontainer
           cp .env.example .env
-          echo "RUBY_VERSION=${{ matrix.ruby_version }}" >> .env
-          echo "DOCKER_IMAGE=ghcr.io/getsentry/sentry-ruby-devcontainer-${{ matrix.flavor }}" >> .env
+          echo "RUBY_VERSION=${{ matrix.ruby.version }}" >> .env
+          echo "DOCKER_IMAGE=ghcr.io/getsentry/sentry-ruby-devcontainer-${{ matrix.ruby.flavor }}" >> .env
           echo "DOCKER_TAG=${{ steps.devcontainer-version.outputs.version }}" >> .env
+          echo "SENTRY_E2E_ACTIVE_JOB_ADAPTER=${{ matrix.adapter }}" >> .env
 
       - name: Log in to GHCR
         uses: docker/login-action@74a5d142397b4f367a81961eba4e8cd7edddf772 # v3
@@ -54,7 +61,7 @@ jobs:
           password: ${{ secrets.GITHUB_TOKEN }}
 
       - name: Pull test container image
-        run: docker pull ghcr.io/getsentry/sentry-ruby-devcontainer-${{ matrix.flavor }}:${{ steps.devcontainer-version.outputs.version }}
+        run: docker pull ghcr.io/getsentry/sentry-ruby-devcontainer-${{ matrix.ruby.flavor }}:${{ steps.devcontainer-version.outputs.version }}
 
       - name: Restore node_modules cache
         uses: actions/cache@6f8efc29b200d32929f49075959781ed54ec270c # v3
@@ -114,7 +121,7 @@ jobs:
         if: failure()
         uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
         with:
-          name: e2e-test-logs-ruby-${{ matrix.ruby_version }}
+          name: e2e-test-logs-ruby-${{ matrix.ruby.version }}-${{ matrix.adapter }}
           path: |
             log/sentry_debug_events.log
           retention-days: 7

.mise.toml

@@ -14,6 +14,10 @@ run = "cd spec/apps/rails-mini && bundle exec ruby app.rb"
 description = "Start the svelte-mini e2e app"
 run = "cd spec/apps/svelte-mini && npm run dev"
 
+[tasks."e2e:worker"]
+description = "Start the rails-mini ActiveJob worker (sidekiq/resque/delayed_job; idles for async/inline)"
+run = "cd spec/apps/rails-mini && bundle exec ruby worker.rb"
+
 [tasks."e2e:serve"]
 description = "Start all e2e apps in parallel"
-depends = ["e2e:rails", "e2e:svelte"]
+depends = ["e2e:rails", "e2e:svelte", "e2e:worker"]