feat: CCP-6150-Migrate queueing from Bull to BullMQ (#286)

* Migrate queueing from Bull to BullMQ

Replaced Bull with BullMQ across backend queues, workers, and tests, including typed Queue/Job usage, new Queue/Worker/QueueEvents creation helpers, and a dedicated `:mq` Redis key prefix to isolate BullMQ keys. Introduced shared job policy utilities (`job-options.ts`) for retry/backoff/retention, refactored worker execution into QueueModule-managed processors, and extracted shared delivery logic into `DeliveryWorker`, `MergeBatchRunner`, and `DeliveryFailureHandler`. Updated monitoring, reconciliation, webhook delivery, cancel/reschedule job handling, and load-test drain scripts/workflow to match BullMQ semantics and naming.

* Parallelize queueing and normalize errors

Sonarqube fixes

* Fixing merge conflict mis-merge

Author: barrfalk

.github/workflows/load-test.yml

@@ -82,7 +82,7 @@ jobs:
         with:
           path: ./tests/load/drain-test.js
 
-      # Sample Bull queue depth in Redis until empty → drain rate + peak depth.
+      # Sample BullMQ queue depth in Redis until empty → drain rate + peak depth.
       - name: Measure queue drain
         if: always()
         continue-on-error: true # measurement must never block the ramps that follow
@@ -125,15 +125,15 @@ jobs:
               exit 0
             fi
             llen() { n=$(rc LLEN "$1" | tr -dc '0-9'); echo "${n:-0}"; }
-            # Bull keys are namespaced per deployment (REDIS_KEY_PREFIX, defaulting to
+            # Queue keys are namespaced per deployment (REDIS_KEY_PREFIX, defaulting to
             # RELEASE_NAME) because every deployment in the namespace shares one Redis.
             PREFIX=$(oc get deploy "$DEP" -o jsonpath='{.spec.template.spec.containers[*].env[?(@.name=="RELEASE_NAME")].value}' 2>/dev/null)
             PREFIX=${PREFIX:-notify}
-            echo "bull key prefix: $PREFIX"
+            echo "queue key prefix: $PREFIX:mq"
             depth() {
               t=0
               for q in notification-ingestion email-delivery; do
-                for s in wait active; do t=$((t + $(llen "$PREFIX:$q:$s"))); done
+                for s in wait active; do t=$((t + $(llen "$PREFIX:mq:$q:$s"))); done
               done
               echo "$t"
             }

backend/package-lock.json

@@ -36,7 +36,7 @@
     "@nestjs/typeorm": "^12.0.0",
     "axios": "^1.20.0",
     "body-parser": "^2.3.0",
-    "bull": "^4.14.1",
+    "bullmq": "^6.3.19",
     "class-transformer": "^0.5.1",
     "class-validator": "^0.15.0",
     "express-prom-bundle": "^8.0.0",