feat(DENG-11237): export referral totals CSV to GCS by phil-lee70 · Pull Request #9738 · mozilla/bigquery-etl

phil-lee70 · 2026-07-28T12:29:50Z

Description

Fast-follow to #9724 (the referral aggregation tables). Wires up the CSV-to-GCS export now that the Website team has provisioned the destination bucket and granted IAM.

⚠️ Stacked on #9724 — base is deng-11237-firefox-referral-pipeline, so this diff shows only the export. Merge #9724 first; GitHub will retarget this to main automatically once it does.

What this adds

referral_installs_totals_to_gcs_v1 — a query.py task that extracts referral_installs_totals_v1 to a timestamped CSV (referral_data-<ts>.csv, no header) at the root of the destination bucket. Joins the existing bqetl_firefox_referral DAG downstream of referral_installs_totals_v1 (via scheduling.referenced_tables).
bigquery_etl/referral_export/ — the CSV extract module. --destination-prefix is now optional (writes to the bucket root when omitted).

Destination

Bucket: fx-referral-data-prod (project moz-fx-springfield-prod), owned by the Website team.
The Airflow workload SA default-workloads@moz-fx-data-airflow-gke-prod.iam.gserviceaccount.com was granted write access via mozilla/webservices-infra#11912.
CSV format invite_code,total_installs (no header), one file per daily run — matches the Website team's requested format.

Notes

Writes to the bucket root per the filename convention Steve specified; worth a quick confirm that the Website ingestion reads the root (not a subfolder). The module supports a --destination-prefix if a subfolder is preferred.
The export produces empty/near-empty CSVs until the fxrefer: instrumentation ships and codes begin flowing — expected.

Related Tickets & Documents

Reviewer, please follow this checklist

Desktop MVP for the Firefox Referral Program. Counts Firefox first_run installs per referral (invite) code via the GA4 / download-attribution path (fxrefer: utm_content -> dl_token -> baseline_clients_first_seen), following the cfs_ga4_attr_v1 pattern, and exports cumulative totals to the Website team's GCS bucket as CSV. - referral_installs_daily_v1: incremental per-code first_run counts - referral_installs_totals_v1: cumulative rollup, zero-install codes omitted - referral_installs_totals_to_gcs_v1: CSV extract to GCS (bucket TBD) - checks.sql: warn on invite_code length drift (expected 17 chars) - tests/sql fixture for the daily query Fenix is stubbed (commented UNION ALL) pending the Play Store attribution field; iOS is out of scope. GCS destination is a placeholder pending the bucket + IAM from the Website team.

referral_installs_daily_v1 sets require_partition_filter: true, so the full-refresh totals query must filter submission_date or BigQuery rejects it at runtime and in dry-run CI. Add an all-time filter that scans every partition while satisfying the requirement.

The stage-deploy CI runs 'bqetl deploy --tables --isolated' over changed paths and discovers the query.py extract task as a table artifact. With no schema.yaml it can't resolve a schema (a .py can't be dry-run, and the new firefox_referral_derived dataset doesn't exist in prod for get_table), so the deploy fails. Ship a schema.yaml so schema resolution short-circuits; it also documents the exported CSV column contract.

The export task was scheduled in bqetl_firefox_referral with a placeholder destination bucket, so post-deploy Airflow would run it daily and fail until the Website team supplies the real bucket + IAM. Remove the export task (referral_installs_totals_to_gcs_v1) and its referral_export module from this PR, keeping just the two aggregate tables. The export lands as a fast-follow once the bucket is available.

The Test SQL job runs the daily-query fixture against BigQuery. attribution_ext is a JSON column; the fixture provided it as a quoted JSON string, which loads as a JSON string scalar so attribution_ext.dltoken is NULL and the join drops every row (query returned []). Use an inline YAML object, matching how other fixtures represent JSON columns (e.g. event_extra: {attempt: 1}).

Fast-follow to the referral aggregation tables. Now that the Website team has provisioned the bucket (fx-referral-data-prod in moz-fx-springfield-prod) and granted the Airflow workload SA write access (webservices-infra#11912), wire up the CSV-to-GCS export: - referral_installs_totals_to_gcs_v1: query.py extract of referral_installs_totals_v1 to referral_data-<ts>.csv at the bucket root, no header, daily. Joins the bqetl_firefox_referral DAG downstream of totals. - bigquery_etl/referral_export: CSV extract module; --destination-prefix is now optional (writes to bucket root when omitted). Stacked on the aggregation-tables PR (#9724).

github-actions

This adds a query.py GKEPodOperator task (referral_installs_totals_to_gcs_v1) that extracts referral_installs_totals_v1 to a timestamped headerless CSV in gs://fx-referral-data-prod, a new bigquery_etl/referral_export module implementing the extract, and updates the bqetl_firefox_referral DAG description. The task wires into the DAG downstream of the totals table via scheduling.referenced_tables, and the CLI/metadata shape closely follows the existing newtab_merino_extract_to_gcs_v3 precedent.

Main thing to resolve is the schema.yaml: unlike the newtab export tasks, its presence makes deploys create an empty table in firefox_referral_derived rather than skipping the artifact. The other two comments are about the unreachable post-result() error branch and the non-idempotent wall-clock filename.

On the open question in the description about root vs. subfolder: that's a contract question for the Website team and not something I can verify from the repo, so it's untouched here.

github-actions · 2026-07-28T12:34:01Z

@@ -0,0 +1,17 @@
+# This task extracts referral_installs_totals_v1 to a CSV in GCS; it does not
+# build a queryable table. The schema is declared so stage-deploy CI can resolve


issue: this schema.yaml will cause deploys to create a real, permanently empty table moz-fx-data-shared-prod.firefox_referral_derived.referral_installs_totals_to_gcs_v1.

The stated reason is backwards. query.* matches query.py, so the dir is already collected as a table artifact (bigquery_etl/cli/query.py schema deploy, and stage deploy via bigquery_etl/cli/stage.py:597) — the thing that keeps a query.py export task from being deployed is the absence of a schema file: deploy_table raises SkippedDeployException("Schema missing for ...") at bigquery_etl/deploy.py:93-96, and skips are non-fatal (bigquery_etl/cli/deploy.py:1129-1131). By adding the file you turn that skip into a _create_or_update against prod and stage.

The two existing GCS-export analogues, sql/moz-fx-data-shared-prod/telemetry_derived/newtab_merino_extract_to_gcs_v3/ and newtab_merino_priors_to_gcs_v1/, contain only metadata.yaml + query.py for exactly this reason.

Fix: delete this file and move the CSV column contract into the metadata.yaml description. If you want to keep the schema as documentation, also set scheduling.destination_table: null in metadata.yaml, which deploy_table honours explicitly (bigquery_etl/deploy.py:71-79).

need to keep the empty schema table to pass CI, the empty table is harmless

github-actions · 2026-07-28T12:34:10Z

+    )
+    extract_job.result()  # Waits for the job to complete.
+
+    if extract_job.state != "DONE":


suggestion: this branch is unreachable. extract_job.result() on line 85 blocks until the job finishes and raises the underlying GoogleAPICallError if it failed, so by the time line 87 runs state is always "DONE" — a failed export surfaces as the exception from result(), never as this raise. (The newtab_merino module has the same shape, but there it only log.errors, so the dead code is harmless there.)

Drop lines 87-88 and let result() propagate, or if you want a distinct message, wrap the call:

try: extract_job.result() # Waits for the job to complete. except Exception as e: raise click.ClickException( f"Export to {destination_uri} failed: {extract_job.errors}" ) from e

added try/except raising click.ClickException(f"Export to {destination_uri} failed: {extract_job.errors}")

github-actions · 2026-07-28T12:34:19Z

+    """Extract the referral totals table to a timestamped CSV file in GCS."""
+    client = bigquery.Client(source_project)
+
+    timestamp = datetime.now(timezone.utc).strftime("%Y-%m-%dZ%H:%M:%S")


suggestion: deriving the filename from wall-clock time makes the task non-idempotent. An Airflow retry (or a manual rerun/clear) after a partially-completed run writes a second referral_data-*.csv for the same logical date rather than replacing the first, and nothing in this module or the destination bucket prunes old objects — so the bucket accumulates one-plus file per day indefinitely and the Website ingestion has to infer which object is authoritative.

Two options that fit existing repo patterns:

Pass the run date through and use it in the name: --date={{ds}} in scheduling.arguments is well established for query.py tasks (e.g. sql/moz-fx-data-shared-prod/ai_usage_derived/claude_usage_v1/metadata.yaml:18). A retry then overwrites the same object.

Write a stable referral_data-latest.csv alongside the timestamped copy, and add a retention option, as bigquery_etl/newtab_merino/__init__.py does with latest.json + --deletion-days-old.

added an idempotent filename using param --date to the module and --date={{ds}}

- No empty companion table: set scheduling.destination_table: null so deploy_table skips table creation (prod + stage). Kept schema.yaml because the --isolated stage deploy resolves the query.py artifact via _resolve_isolated_schema, which raises (not skips) when no schema resolves; destination_table: null is the mechanism that prevents the table. Moved the CSV column contract into metadata.yaml. - Idempotent output filename: take --date={{ds}} and name the file referral_data-<date>.csv so an Airflow retry overwrites the same object instead of accumulating one file per run. - Dropped the unreachable post-result() state check; wrap result() to raise a distinct ClickException with the destination and job errors.

…port The prior fix used scheduling.destination_table: null, but that field with a query.py trips DAG task validation (task.py: destination_table None and query_file_path None -> 'One of destination_table or query_file_path must be specified'); the only existing destination_table:null tasks are query.sql. Use the reviewer's primary suggestion instead: ship only metadata.yaml + query.py (no schema.yaml), matching the newtab_merino GCS-export tasks. On the current deploy code a missing schema is a non-fatal SkippedDeployException (dry-run path) / _resolve_isolated_schema is skipped when there's no manifest, so no empty table is created and stage-deploy doesn't fail. CSV column contract stays documented in metadata.yaml.

…ds it) Removing schema.yaml re-broke 'Deploy Changes to Stage': the --isolated deploy resolves the query.py artifact via the schema ladder and hard-fails (can't dry-run .py; SELECT*/get_table on the nonexistent table denied) — it does NOT skip. And destination_table: null (the clean no-table signal) is incompatible with query.py: of_python_script sets query_file_path after construction, so the Task __attrs_post_init__ check fails DAG generation. So the only config that passes both Test bqetl and stage-deploy is schema.yaml present + no destination_table: null. Downside: an empty companion table is created on deploy. Documented the constraint + tradeoff in schema.yaml. Keeps the --date={{ds}} idempotency and distinct-error-message fixes.

alekhyamoz · 2026-08-05T18:43:56Z

Couple of comments
Not sure what the expected behavior here is. Do we archive the files or is Website team responsible. Similar existing query.py Merino's equivalent takes deletes the file. --deletion-days-old=90 and calls delete_old_files()

alekhyamoz · 2026-08-05T18:41:33Z

+    referral_installs_daily_v1 -> referral_installs_totals_v1 ->
+    referral_installs_totals_to_gcs_v1.
  repo: bigquery-etl
  schedule_interval: 0 4 * * *


The date is off by one: the DAG runs 0 4 * * *, so {{ds}} is the previous day. The file written on Aug 6 is named referral_data-2026-08-05.csv. If the Website team's job looks for today's date it might not find any file.

this is expected and intentional. {{ds}} is the logical date (interval start), so the 0 4 * * * run on Aug 6 writes referral_data-2026-08-05.csv — a day behind wall-clock. This keeps the export idempotent: a retry or backfill for a logical date overwrites the same file rather than creating a new one.

alekhyamoz · 2026-08-05T18:45:14Z

+-- test codes (e.g. TESTCODE01) and any pre-rollout data may not be 17 chars yet.
+
+#warn
+ASSERT (


Do we need checks.sql for exporting jobs? Seems redundant.
Also checks.sql is obsolete. We have moved to Bigeye checks.

good catch - that checks.sql was a leftover from PR 9724 and should not be included - removed!

alekhyamoz

r+wc
Approved as long as concerns are resolved

This branch predates the checks->Bigeye migration in #9724, so its diff was re-adding referral_installs_daily_v1/checks.sql — a file that no longer exists on main (replaced by bigconfig.yml) and that belongs to the daily table, not this export PR. Remove it so #9738 only touches the export artifacts.

scholtzan · 2026-08-05T19:13:57Z

Integration report

View full diff, if available (SQL + DAGs) (requires private-bigquery-etl access)
Private CI run
Bigeye monitoring dry-run: ✅ plan clean — view plan (requires private-bigquery-etl access)

phil-lee70 · 2026-08-06T13:09:43Z

Couple of comments Not sure what the expected behavior here is. Do we archive the files or is Website team responsible. Similar existing query.py Merino's equivalent takes deletes the file. --deletion-days-old=90 and calls delete_old_files()

Right now referral_export has no retention, it writes one referral_data-.csv per day (a retry overwrites the same day's file, but each new day adds one), so left alone the bucket accumulates a small CSV per day.

Regarding ownership, since fx-referral-data-prod is the Website team's bucket in their project - if they want to delete files after a certain amount of time - my preference is a GCS Object Lifecycle rule on their side (auto-delete objects older than N days) - zero code, and it keeps our Airflow SA write-only (objectCreator).

@stevejalim it is your bucket: do you prefer to keep the CSVs accumulating each day, creating lifecycle TTL rule on your side, or us pruning in the exporter? Not urgent (files are tiny and empty until the instrumentation is live).

stevejalim · 2026-08-06T13:48:56Z

@stevejalim it is your bucket: do you prefer to keep the CSVs accumulating each day, creating lifecycle TTL rule on your side, or us pruning in the exporter? Not urgent (files are tiny and empty until the instrumentation is live).

Yeah, we have a 90-day retention policy on the bucket via GCS Object Lifecycle - sounds like we're on the same page ✅

phil-lee70 added 6 commits July 24, 2026 10:22

phil-lee70 requested a review from a team as a code owner July 28, 2026 12:29

github-actions Bot reviewed Jul 28, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

phil-lee70 added 2 commits July 28, 2026 10:43

phil-lee70 force-pushed the deng-11237-referral-gcs-export branch from 4fad00f to 44091d9 Compare July 28, 2026 15:10

Base automatically changed from deng-11237-firefox-referral-pipeline to main July 28, 2026 15:18

This comment has been minimized.

Sign in to view

Merge branch 'main' into deng-11237-referral-gcs-export

9c52ab2

This comment has been minimized.

Sign in to view

alekhyamoz reviewed Aug 5, 2026

View reviewed changes

alekhyamoz approved these changes Aug 5, 2026

View reviewed changes

phil-lee70 added this pull request to the merge queue Aug 6, 2026

github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Aug 6, 2026

phil-lee70 added this pull request to the merge queue Aug 6, 2026

github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Aug 6, 2026

		@@ -0,0 +1,17 @@
		# This task extracts referral_installs_totals_v1 to a CSV in GCS; it does not
		# build a queryable table. The schema is declared so stage-deploy CI can resolve

Uh oh!

Conversation

phil-lee70 commented Jul 28, 2026 • edited by atlassian Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

What this adds

Destination

Notes

Related Tickets & Documents

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

phil-lee70 Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

phil-lee70 Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

phil-lee70 Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

alekhyamoz commented Aug 5, 2026

Uh oh!

alekhyamoz Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

phil-lee70 Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

alekhyamoz Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

phil-lee70 Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

alekhyamoz left a comment

Choose a reason for hiding this comment

Uh oh!

scholtzan commented Aug 5, 2026

Integration report

Uh oh!

phil-lee70 commented Aug 6, 2026

Uh oh!

stevejalim commented Aug 6, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

phil-lee70 commented Jul 28, 2026 •

edited by atlassian Bot

Loading

phil-lee70 Jul 28, 2026 •

edited

Loading