Skip to content

Commit 44091d9

Browse files
committed
fix(DENG-11237): drop schema.yaml + destination_table:null for GCS export
The prior fix used scheduling.destination_table: null, but that field with a query.py trips DAG task validation (task.py: destination_table None and query_file_path None -> 'One of destination_table or query_file_path must be specified'); the only existing destination_table:null tasks are query.sql. Use the reviewer's primary suggestion instead: ship only metadata.yaml + query.py (no schema.yaml), matching the newtab_merino GCS-export tasks. On the current deploy code a missing schema is a non-fatal SkippedDeployException (dry-run path) / _resolve_isolated_schema is skipped when there's no manifest, so no empty table is created and stage-deploy doesn't fail. CSV column contract stays documented in metadata.yaml.
1 parent eb3dc5e commit 44091d9

2 files changed

Lines changed: 3 additions & 25 deletions

File tree

sql/moz-fx-data-shared-prod/firefox_referral_derived/referral_installs_totals_to_gcs_v1/metadata.yaml

Lines changed: 3 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -10,10 +10,9 @@ description: |-
1010
One file per run date named `referral_data-<date>.csv` (date = Airflow {{ds}}),
1111
so an Airflow retry overwrites the same object. Runs daily.
1212
13-
This task only extracts to GCS; it does not build a queryable BigQuery table
14-
(scheduling.destination_table is null so deploy skips table creation). The
15-
sibling schema.yaml documents the CSV columns and lets stage-deploy resolve
16-
the query.py artifact.
13+
This task only extracts to GCS; it does not build a queryable BigQuery table.
14+
Like the newtab_merino GCS-export tasks, it ships only metadata.yaml + query.py
15+
(no schema.yaml), so deploy skips table creation (missing-schema skip).
1716
1817
Destination bucket `fx-referral-data-prod` (project moz-fx-springfield-prod) is
1918
owned by the Website team; the Airflow workload service account
@@ -26,7 +25,6 @@ labels:
2625
owner1: phlee
2726
scheduling:
2827
dag_name: bqetl_firefox_referral
29-
destination_table: null
3028
arguments:
3129
- --source-project=moz-fx-data-shared-prod
3230
- --source-dataset=firefox_referral_derived

sql/moz-fx-data-shared-prod/firefox_referral_derived/referral_installs_totals_to_gcs_v1/schema.yaml

Lines changed: 0 additions & 20 deletions
This file was deleted.

0 commit comments

Comments
 (0)