backfill(fenix): sidefill Play Store attribution for firefox_android_clients (DENG-11356) #9744
+224
−0
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| 2026-07-30: | ||
| start_date: 2026-04-06 | ||
| end_date: 2026-07-29 | ||
| reason: | ||
| DENG-11356, populate play_store_attribution_* fields for clients first seen since the | ||
| 2026-04-06 ping cutover, from the dedicated play_store_attribution ping. | ||
| watchers: | ||
| - phlee@mozilla.com | ||
| status: Initiate | ||
| # IMPORTANT for whoever runs `backfill complete`: pause the | ||
| # bqetl_analytics_tables.firefox_android_clients task first. `complete` copies the staging | ||
| # partitions into production one at a time, while the scheduled task overwrites the whole table | ||
| # on every run (date_partition_parameter is null). A run that reads _previous before the copy | ||
| # loop finishes and writes after it would silently revert partitions already copied. | ||
| # firefox_android_clients_v1 is depends_on_past with a null date_partition_parameter, so the | ||
| # day-by-day path via query.sql is rejected. This is a targeted per-partition sidefill of the six | ||
| # play_store_attribution_* fields rather than a full --reinitialize-table rebuild back to 2020. | ||
|
Contributor
There was a problem hiding this comment. suggestion: this warning won't survive the next edit to the file. Move the operationally load-bearing part into
Contributor
Author
There was a problem hiding this comment. done, moved comment to the |
||
| custom_query_path: sql/moz-fx-data-shared-prod/fenix_derived/firefox_android_clients_v1/backfill_deng_11356.sql | ||
| override_depends_on_past_null_partition: true | ||
| # end_date is the latest available first_seen_date partition, which is before the entry date. | ||
| override_depends_on_past_end_date: true | ||
| shredder_mitigation: false | ||
| override_retention_limit: false | ||
| ignore_date_partition_offset: false | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,192 @@ | ||
| -- Sidefill for DENG-11356: populate the play_store_attribution_* fields for clients first seen | ||
| -- from the 2026-04-06 ping cutover onwards, from the dedicated `play_store_attribution` ping. | ||
| -- | ||
| -- `firefox_android_clients_v1` has depends_on_past: true with a null date_partition_parameter, so | ||
| -- it can't be backfilled one partition at a time via query.sql. Rather than reinitialize the whole | ||
| -- table (which would rewrite every partition back to 2020), this reads each first_seen_date | ||
| -- partition's own production data and only replaces the six play_store_attribution_* fields and | ||
| -- their metadata ping datetimes. It is therefore partition-independent, as the backfill code | ||
| -- requires for `override_depends_on_past_null_partition`. | ||
| -- | ||
| -- The ping is matched on the client's first-seen day, which is where 23,489 of the 23,516 | ||
| -- attributed clients in the validation week land. The ping preference and the COALESCE fallback | ||
| -- to any pre-existing value match the updated query.sql exactly. | ||
| WITH play_store_attribution_ping AS ( | ||
| SELECT | ||
| client_info.client_id AS client_id, | ||
| ARRAY_AGG( | ||
| IF( | ||
| NULLIF(metrics.string.play_store_attribution_campaign, "") IS NULL, | ||
| NULL, | ||
| STRUCT( | ||
| NULLIF(metrics.string.play_store_attribution_campaign, "") AS value, | ||
| DATETIME(submission_timestamp) AS ping_datetime | ||
| ) | ||
| ) IGNORE NULLS | ||
| ORDER BY | ||
| ping_info.seq ASC, | ||
| submission_timestamp ASC | ||
| LIMIT | ||
| 1 | ||
| )[SAFE_OFFSET(0)] AS play_store_attribution_campaign, | ||
| ARRAY_AGG( | ||
| IF( | ||
| NULLIF(metrics.string.play_store_attribution_content, "") IS NULL, | ||
| NULL, | ||
| STRUCT( | ||
| NULLIF(metrics.string.play_store_attribution_content, "") AS value, | ||
| DATETIME(submission_timestamp) AS ping_datetime | ||
| ) | ||
| ) IGNORE NULLS | ||
| ORDER BY | ||
| ping_info.seq ASC, | ||
| submission_timestamp ASC | ||
| LIMIT | ||
| 1 | ||
| )[SAFE_OFFSET(0)] AS play_store_attribution_content, | ||
| ARRAY_AGG( | ||
| IF( | ||
| NULLIF(metrics.string.play_store_attribution_medium, "") IS NULL, | ||
| NULL, | ||
| STRUCT( | ||
| NULLIF(metrics.string.play_store_attribution_medium, "") AS value, | ||
| DATETIME(submission_timestamp) AS ping_datetime | ||
| ) | ||
| ) IGNORE NULLS | ||
| ORDER BY | ||
| ping_info.seq ASC, | ||
| submission_timestamp ASC | ||
| LIMIT | ||
| 1 | ||
| )[SAFE_OFFSET(0)] AS play_store_attribution_medium, | ||
| ARRAY_AGG( | ||
| IF( | ||
| NULLIF(metrics.string.play_store_attribution_source, "") IS NULL, | ||
| NULL, | ||
| STRUCT( | ||
| NULLIF(metrics.string.play_store_attribution_source, "") AS value, | ||
| DATETIME(submission_timestamp) AS ping_datetime | ||
| ) | ||
| ) IGNORE NULLS | ||
| ORDER BY | ||
| ping_info.seq ASC, | ||
| submission_timestamp ASC | ||
| LIMIT | ||
| 1 | ||
| )[SAFE_OFFSET(0)] AS play_store_attribution_source, | ||
| ARRAY_AGG( | ||
| IF( | ||
| NULLIF(metrics.string.play_store_attribution_term, "") IS NULL, | ||
| NULL, | ||
| STRUCT( | ||
| NULLIF(metrics.string.play_store_attribution_term, "") AS value, | ||
| DATETIME(submission_timestamp) AS ping_datetime | ||
| ) | ||
| ) IGNORE NULLS | ||
| ORDER BY | ||
| ping_info.seq ASC, | ||
| submission_timestamp ASC | ||
| LIMIT | ||
| 1 | ||
| )[SAFE_OFFSET(0)] AS play_store_attribution_term, | ||
| ARRAY_AGG( | ||
| IF( | ||
| NULLIF(metrics.text2.play_store_attribution_install_referrer_response, "") IS NULL, | ||
| NULL, | ||
| STRUCT( | ||
| NULLIF(metrics.text2.play_store_attribution_install_referrer_response, "") AS value, | ||
| DATETIME(submission_timestamp) AS ping_datetime | ||
| ) | ||
| ) IGNORE NULLS | ||
| ORDER BY | ||
| ping_info.seq ASC, | ||
| submission_timestamp ASC | ||
| LIMIT | ||
| 1 | ||
| )[SAFE_OFFSET(0)] AS play_store_attribution_install_referrer_response, | ||
| FROM | ||
| `moz-fx-data-shared-prod.fenix.play_store_attribution` | ||
| WHERE | ||
| DATE(submission_timestamp) = @submission_date | ||
| AND DATE(submission_timestamp) >= "2026-04-06" | ||
| AND client_info.client_id IS NOT NULL | ||
| GROUP BY | ||
| client_id | ||
| ) | ||
| SELECT | ||
| clients.* REPLACE ( | ||
| COALESCE( | ||
| play_store.play_store_attribution_campaign.value, | ||
| clients.play_store_attribution_campaign | ||
| ) AS play_store_attribution_campaign, | ||
| COALESCE( | ||
| play_store.play_store_attribution_content.value, | ||
| clients.play_store_attribution_content | ||
| ) AS play_store_attribution_content, | ||
| COALESCE( | ||
| play_store.play_store_attribution_medium.value, | ||
| clients.play_store_attribution_medium | ||
| ) AS play_store_attribution_medium, | ||
| COALESCE( | ||
| play_store.play_store_attribution_source.value, | ||
| clients.play_store_attribution_source | ||
| ) AS play_store_attribution_source, | ||
| COALESCE( | ||
| play_store.play_store_attribution_term.value, | ||
| clients.play_store_attribution_term | ||
| ) AS play_store_attribution_term, | ||
| COALESCE( | ||
| play_store.play_store_attribution_install_referrer_response.value, | ||
| clients.play_store_attribution_install_referrer_response | ||
| ) AS play_store_attribution_install_referrer_response, | ||
| STRUCT( | ||
| clients.metadata.reported_first_session_ping AS reported_first_session_ping, | ||
| clients.metadata.reported_metrics_ping AS reported_metrics_ping, | ||
| clients.metadata.reported_baseline_ping AS reported_baseline_ping, | ||
| clients.metadata.min_first_session_ping_submission_date AS min_first_session_ping_submission_date, | ||
| clients.metadata.min_first_session_ping_run_date AS min_first_session_ping_run_date, | ||
| clients.metadata.min_metrics_ping_submission_date AS min_metrics_ping_submission_date, | ||
| clients.metadata.adjust_network__source_ping AS adjust_network__source_ping, | ||
| clients.metadata.install_source__source_ping AS install_source__source_ping, | ||
| clients.metadata.adjust_network__source_ping_datetime AS adjust_network__source_ping_datetime, | ||
| clients.metadata.install_source__source_ping_datetime AS install_source__source_ping_datetime, | ||
| CASE | ||
| WHEN play_store.play_store_attribution_campaign.value IS NOT NULL | ||
| THEN play_store.play_store_attribution_campaign.ping_datetime | ||
| ELSE clients.metadata.play_store_attribution_campaign__ping_datetime | ||
| END AS play_store_attribution_campaign__ping_datetime, | ||
| CASE | ||
| WHEN play_store.play_store_attribution_content.value IS NOT NULL | ||
| THEN play_store.play_store_attribution_content.ping_datetime | ||
| ELSE clients.metadata.play_store_attribution_content__ping_datetime | ||
| END AS play_store_attribution_content__ping_datetime, | ||
| CASE | ||
| WHEN play_store.play_store_attribution_medium.value IS NOT NULL | ||
| THEN play_store.play_store_attribution_medium.ping_datetime | ||
| ELSE clients.metadata.play_store_attribution_medium__ping_datetime | ||
| END AS play_store_attribution_medium__ping_datetime, | ||
| CASE | ||
| WHEN play_store.play_store_attribution_source.value IS NOT NULL | ||
| THEN play_store.play_store_attribution_source.ping_datetime | ||
| ELSE clients.metadata.play_store_attribution_source__ping_datetime | ||
| END AS play_store_attribution_source__ping_datetime, | ||
| CASE | ||
| WHEN play_store.play_store_attribution_term.value IS NOT NULL | ||
| THEN play_store.play_store_attribution_term.ping_datetime | ||
| ELSE clients.metadata.play_store_attribution_term__ping_datetime | ||
| END AS play_store_attribution_term__ping_datetime, | ||
| CASE | ||
| WHEN play_store.play_store_attribution_install_referrer_response.value IS NOT NULL | ||
| THEN play_store.play_store_attribution_install_referrer_response.ping_datetime | ||
| ELSE clients.metadata.play_store_attribution_install_referrer_response__ping_datetime | ||
| END AS play_store_attribution_install_referrer_response__ping_datetime, | ||
| clients.metadata.meta_attribution_app__ping_datetime AS meta_attribution_app__ping_datetime | ||
| ) AS metadata | ||
| ) | ||
| FROM | ||
| `moz-fx-data-shared-prod.fenix_derived.firefox_android_clients_v1` AS clients | ||
| LEFT JOIN | ||
| play_store_attribution_ping AS play_store | ||
| USING (client_id) | ||
| WHERE | ||
| clients.first_seen_date = @submission_date |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
issue:
end_date: 2026-07-29plus the "don't merge until #9740 is merged and deployed" instruction leaves a permanently unattributed window.#9740's fix only populates
play_store_attribution_*for clients that appear in_current, i.e. new profiles on their own first-seen day; for every later run the finalSELECTtakesCOALESCE(_previous.play_store_attribution_*, _current....), so a NULL already written for an older client is never revisited. Clients first seen from 2026-07-30 up to the day before #9740's first fixed scheduled run therefore fall between this backfill'send_dateand the fix's effective start, and stay NULL forever.Bump
entry_dateandend_dateat the time the entry is actually merged/run so the range ends the day before #9740's first fixed run. Note this also interacts withMAX_BACKFILL_ENTRY_AGE_DAYS = 28(bigquery_etl/backfill/utils.py:30): if the merge slips more than 28 days past the entry date, anInitiateentry stops being picked up bybackfill scheduledat all.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done, dates adjusted based on today's date