backfill(fenix): sidefill Play Store attribution for firefox_android_clients (DENG-11356) - #9744
backfill(fenix): sidefill Play Store attribution for firefox_android_clients (DENG-11356)#9744phil-lee70 wants to merge 4 commits into
Conversation
…clients (DENG-11356) Follow-up to #9740, which fixed firefox_android_clients_v1 to read Play Store attribution from its dedicated ping. This backfills the clients first seen between the 2026-04-06 cutover and 2026-07-29, whose attribution was written before the fix. Split out of #9740 so the query change is deployed and running before any partitions are swapped into production.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
This adds a backfill entry and a custom per-partition sidefill query for fenix_derived.firefox_android_clients_v1, filling play_store_attribution_* (and their metadata.*__ping_datetime fields) for first_seen_date partitions from 2026-04-06 to 2026-07-29 from the dedicated play_store_attribution ping.
I checked the sidefill against the _current logic in #9740: the per-field ARRAY_AGG(... IGNORE NULLS ORDER BY ping_info.seq, submission_timestamp LIMIT 1)[SAFE_OFFSET(0)] shape, the ping-over-existing COALESCE preference, and the CASE-based ping datetimes all line up, and the enumerated metadata struct matches the field order in schema.yaml exactly (reported_first_session_ping … meta_attribution_app__ping_datetime). The entry also uses the custom_query_path + override_depends_on_past_null_partition combination the way bigquery_etl/backfill/validate.py and _initiate_backfill expect, matching the clients_first_seen_v3 precedent, and the * REPLACE against prod is the read pattern the override's staging-rewrite skip is built for.
Two comments inline, both on backfill.yaml rather than the SQL: a coverage window that will drift once the merge waits on #9740, and the fact that the operational pause instruction is stored in YAML comments that the backfill tooling strips.
| @@ -0,0 +1,24 @@ | |||
| 2026-07-30: | |||
| start_date: 2026-04-06 | |||
| end_date: 2026-07-29 | |||
There was a problem hiding this comment.
issue: end_date: 2026-07-29 plus the "don't merge until #9740 is merged and deployed" instruction leaves a permanently unattributed window.
#9740's fix only populates play_store_attribution_* for clients that appear in _current, i.e. new profiles on their own first-seen day; for every later run the final SELECT takes COALESCE(_previous.play_store_attribution_*, _current....), so a NULL already written for an older client is never revisited. Clients first seen from 2026-07-30 up to the day before #9740's first fixed scheduled run therefore fall between this backfill's end_date and the fix's effective start, and stay NULL forever.
Bump entry_date and end_date at the time the entry is actually merged/run so the range ends the day before #9740's first fixed run. Note this also interacts with MAX_BACKFILL_ENTRY_AGE_DAYS = 28 (bigquery_etl/backfill/utils.py:30): if the merge slips more than 28 days past the entry date, an Initiate entry stops being picked up by backfill scheduled at all.
There was a problem hiding this comment.
done, dates adjusted based on today's date
| # IMPORTANT for whoever runs `backfill complete`: pause the | ||
| # bqetl_analytics_tables.firefox_android_clients task first. `complete` copies the staging | ||
| # partitions into production one at a time, while the scheduled task overwrites the whole table | ||
| # on every run (date_partition_parameter is null). A run that reads _previous before the copy | ||
| # loop finishes and writes after it would silently revert partitions already copied. | ||
| # firefox_android_clients_v1 is depends_on_past with a null date_partition_parameter, so the | ||
| # day-by-day path via query.sql is rejected. This is a targeted per-partition sidefill of the six | ||
| # play_store_attribution_* fields rather than a full --reinitialize-table rebuild back to 2020. |
There was a problem hiding this comment.
suggestion: this warning won't survive the next edit to the file. bqetl backfill create rewrites the whole backfill.yaml from Backfill.to_yaml() (bigquery_etl/cli/backfill.py:371), which is a plain yaml.dump of the entry fields (bigquery_etl/backfill/parse.py:263-283) — every YAML comment is dropped. As soon as anyone adds a second entry for this table, the pause/unpause instruction disappears.
Move the operationally load-bearing part into reason: (a |- block scalar, as in telemetry_derived/clients_first_seen_v3/backfill.yaml), which is a real field and is round-tripped.
There was a problem hiding this comment.
done, moved comment to the reason: parameter
This comment has been minimized.
This comment has been minimized.
Two review points: - YAML comments do not survive `bqetl backfill create`, which rewrites the whole file from a plain yaml.dump of the entry fields. The pause/unpause instruction would have vanished as soon as anyone added a second entry for this table, so it now lives in `reason:` as a block scalar, which is round-tripped. - end_date plus the "merge after #9740" ordering left a permanently unattributed window: the fix only writes attribution on a client's own first-seen day, and later runs keep the already-written NULL via COALESCE(_previous, _current), so any first_seen_date between end_date and the fix's first run would stay NULL forever. The entry now says to bump entry_date and end_date before initiating, and notes the 28-day entry age limit. Also refreshed entry_date to 2026-07-31 and end_date to the latest available partition, 2026-07-30.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
1 similar comment
This comment has been minimized.
This comment has been minimized.
Integration report
|
2 participants
Description
Backfill for DENG-11356, split out of #9740 per review.
#9740 fixes
firefox_android_clients_v1to read Play Store attribution from the dedicatedplay_store_attributionping. This backfills the clients first seen between the 2026-04-06 cutover and 2026-07-30, whose attribution was written before that fix.entry_dateandend_date. The dates in the entry are placeholders (currently 2026-07-31 / 2026-07-30, the latest available partition).end_datemust become the day before the first scheduled run of the fixed query.Why this matters: #9740 only writes attribution for a client on its own first-seen day. On later runs the final
SELECTkeeps the already-written value viaCOALESCE(_previous.play_store_attribution_*, _current.play_store_attribution_*), so a NULL written for an older client is never revisited. Anyfirst_seen_datefalling between this entry'send_dateand the fix's first run would stay NULL permanently — roughly 3,500 attributed clients per day in that window.bqetl_analytics_tables.firefox_android_clientsaroundbackfill complete, and unpause after. Completion copies the staging partitions into production one at a time (backfill.py:1412-1427), while the scheduled task overwrites the whole table on every run (date_partition_parameter: null). A run overlapping the copy loop would silently revert partitions already copied.Merge order: #9740 must be merged, deployed, and have run at least once first.
Why a sidefill and not a reinitialize
The table is
depends_on_past: truewith a nulldate_partition_parameter, so the normal day-by-day path is rejected byvalidate_depends_on_past. The two supported options are:--reinitialize-table— rebuilds every partition back to 2020, ~2.7 TiB, and would rewrite history unrelated to this fix.custom_query_pathwithoverride_depends_on_past_null_partition: true. The custom query reads eachfirst_seen_datepartition's own production data and replaces only the sixplay_store_attribution_*fields and their metadata ping datetimes, so it is partition-independent as that override requires.override_depends_on_past_end_date: trueis set becauseend_date(the latest available partition) precedes the entry date.Validation
The sidefill was cross-checked against the updated
query.sqlfrom #9740 on 2026-07-01 and reproduces it exactly:play_store_attribution_termnon-emptyplay_store_attribution_campaignnon-emptyCost is 0.2 GiB per partition instead of 291 GiB for a full query.sql run.
Also verified: the sidefill's output column order and
metadatastruct field order are identical to the live table, which matters because completion writes decorated partitions.Full validation results, including the acceptance criterion (112 → 23,502 for the week of 2026-06-29, matching
new_profile_clients), are in the ticket and in #9740.Downstream
No downstream backfill is needed.
active_users_aggregates_v3andv4read onlyclient_id,adjust_network, andinstall_sourcefromfenix.firefox_android_clients(mobile_query.sql:4-11) — none of the fields this touches. Of the other consumers (attributable_clients_v1,ltv_states_v1,android_app_campaign_stats_v1,active_users_aggregates_attribution_v1,attributable_clients_v2), none referenceplay_store_attribution_*; the only other mention is commented-out, blocked code inreferral_installs_daily_v1.