feat(): stores update clients references to new profile clients #7720

kik-kik · 2025-07-02T12:58:21Z

feat(): stores update clients references to new profile clients

This PR aims to update references to firefox_android_clients and firefox_ios_clients to the new new_profile_clients datasets so that the former can be deprecated.

…stats_v1 with new_profile_clients and update retention source

…profiles dataset

…rofile stats

… new datatasets so that the other two can be deprecated

dataops-ci-bot · 2025-07-09T09:42:56Z

Integration report for "update firefox_android_clients and retention_week_4 references to use new datatasets so that the other two can be deprecated"

`sql.diff`

Click to expand!

diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_analytics_tables.py /tmp/workspace/generated-sql/dags/bqetl_analytics_tables.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_analytics_tables.py	2025-07-09 09:37:55.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_analytics_tables.py	2025-07-09 09:41:57.000000000 +0000
@@ -452,12 +452,6 @@
             external_task_id="wait_for_checks__fail_fenix_derived__firefox_android_clients__v1",
         )
 
-        ExternalTaskMarker(
-            task_id="bqetl_fivetran_google_ads__wait_for_checks__fail_fenix_derived__firefox_android_clients__v1",
-            external_dag_id="bqetl_fivetran_google_ads",
-            external_task_id="wait_for_checks__fail_fenix_derived__firefox_android_clients__v1",
-        )
-
         checks__fail_fenix_derived__firefox_android_clients__v1_external.set_upstream(
             checks__fail_fenix_derived__firefox_android_clients__v1
         )
@@ -685,19 +679,6 @@
         depends_on_past=False,
     )
 
-    with TaskGroup(
-        "fenix_derived__funnel_retention_week_4__v1_external",
-    ) as fenix_derived__funnel_retention_week_4__v1_external:
-        ExternalTaskMarker(
-            task_id="bqetl_fivetran_google_ads__wait_for_fenix_derived__funnel_retention_week_4__v1",
-            external_dag_id="bqetl_fivetran_google_ads",
-            external_task_id="wait_for_fenix_derived__funnel_retention_week_4__v1",
-        )
-
-        fenix_derived__funnel_retention_week_4__v1_external.set_upstream(
-            fenix_derived__funnel_retention_week_4__v1
-        )
-
     firefox_android_clients = bigquery_etl_query(
         task_id="firefox_android_clients",
         destination_table="firefox_android_clients_v1",
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_firefox_ios.py /tmp/workspace/generated-sql/dags/bqetl_firefox_ios.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_firefox_ios.py	2025-07-09 09:37:54.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_firefox_ios.py	2025-07-09 09:41:54.000000000 +0000
@@ -90,6 +90,19 @@
     catchup=False,
 ) as dag:
 
+    wait_for_bigeye__firefox_ios_derived__new_profiles__v1 = ExternalTaskSensor(
+        task_id="wait_for_bigeye__firefox_ios_derived__new_profiles__v1",
+        external_dag_id="bqetl_mobile_kpi_metrics",
+        external_task_id="firefox_ios.bigeye__firefox_ios_derived__new_profiles__v1",
+        execution_delta=datetime.timedelta(days=-1, seconds=57600),
+        check_existence=True,
+        mode="reschedule",
+        poke_interval=datetime.timedelta(minutes=5),
+        allowed_states=ALLOWED_STATES,
+        failed_states=FAILED_STATES,
+        pool="DATA_ENG_EXTERNALTASKSENSOR",
+    )
+
     wait_for_copy_deduplicate_all = ExternalTaskSensor(
         task_id="wait_for_copy_deduplicate_all",
         external_dag_id="copy_deduplicate",
@@ -285,13 +298,6 @@
         "checks__fail_firefox_ios_derived__firefox_ios_clients__v1_external",
     ) as checks__fail_firefox_ios_derived__firefox_ios_clients__v1_external:
         ExternalTaskMarker(
-            task_id="bqetl_ios_campaign_reporting__wait_for_checks__fail_firefox_ios_derived__firefox_ios_clients__v1",
-            external_dag_id="bqetl_ios_campaign_reporting",
-            external_task_id="wait_for_checks__fail_firefox_ios_derived__firefox_ios_clients__v1",
-            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=57600)).isoformat() }}",
-        )
-
-        ExternalTaskMarker(
             task_id="bqetl_analytics_aggregations__wait_for_checks__fail_firefox_ios_derived__firefox_ios_clients__v1",
             external_dag_id="bqetl_analytics_aggregations",
             external_task_id="wait_for_checks__fail_firefox_ios_derived__firefox_ios_clients__v1",
@@ -495,13 +501,6 @@
         "firefox_ios_derived__clients_activation__v1_external",
     ) as firefox_ios_derived__clients_activation__v1_external:
         ExternalTaskMarker(
-            task_id="bqetl_ios_campaign_reporting__wait_for_firefox_ios_derived__clients_activation__v1",
-            external_dag_id="bqetl_ios_campaign_reporting",
-            external_task_id="wait_for_firefox_ios_derived__clients_activation__v1",
-            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=57600)).isoformat() }}",
-        )
-
-        ExternalTaskMarker(
             task_id="bqetl_analytics_aggregations__wait_for_firefox_ios_derived__clients_activation__v1",
             external_dag_id="bqetl_analytics_aggregations",
             external_task_id="wait_for_firefox_ios_derived__clients_activation__v1",
@@ -577,17 +576,13 @@
     )
 
     checks__fail_firefox_ios_derived__app_store_funnel__v1.set_upstream(
-        checks__fail_firefox_ios_derived__firefox_ios_clients__v1
+        wait_for_bigeye__firefox_ios_derived__new_profiles__v1
     )
 
     checks__fail_firefox_ios_derived__app_store_funnel__v1.set_upstream(
         firefox_ios_derived__app_store_funnel__v1
     )
 
-    checks__fail_firefox_ios_derived__app_store_funnel__v1.set_upstream(
-        firefox_ios_derived__clients_activation__v1
-    )
-
     checks__fail_firefox_ios_derived__baseline_clients_yearly__v1.set_upstream(
         firefox_ios_derived__baseline_clients_yearly__v1
     )
@@ -605,17 +600,13 @@
     )
 
     checks__warn_firefox_ios_derived__app_store_funnel__v1.set_upstream(
-        checks__fail_firefox_ios_derived__firefox_ios_clients__v1
+        wait_for_bigeye__firefox_ios_derived__new_profiles__v1
     )
 
     checks__warn_firefox_ios_derived__app_store_funnel__v1.set_upstream(
         firefox_ios_derived__app_store_funnel__v1
     )
 
-    checks__warn_firefox_ios_derived__app_store_funnel__v1.set_upstream(
-        firefox_ios_derived__clients_activation__v1
-    )
-
     checks__warn_firefox_ios_derived__firefox_ios_clients__v1.set_upstream(
         firefox_ios_derived__firefox_ios_clients__v1
     )
@@ -625,11 +616,7 @@
     )
 
     firefox_ios_derived__app_store_funnel__v1.set_upstream(
-        checks__fail_firefox_ios_derived__firefox_ios_clients__v1
-    )
-
-    firefox_ios_derived__app_store_funnel__v1.set_upstream(
-        firefox_ios_derived__clients_activation__v1
+        wait_for_bigeye__firefox_ios_derived__new_profiles__v1
     )
 
     firefox_ios_derived__attributable_clients__v1.set_upstream(
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_fivetran_google_ads.py /tmp/workspace/generated-sql/dags/bqetl_fivetran_google_ads.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_fivetran_google_ads.py	2025-07-09 09:37:56.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_fivetran_google_ads.py	2025-07-09 09:41:58.000000000 +0000
@@ -52,11 +52,11 @@
     catchup=False,
 ) as dag:
 
-    wait_for_checks__fail_fenix_derived__firefox_android_clients__v1 = (
-        ExternalTaskSensor(
-            task_id="wait_for_checks__fail_fenix_derived__firefox_android_clients__v1",
-            external_dag_id="bqetl_analytics_tables",
-            external_task_id="checks__fail_fenix_derived__firefox_android_clients__v1",
+    wait_for_bigeye__fenix_derived__new_profile_activation_clients__v1 = ExternalTaskSensor(
+        task_id="wait_for_bigeye__fenix_derived__new_profile_activation_clients__v1",
+        external_dag_id="bqetl_mobile_kpi_metrics",
+        external_task_id="fenix.bigeye__fenix_derived__new_profile_activation_clients__v1",
+        execution_delta=datetime.timedelta(days=-1, seconds=50400),
             check_existence=True,
             mode="reschedule",
             poke_interval=datetime.timedelta(minutes=5),
@@ -64,13 +64,12 @@
             failed_states=FAILED_STATES,
             pool="DATA_ENG_EXTERNALTASKSENSOR",
         )
-    )
 
-    wait_for_checks__fail_ltv_derived__fenix_client_ltv__v1 = ExternalTaskSensor(
-        task_id="wait_for_checks__fail_ltv_derived__fenix_client_ltv__v1",
-        external_dag_id="private_bqetl_ltv",
-        external_task_id="checks__fail_ltv_derived__fenix_client_ltv__v1",
-        execution_delta=datetime.timedelta(seconds=7200),
+    wait_for_bigeye__fenix_derived__retention__v1 = ExternalTaskSensor(
+        task_id="wait_for_bigeye__fenix_derived__retention__v1",
+        external_dag_id="bqetl_mobile_kpi_metrics",
+        external_task_id="fenix.bigeye__fenix_derived__retention__v1",
+        execution_delta=datetime.timedelta(days=-1, seconds=50400),
         check_existence=True,
         mode="reschedule",
         poke_interval=datetime.timedelta(minutes=5),
@@ -79,10 +78,11 @@
         pool="DATA_ENG_EXTERNALTASKSENSOR",
     )
 
-    wait_for_fenix_derived__funnel_retention_week_4__v1 = ExternalTaskSensor(
-        task_id="wait_for_fenix_derived__funnel_retention_week_4__v1",
-        external_dag_id="bqetl_analytics_tables",
-        external_task_id="fenix_derived__funnel_retention_week_4__v1",
+    wait_for_checks__fail_ltv_derived__fenix_client_ltv__v1 = ExternalTaskSensor(
+        task_id="wait_for_checks__fail_ltv_derived__fenix_client_ltv__v1",
+        external_dag_id="private_bqetl_ltv",
+        external_task_id="checks__fail_ltv_derived__fenix_client_ltv__v1",
+        execution_delta=datetime.timedelta(seconds=7200),
         check_existence=True,
         mode="reschedule",
         poke_interval=datetime.timedelta(minutes=5),
@@ -104,19 +104,6 @@
         pool="DATA_ENG_EXTERNALTASKSENSOR",
     )
 
-    wait_for_bigeye__fenix_derived__new_profile_activation_clients__v1 = ExternalTaskSensor(
-        task_id="wait_for_bigeye__fenix_derived__new_profile_activation_clients__v1",
-        external_dag_id="bqetl_mobile_kpi_metrics",
-        external_task_id="fenix.bigeye__fenix_derived__new_profile_activation_clients__v1",
-        execution_delta=datetime.timedelta(days=-1, seconds=50400),
-        check_existence=True,
-        mode="reschedule",
-        poke_interval=datetime.timedelta(minutes=5),
-        allowed_states=ALLOWED_STATES,
-        failed_states=FAILED_STATES,
-        pool="DATA_ENG_EXTERNALTASKSENSOR",
-    )
-
     wait_for_bigeye__firefox_ios_derived__attribution_clients__v1 = ExternalTaskSensor(
         task_id="wait_for_bigeye__firefox_ios_derived__attribution_clients__v1",
         external_dag_id="bqetl_mobile_kpi_metrics",
@@ -665,23 +652,23 @@
     )
 
     google_ads_derived__android_app_campaign_stats__v1.set_upstream(
-        wait_for_checks__fail_fenix_derived__firefox_android_clients__v1
+        wait_for_bigeye__fenix_derived__new_profile_activation_clients__v1
     )
 
     google_ads_derived__android_app_campaign_stats__v1.set_upstream(
-        checks__fail_google_ads_derived__ad_groups__v1
+        wait_for_bigeye__fenix_derived__retention__v1
     )
 
     google_ads_derived__android_app_campaign_stats__v1.set_upstream(
-        checks__fail_google_ads_derived__campaigns__v2
+        checks__fail_google_ads_derived__ad_groups__v1
     )
 
     google_ads_derived__android_app_campaign_stats__v1.set_upstream(
-        wait_for_checks__fail_ltv_derived__fenix_client_ltv__v1
+        checks__fail_google_ads_derived__campaigns__v2
     )
 
     google_ads_derived__android_app_campaign_stats__v1.set_upstream(
-        wait_for_fenix_derived__funnel_retention_week_4__v1
+        wait_for_checks__fail_ltv_derived__fenix_client_ltv__v1
     )
 
     google_ads_derived__android_app_campaign_stats__v1.set_upstream(
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_ios_campaign_reporting.py /tmp/workspace/generated-sql/dags/bqetl_ios_campaign_reporting.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_ios_campaign_reporting.py	2025-07-09 09:37:55.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_ios_campaign_reporting.py	2025-07-09 09:41:58.000000000 +0000
@@ -65,11 +65,22 @@
         pool="DATA_ENG_EXTERNALTASKSENSOR",
     )
 
-    wait_for_checks__fail_firefox_ios_derived__firefox_ios_clients__v1 = ExternalTaskSensor(
-        task_id="wait_for_checks__fail_firefox_ios_derived__firefox_ios_clients__v1",
-        external_dag_id="bqetl_firefox_ios",
-        external_task_id="checks__fail_firefox_ios_derived__firefox_ios_clients__v1",
-        execution_delta=datetime.timedelta(seconds=28800),
+    wait_for_bigeye__firefox_ios_derived__new_profile_activation_clients__v1 = ExternalTaskSensor(
+        task_id="wait_for_bigeye__firefox_ios_derived__new_profile_activation_clients__v1",
+        external_dag_id="bqetl_mobile_kpi_metrics",
+        external_task_id="firefox_ios.bigeye__firefox_ios_derived__new_profile_activation_clients__v1",
+        check_existence=True,
+        mode="reschedule",
+        poke_interval=datetime.timedelta(minutes=5),
+        allowed_states=ALLOWED_STATES,
+        failed_states=FAILED_STATES,
+        pool="DATA_ENG_EXTERNALTASKSENSOR",
+    )
+
+    wait_for_bigeye__firefox_ios_derived__retention__v1 = ExternalTaskSensor(
+        task_id="wait_for_bigeye__firefox_ios_derived__retention__v1",
+        external_dag_id="bqetl_mobile_kpi_metrics",
+        external_task_id="firefox_ios.bigeye__firefox_ios_derived__retention__v1",
         check_existence=True,
         mode="reschedule",
         poke_interval=datetime.timedelta(minutes=5),
@@ -117,19 +128,6 @@
         pool="DATA_ENG_EXTERNALTASKSENSOR",
     )
 
-    wait_for_firefox_ios_derived__clients_activation__v1 = ExternalTaskSensor(
-        task_id="wait_for_firefox_ios_derived__clients_activation__v1",
-        external_dag_id="bqetl_firefox_ios",
-        external_task_id="firefox_ios_derived__clients_activation__v1",
-        execution_delta=datetime.timedelta(seconds=28800),
-        check_existence=True,
-        mode="reschedule",
-        poke_interval=datetime.timedelta(minutes=5),
-        allowed_states=ALLOWED_STATES,
-        failed_states=FAILED_STATES,
-        pool="DATA_ENG_EXTERNALTASKSENSOR",
-    )
-
     apple_ads_external__ios_app_campaign_stats__v1 = bigquery_etl_query(
         task_id="apple_ads_external__ios_app_campaign_stats__v1",
         destination_table='ios_app_campaign_stats_v1${{ macros.ds_format(macros.ds_add(ds, -27), "%Y-%m-%d", "%Y%m%d") }}',
@@ -159,23 +157,23 @@
     )
 
     apple_ads_external__ios_app_campaign_stats__v1.set_upstream(
-        wait_for_checks__fail_firefox_ios_derived__firefox_ios_clients__v1
+        wait_for_bigeye__firefox_ios_derived__new_profile_activation_clients__v1
     )
 
     apple_ads_external__ios_app_campaign_stats__v1.set_upstream(
-        wait_for_checks__fail_ltv_derived__ios_state_values__v1
+        wait_for_bigeye__firefox_ios_derived__retention__v1
     )
 
     apple_ads_external__ios_app_campaign_stats__v1.set_upstream(
-        wait_for_checks__fail_revenue_derived__mobile_rpcs__v1
+        wait_for_checks__fail_ltv_derived__ios_state_values__v1
     )
 
     apple_ads_external__ios_app_campaign_stats__v1.set_upstream(
-        wait_for_firefox_ios_derived__client_ltv__v1
+        wait_for_checks__fail_revenue_derived__mobile_rpcs__v1
     )
 
     apple_ads_external__ios_app_campaign_stats__v1.set_upstream(
-        wait_for_firefox_ios_derived__clients_activation__v1
+        wait_for_firefox_ios_derived__client_ltv__v1
     )
 
     bigeye__apple_ads_external__ios_app_campaign_stats__v1.set_upstream(
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_mobile_kpi_metrics.py /tmp/workspace/generated-sql/dags/bqetl_mobile_kpi_metrics.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_mobile_kpi_metrics.py	2025-07-09 09:37:56.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_mobile_kpi_metrics.py	2025-07-09 09:42:01.000000000 +0000
@@ -593,6 +593,20 @@
         task_group=task_group_fenix,
     )
 
+    with TaskGroup(
+        "bigeye__fenix_derived__retention__v1_external", parent_group=task_group_fenix
+    ) as bigeye__fenix_derived__retention__v1_external:
+        ExternalTaskMarker(
+            task_id="bqetl_fivetran_google_ads__wait_for_bigeye__fenix_derived__retention__v1",
+            external_dag_id="bqetl_fivetran_google_ads",
+            external_task_id="wait_for_bigeye__fenix_derived__retention__v1",
+            execution_date="{{ (execution_date - macros.timedelta(seconds=36000)).isoformat() }}",
+        )
+
+        bigeye__fenix_derived__retention__v1_external.set_upstream(
+            bigeye__fenix_derived__retention__v1
+        )
+
     bigeye__firefox_ios_derived__attribution_clients__v1 = bigquery_bigeye_check(
         task_id="bigeye__firefox_ios_derived__attribution_clients__v1",
         table_id="moz-fx-data-shared-prod.firefox_ios_derived.attribution_clients_v1",
@@ -675,6 +689,12 @@
         parent_group=task_group_firefox_ios,
     ) as bigeye__firefox_ios_derived__new_profile_activation_clients__v1_external:
         ExternalTaskMarker(
+            task_id="bqetl_ios_campaign_reporting__wait_for_bigeye__firefox_ios_derived__new_profile_activation_clients__v1",
+            external_dag_id="bqetl_ios_campaign_reporting",
+            external_task_id="wait_for_bigeye__firefox_ios_derived__new_profile_activation_clients__v1",
+        )
+
+        ExternalTaskMarker(
             task_id="bqetl_marketing_analysis__wait_for_bigeye__firefox_ios_derived__new_profile_activation_clients__v1",
             external_dag_id="bqetl_marketing_analysis",
             external_task_id="wait_for_bigeye__firefox_ios_derived__new_profile_activation_clients__v1",
@@ -720,6 +740,21 @@
         task_group=task_group_firefox_ios,
     )
 
+    with TaskGroup(
+        "bigeye__firefox_ios_derived__new_profiles__v1_external",
+        parent_group=task_group_firefox_ios,
+    ) as bigeye__firefox_ios_derived__new_profiles__v1_external:
+        ExternalTaskMarker(
+            task_id="bqetl_firefox_ios__wait_for_bigeye__firefox_ios_derived__new_profiles__v1",
+            external_dag_id="bqetl_firefox_ios",
+            external_task_id="wait_for_bigeye__firefox_ios_derived__new_profiles__v1",
+            execution_date="{{ (execution_date - macros.timedelta(seconds=28800)).isoformat() }}",
+        )
+
+        bigeye__firefox_ios_derived__new_profiles__v1_external.set_upstream(
+            bigeye__firefox_ios_derived__new_profiles__v1
+        )
+
     bigeye__firefox_ios_derived__retention__v1 = bigquery_bigeye_check(
         task_id="bigeye__firefox_ios_derived__retention__v1",
         table_id="moz-fx-data-shared-prod.firefox_ios_derived.retention_v1",
@@ -732,6 +767,20 @@
         task_group=task_group_firefox_ios,
     )
 
+    with TaskGroup(
+        "bigeye__firefox_ios_derived__retention__v1_external",
+        parent_group=task_group_firefox_ios,
+    ) as bigeye__firefox_ios_derived__retention__v1_external:
+        ExternalTaskMarker(
+            task_id="bqetl_ios_campaign_reporting__wait_for_bigeye__firefox_ios_derived__retention__v1",
+            external_dag_id="bqetl_ios_campaign_reporting",
+            external_task_id="wait_for_bigeye__firefox_ios_derived__retention__v1",
+        )
+
+        bigeye__firefox_ios_derived__retention__v1_external.set_upstream(
+            bigeye__firefox_ios_derived__retention__v1
+        )
+
     fenix_derived__attribution_clients__v1 = bigquery_etl_query(
         task_id="fenix_derived__attribution_clients__v1",
         destination_table="attribution_clients_v1",
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/apple_ads_external/ios_app_campaign_stats_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/apple_ads_external/ios_app_campaign_stats_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/apple_ads_external/ios_app_campaign_stats_v1/metadata.yaml	2025-07-09 09:34:26.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/apple_ads_external/ios_app_campaign_stats_v1/metadata.yaml	2025-07-09 09:34:03.000000000 +0000
@@ -33,9 +33,9 @@
 references:
   query.sql:
   - moz-fx-data-shared-prod.apple_ads.ad_group_report
-  - moz-fx-data-shared-prod.firefox_ios.firefox_ios_clients
-  - moz-fx-data-shared-prod.firefox_ios.funnel_retention_week_4
-  - mozdata.ltv.firefox_ios_client_ltv
+  - moz-fx-data-shared-prod.firefox_ios.new_profile_activation_clients
+  - moz-fx-data-shared-prod.firefox_ios.retention
+  - moz-fx-data-shared-prod.ltv.firefox_ios_client_ltv
 monitoring:
   enabled: true
   collection: null
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/apple_ads_external/ios_app_campaign_stats_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/apple_ads_external/ios_app_campaign_stats_v1/query.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/apple_ads_external/ios_app_campaign_stats_v1/query.sql	2025-07-09 09:34:26.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/apple_ads_external/ios_app_campaign_stats_v1/query.sql	2025-07-09 09:30:18.000000000 +0000
@@ -38,12 +38,12 @@
     COUNTIF(clients.is_activated) AS activated,
     SUM(ltv.lifetime_value) AS lifetime_value,
   FROM
-    `mozdata.ltv.firefox_ios_client_ltv` AS ltv
+    `moz-fx-data-shared-prod.ltv.firefox_ios_client_ltv` AS ltv
   INNER JOIN
-    `moz-fx-data-shared-prod.firefox_ios.firefox_ios_clients` AS clients
-    USING (client_id, sample_id)
+    `moz-fx-data-shared-prod.firefox_ios.new_profile_activation_clients` AS clients
+    USING (client_id)
   WHERE
-    clients.channel = "release"
+    clients.normalized_channel = "release"
   GROUP BY
     `date`,
     campaign_id,
@@ -54,16 +54,16 @@
     first_seen_date AS `date`,
     CAST(REGEXP_EXTRACT(adjust_campaign, r' \((\d+)\)$') AS INT64) AS campaign_id,
     CAST(REGEXP_EXTRACT(adjust_ad_group, r' \((\d+)\)$') AS INT64) AS ad_group_id,
-    SUM(repeat_user) AS repeat_users,
-    SUM(retained_week_4) AS retained_week_4,
+    SUM(repeat_profiles) AS repeat_users,
+    SUM(retained_week_4_new_profiles) AS retained_week_4,
   FROM
-    -- TODO: we should update this to use retention_clients view instead
-    `moz-fx-data-shared-prod.firefox_ios.funnel_retention_week_4`
+    `moz-fx-data-shared-prod.firefox_ios.retention`
   WHERE
     {% if is_init() %}
-      submission_date <= CURRENT_DATE
+      metric_date <= CURRENT_DATE
     {% else %}
-      submission_date = @submission_date
+      metric_date = @submission_date
+      AND first_seen_date = @submission_date
     {% endif %}
   GROUP BY
     `date`,
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_ios/ad_activation_performance/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_ios/ad_activation_performance/view.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_ios/ad_activation_performance/view.sql	2025-07-09 09:34:27.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_ios/ad_activation_performance/view.sql	2025-07-09 09:30:18.000000000 +0000
@@ -8,7 +8,7 @@
     COUNTIF(is_activated) AS clients_activated,
     COUNT(*) AS new_profiles,
   FROM
-    `moz-fx-data-shared-prod.firefox_ios.firefox_ios_clients`
+    `moz-fx-data-shared-prod.firefox_ios.new_profile_activation_clients`
   WHERE
     is_activated IS NOT NULL
   GROUP BY
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_ios_derived/app_store_funnel_v1/checks.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_ios_derived/app_store_funnel_v1/checks.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_ios_derived/app_store_funnel_v1/checks.sql	2025-07-09 09:34:27.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_ios_derived/app_store_funnel_v1/checks.sql	2025-07-09 09:30:18.000000000 +0000
@@ -9,12 +9,12 @@
 #warn
 WITH fx_ios_count AS (
   SELECT
-    COUNT(*)
+    SUM(new_profiles)
   FROM
-    `moz-fx-data-shared-prod.firefox_ios.firefox_ios_clients`
+    `moz-fx-data-shared-prod.firefox_ios.new_profiles`
   WHERE
     first_seen_date = DATE_SUB(@submission_date, INTERVAL 7 DAY)
-    AND channel = "release"
+    AND normalized_channel = "release"
 ),
 new_profiles_count AS (
   SELECT
@@ -27,7 +27,7 @@
 SELECT
   IF(
     (SELECT * FROM fx_ios_count) - (SELECT * FROM new_profiles_count) <> 0,
-    ERROR("There's a 'new_profiles' mismatch between firefox_ios_clients and this funnel table"),
+    ERROR("There's a 'new_profiles' mismatch between new_profiles dataset and this funnel table"),
     NULL
   );
 
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_ios_derived/app_store_funnel_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_ios_derived/app_store_funnel_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_ios_derived/app_store_funnel_v1/metadata.yaml	2025-07-09 09:34:27.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_ios_derived/app_store_funnel_v1/metadata.yaml	2025-07-09 09:34:30.000000000 +0000
@@ -31,11 +31,11 @@
 references:
   checks.sql:
   - ..
-  - moz-fx-data-shared-prod.firefox_ios.firefox_ios_clients
+  - moz-fx-data-shared-prod.firefox_ios.new_profiles
   - moz-fx-data-shared-prod.firefox_ios_derived.app_store_funnel_v1
   query.sql:
   - moz-fx-data-shared-prod.app_store.firefox_app_store_territory_source_type_report
   - moz-fx-data-shared-prod.app_store.firefox_downloads_territory_source_type_report
-  - moz-fx-data-shared-prod.firefox_ios.firefox_ios_clients
+  - moz-fx-data-shared-prod.firefox_ios.new_profiles
   - moz-fx-data-shared-prod.static.country_names_v1
 require_column_descriptions: false
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_ios_derived/app_store_funnel_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_ios_derived/app_store_funnel_v1/query.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_ios_derived/app_store_funnel_v1/query.sql	2025-07-09 09:34:27.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_ios_derived/app_store_funnel_v1/query.sql	2025-07-09 09:30:18.000000000 +0000
@@ -48,13 +48,13 @@
 _new_profiles AS (
   SELECT
     first_seen_date AS `date`,
-    first_reported_country AS country,
-    COUNT(*) AS new_profiles,
+    country,
+    SUM(new_profiles) AS new_profiles,
   FROM
-    `moz-fx-data-shared-prod.firefox_ios.firefox_ios_clients`
+    `moz-fx-data-shared-prod.firefox_ios.new_profiles`
   WHERE
     first_seen_date = DATE_SUB(@submission_date, INTERVAL 7 DAY)
-    AND channel = "release"
+    AND normalized_channel = "release"
   GROUP BY
     `date`,
     country
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/google_ads_derived/android_app_campaign_stats_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/google_ads_derived/android_app_campaign_stats_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/google_ads_derived/android_app_campaign_stats_v1/metadata.yaml	2025-07-09 09:34:26.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/google_ads_derived/android_app_campaign_stats_v1/metadata.yaml	2025-07-09 09:34:11.000000000 +0000
@@ -38,8 +38,8 @@
   checks.sql:
   - ..
   query.sql:
-  - moz-fx-data-shared-prod.fenix.firefox_android_clients
-  - moz-fx-data-shared-prod.fenix.funnel_retention_week_4
+  - moz-fx-data-shared-prod.fenix.new_profile_activation_clients
+  - moz-fx-data-shared-prod.fenix.retention
   - moz-fx-data-shared-prod.google_ads_derived.ad_groups_v1
   - moz-fx-data-shared-prod.google_ads_derived.campaigns_v2
   - moz-fx-data-shared-prod.google_ads_derived.daily_ad_group_stats_v1
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/google_ads_derived/android_app_campaign_stats_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/google_ads_derived/android_app_campaign_stats_v1/query.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/google_ads_derived/android_app_campaign_stats_v1/query.sql	2025-07-09 09:34:26.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/google_ads_derived/android_app_campaign_stats_v1/query.sql	2025-07-09 09:30:18.000000000 +0000
@@ -20,12 +20,12 @@
 activations AS (
   SELECT
     first_seen_date AS `date`,
-    ad_group_id,
-    COUNTIF(activated) AS activated,
+    CAST(REGEXP_EXTRACT(adjust_ad_group, r' \((\d+)\)$') AS INT64) AS ad_group_id,
+    COUNTIF(is_activated) AS activated,
     COUNT(*) AS new_profiles,
     SUM(lifetime_value) AS lifetime_value,
   FROM
-    `moz-fx-data-shared-prod.fenix.firefox_android_clients`
+    `moz-fx-data-shared-prod.fenix.new_profile_activation_clients`
   JOIN
     `moz-fx-data-shared-prod.ltv.fenix_client_ltv`
     USING (client_id)
@@ -37,15 +37,16 @@
   SELECT
     first_seen_date AS `date`,
     CAST(REGEXP_EXTRACT(adjust_ad_group, r' \((\d+)\)$') AS INT64) AS ad_group_id,
-    SUM(repeat_user) AS repeat_users,
-    SUM(retained_week_4) AS retained_week_4
+    SUM(repeat_profiles) AS repeat_users,
+    SUM(retained_week_4_new_profiles) AS retained_week_4,
   FROM
-    `moz-fx-data-shared-prod.fenix.funnel_retention_week_4`
+    `moz-fx-data-shared-prod.fenix.retention`
   WHERE
     {% if is_init() %}
-      submission_date <= CURRENT_DATE
+      metric_date <= CURRENT_DATE
     {% else %}
-      submission_date = @submission_date
+      metric_date = @submission_date
+      AND first_seen_date = @submission_date
     {% endif %}
   GROUP BY
     `date`,

Link to full diff

kwindau · 2025-07-09T13:45:19Z

Would we be able to make these changes in a V2 table, that way you can backfill the V2 and compare with V1? Then once we are satisfied everything looks good and have run QA checks, we can swap the views to point at the new V2 tables and deprecate the V1 tables at that time?

soGaussian · 2025-07-10T13:40:12Z

The iOS app campaign table is not needed right now.

kik-kik · 2025-07-10T13:53:10Z

The iOS app campaign table is not needed right now.

@soGaussian are you saying sql/moz-fx-data-shared-prod/apple_ads_external/ios_app_campaign_stats_v1/query.sql should be deprecated?

kik-kik · 2025-07-10T15:55:02Z

The iOS app campaign table is not needed right now.

@soGaussian are you saying sql/moz-fx-data-shared-prod/apple_ads_external/ios_app_campaign_stats_v1/query.sql should be deprecated?

ok, I will create a follow-up ticket then to deprecate that dataset.

kik-kik · 2025-07-15T09:37:28Z

sql/moz-fx-data-shared-prod/apple_ads_external/ios_app_campaign_stats_v1/query.sql

This has now been marked as deprecated.

kik-kik · 2025-07-15T10:20:35Z

sql/moz-fx-data-shared-prod/firefox_ios/ad_activation_performance/view.sql

Found some differences, dived into a specific client and it appears the new table is reporting activation field correctly. The old dataset not (as per the activation logic). Hard to say why exactly, but this could be because the underlaying dataset we rely on for calculating this changes somehow or potentially the definition of activation changed since when the activation information for that client was generated.

We only see differences for dates older than (including) 2024 August.

ok, so historically there's definitely bigger differences. And seeing that the aggregation is generated at runtime using client level info so even historically I'd expect more alignment since both datasets should be Shredded so this should not be a factor for these differences. The only thing I can really think of is that since the differences between new_profile count and activation count show up starting August 2024 is that the definition of activation changes around then and this is the definition used by the new dataset (since we seem to have match since then).

Another theory would be that the new dataset was created and backfilled around the time we see differences and historical data differs because at the backfill time the underlaying data (used to create new profiles and activation datasets) may have changed which is why historically we see differences and going forward we do not as both datasets would be updated around the same time.

@soGaussian would you be the main consumer of this dataset?

This comment has been minimized.

Sign in to view

kik-kik requested a review from kwindau July 2, 2025 13:30

kik-kik marked this pull request as ready for review July 2, 2025 13:30

This comment has been minimized.

Sign in to view

kik-kik self-assigned this Jul 9, 2025

kik-kik added the enhancement New feature or request label Jul 9, 2025

kik-kik added 4 commits July 9, 2025 11:28

feat: replace firefox_ios_clients references inside ios_app_campaign_…

0447484

…stats_v1 with new_profile_clients and update retention source

feat: update ad_activation_performance view to reference the new new_…

4eebf53

…profiles dataset

feat: update app_store_funnel to use the new_profiles table for new p…

a92289c

…rofile stats

update firefox_android_clients and retention_week_4 references to use…

186963f

… new datatasets so that the other two can be deprecated

kik-kik force-pushed the feat/stores-update-clients-references-to-new_profile_clients branch from 0e6cf1c to 186963f Compare July 9, 2025 09:28

kik-kik commented Jul 15, 2025

View reviewed changes

sql/moz-fx-data-shared-prod/apple_ads_external/ios_app_campaign_stats_v1/query.sql

Copy link

Contributor Author

kik-kik Jul 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has now been marked as deprecated.

kik-kik commented Jul 15, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(): stores update clients references to new profile clients #7720

feat(): stores update clients references to new profile clients #7720

kik-kik commented Jul 2, 2025 •

edited

Loading

Uh oh!

This comment has been minimized.

This comment has been minimized.

dataops-ci-bot commented Jul 9, 2025

Uh oh!

kwindau commented Jul 9, 2025

Uh oh!

soGaussian commented Jul 10, 2025

Uh oh!

kik-kik commented Jul 10, 2025

Uh oh!

kik-kik commented Jul 10, 2025

Uh oh!

kik-kik Jul 15, 2025

Uh oh!

kik-kik Jul 15, 2025

Uh oh!

kik-kik Jul 15, 2025

Uh oh!

kik-kik Jul 15, 2025

Uh oh!

Uh oh!

feat(): stores update clients references to new profile clients #7720

Are you sure you want to change the base?

feat(): stores update clients references to new profile clients #7720

Conversation

kik-kik commented Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

feat(): stores update clients references to new profile clients

Uh oh!

This comment has been minimized.

This comment has been minimized.

dataops-ci-bot commented Jul 9, 2025

Integration report for "update firefox_android_clients and retention_week_4 references to use new datatasets so that the other two can be deprecated"

sql.diff

Uh oh!

kwindau commented Jul 9, 2025

Uh oh!

soGaussian commented Jul 10, 2025

Uh oh!

kik-kik commented Jul 10, 2025

Uh oh!

kik-kik commented Jul 10, 2025

Uh oh!

kik-kik Jul 15, 2025

Choose a reason for hiding this comment

Uh oh!

kik-kik Jul 15, 2025

Choose a reason for hiding this comment

Uh oh!

kik-kik Jul 15, 2025

Choose a reason for hiding this comment

Uh oh!

kik-kik Jul 15, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kik-kik commented Jul 2, 2025 •

edited

Loading

`sql.diff`