Data Review: FxA/SubPlat Glean Tracking Metrics

[david1alvarez]

1) What questions will you answer with this data?

• How many customers do we support?
• When do the customers drop off during the checkout process?
• Are customers dropping off because of code or technical failures?
• Are there devices we are failing to support appropriately?
• What marketing channels are most successful at bringing in customers?
• What is the interest in our different offerings?

2) Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements? Some example responses:

• Mozilla needs to understand the user experience of the subscription platform process in order to eliminate pain points in the process, and to gain the business intelligence needed to determine how well marketing campaigns are working.

3) What alternative methods did you consider to answer these questions? Why were they not sufficient?

Sentry can be used to raise errors to the Sentry dashboard, and Stripe can be used to determine the ratios of cart creations, and payment completions. Neither of these platforms record the necessary information to understand the experience of the user, nor to identify the success-rate of marketing campaigns.

4) Can current instrumentation answer these questions?

No. The Subscription Platform 3.0 is not in active use yet, and does not have prior metrics collections.

5) List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories found on the Mozilla wiki.

Note that the data steward reviewing your request will characterize your data collection based on the highest (and most sensitive) category.

Measurement Description	Data Collection Category	Tracking Bug #	Measurement Name
Checkout engage event	Category 2	#FXA-7531	pay_setup.engage
Checkout fail event	Category 2	#FXA-7531	pay_setup.fail
Checkout submit event	Category 2	#FXA-7531	pay_setup.submit
Checkout engage event	Category 2	#FXA-7531	pay_setup.success
Checkout view event	Category 2	#FXA-7531	pay_setup.view
Whether the checkout flow is for new users or existing users. One of “with-account” or “without-account"	Category 2	#FXA-7531	subscription.checkout_type
Currency of a subscription used at checkout	Category 2	#FXA-7531	subscription.currency
The error id, if any, encountered for a fxa_pay_setup - fail event	Category 1	#FXA-7531	subscription.error_id
Interval of a subscription used at checkout	Category 2	#FXA-7531	subscription.interval
ID of the offering a customer subscribed to	Category 2	#FXA-7531	subscription.offering_id
The third party service ultimately processing a user’s payments (e.g. 'stripe')	Category 2	#FXA-7531	subscription.payment_provider
Plan ID of a subscription	Category 2	#FXA-7531	subscription.plan_id
Product ID of a subscription	Category 2	#FXA-7531	subscription.product_id
The Stripe customer-facing promotion code applied, if any	Category 2	#FXA-7531	subscription.promotion_code
Comma-separated list of Stripe price/plan IDs the user is already subscribed to	Category 2	#FXA-7531	subscription.subscribed_plan_ids

6) Please provide a link to the documentation for this data collection which describes the ultimate data set in a public, complete, and accurate way.

This collection is Glean so is documented in the Glean Dictionary.

Note: The full discussion from the original data review request, which notably included the category 4 data, FxA's account.user_id field (no longer included in the dataset), can be found at this internally-available google drive document.

7) How long will this data be collected? Choose one of the following:

This collection will be collected permanently.
[email protected], [email protected] will be responsible for the permanent collections.

8) What populations will you measure?

All channels, countries, and locales. No filters.

9) If this data collection is default on, what is the opt-out mechanism for users?

These collections are Glean. The opt-out can be found in the product's preferences. Session data will be recorded up until a user signs in with an account that has opted-out of metrics collection, or until they opt-out of collection, whichever comes first.

10) Please provide a general description of how you will analyze this data.

• We will work with DS and DE to create dashboards for analysis of the metrics.
• The data will be analyzed as events, rather than reviewed as aggregate data.

11) Where do you intend to share the results of your analysis?

• The results of the analysis will be available to internal teams and NDA'd Mozillians only.

12) Is there a third-party tool (i.e. not Glean or Telemetry) that you are proposing to use for this data collection? If so:

No.

┆Issue is synchronized with this Jira Task

[jaredhirsch]

data-review+, you're good to proceed 👍

---

Data Review Form (to be filled by Data Stewards)

1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?

Yes, accounts Glean data is exposed in the Glean dictionary.

2. Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, the accounts website provides opt out controls.

3. If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, [email protected] and [email protected] will monitor.

4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2.

Note that Stripe supports promo codes that are customized to a specific user. Based on discussion outside this thread, my understanding is that we are not customizing promotions. If this changes, we'll want to revisit the classification for the promotion_code probe.

5. Is the data collection request for default-on or default-off?

Default-on.

6. Does the instrumentation include the addition of any new identifiers?

No.

7. Is the data collection covered by the existing Firefox privacy notice?

Yes.

8. Does the data collection use a third-party collection tool?

No.