Standardize unioning macros to use fivetran_utils versions

.gitignore

@@ -71,3 +71,4 @@ env/
 env.bak/
 venv/
 venv.bak/
+worktrees/

CHANGELOG.md

@@ -1,3 +1,19 @@
+# dbt_stripe v1.8.0
+
+[PR #152](https://github.com/fivetran/dbt_stripe/pull/152) includes the following updates:
+
+## Schema/Data Changes (--full-refresh required after upgrading)
+**1 total change • 2 possible breaking change**
+
+| Data Model(s) | Change type | Old | New | Notes |
+| ------------- | ----------- | --- | --- | ----- |
+| All models | `source_relation` column (when using a single stripe schema) | Empty string (`''`) | `<database>.<schema>` |  |
+| All models | `source_relation` column (when using `stripe_union_schemas` or `stripe_union_databases` schema) | `<database>.<schema>` | only `<schema>` or `<database>` respectively | Updated to align with standard applied to other Fivetran packages. |
+
+## Under the Hood
+- Migrates the `union_connections`, `apply_source_relation`, and `partition_by_source_relation` macros to the `dbt_fivetran_utils` package.
+- Adds the `fivetran_using_source_casing` variable for case-sensitive destination support. When enabled, downstream transformations respect source casing to ensure consistent results. See the [Additional Configurations](https://github.com/fivetran/dbt_stripe/#source-casing-for-case-sensitive-destinations) section of the README for details.
+
 # dbt_stripe v1.7.0
 
 [PR #147](https://github.com/fivetran/dbt_stripe/pull/147) includes the following updates:

README.md

@@ -80,7 +80,7 @@ Include the following stripe package version in your `packages.yml` file:
 ```yaml
 packages:
   - package: fivetran/stripe
-    version: [">=1.7.0", "<1.8.0"]
+    version: [">=1.8.0", "<1.9.0"]
 ```
 > All required sources and staging models are now bundled into this transformation package. Do not include `fivetran/stripe_source` in your `packages.yml` since this package has been deprecated.
 
@@ -93,43 +93,6 @@ dispatch:
 ```
 
 ### Define database and schema variables
-By default, this package runs using your destination and the `stripe` schema. If this is not where your stripe data is (for example, if your stripe schema is named `stripe_fivetran`), add the following configuration to your root `dbt_project.yml` file:
-
-```yml
-vars:
-    stripe_database: your_destination_name
-    stripe_schema: your_schema_name 
-```
-
-### Disable models for non-existent sources
-This package takes into consideration that not every Stripe account utilizes the `invoice`, `invoice_line_item`, `payment_method`, `payment_method_card`, `plan`, `price`, `subscription`, `coupon`, `subscription_discount`, `transfer`, `payout`, `payout_balance_transaction`, or `credit_note` features, and allows you to disable the corresponding functionality. By default, all variables' values are assumed to be `true` with the exception of `credit_note`. Add variables for only the tables you want to disable or enable respectively:
-
-```yml
-# dbt_project.yml
-
-...
-vars:
-    stripe__using_invoices:                False  #Disable if you are not using the invoice and invoice_line_item tables.
-    stripe__using_payment_method:          False  #Disable if you are not using the payment_method and payment_method_card tables.
-    stripe__using_subscriptions:           False  #Disable if you are not using the subscription, subscription_item, and plan/price tables.
-    stripe__using_coupons:                 False  #Disable if you are not using coupon codes to apply discounts.
-    stripe__using_subscription_discounts:  False  #Disable if you are not using the subscription_discount table.
-    stripe__using_credit_notes:            True   #Enable if you are using the credit note tables.
-    stripe__using_transfers:               False  #Disable to turn off the transfer table temporarily.
-    stripe__using_payouts:                 False  #Disable to turn off the payout or payout_balance_transaction table temporarily.
-```
-### (Optional) Additional configurations
-<details open><summary>Expand to view configurations</summary>
-
-#### Enabling Standardized Billing Model
-This package contains the `stripe__line_item_enhanced` model which constructs a comprehensive, denormalized analytical table that enables reporting on key revenue, subscription, customer, and product metrics from your billing platform. It's designed to align with the schema of the `*__line_item_enhanced` model found in Recurly, Recharge, Stripe, Shopify, and Zuora, offering standardized reporting across various billing platforms. To see the kinds of insights this model can generate, explore example visualizations in the [Fivetran Billing Model Streamlit App](https://fivetran-billing-model.streamlit.app/). This model is enabled by default. To disable it, set the `stripe__standardized_billing_model_enabled` variable to `false` in your `dbt_project.yml`:
-
-```yml
-vars:
-  stripe__standardized_billing_model_enabled: false # true by default.
-```
-> All required sources and staging models are now bundled into this transformation package. Do not include `fivetran/stripe_source` in your `packages.yml` since this package has been deprecated.
-
 #### Option A: Single connection
 By default, this package runs using your destination and the `stripe` schema. If this is not where your Stripe data is (for example, if your Stripe schema is named `stripe_fivetran`), add the following configuration to your root `dbt_project.yml` file:
 
@@ -159,41 +122,41 @@ vars:
         name: connection_2_source_name
 ```
 
-> Previous versions of this package employed two separate, mutually exclusive variables for unioning: `union_schemas` and `union_databases`. While these variables are still supported, `stripe_sources` is the recommended variable to configure.
+> Previous versions of this package employed two separate, mutually exclusive variables for unioning: `stripe_union_schemas` and `stripe_union_databases`. While these variables are still supported, `stripe_sources` is the recommended variable to configure.
 
-##### Recommended: Incorporate unioned sources into DAG
-> *If you are running the package through [Fivetran Transformations for dbt Core™](https://fivetran.com/docs/transformations/dbt#transformationsfordbtcore), the below step is necessary in order to synchronize model runs with your Stripe connections. Alternatively, you may choose to run the package through Fivetran [Quickstart](https://fivetran.com/docs/transformations/quickstart), which would create separate sets of models for each Stripe source rather than one set of unioned models.*
+#### Optional: Incorporate unioned sources into DAG
 
-By default, this package defines one single-connection source, called `stripe`, which will be disabled if you are unioning multiple connections. This means that your DAG will not include your Stripe sources, though the package will run successfully.
+If you use [Fivetran Transformations for dbt Core™](https://fivetran.com/docs/transformations/dbt#transformationsfordbtcore) and are unioning multiple Stripe connections, you can define your sources in a property `.yml` file, [using this as a template](https://github.com/fivetran/dbt_stripe/blob/main/models/staging/src_stripe.yml). Set the variable `has_defined_sources: true` under the Stripe namespace in your `dbt_project.yml`. Otherwise, your Stripe connections won't appear in your DAG. See the `union_connections` macro [documentation](https://github.com/fivetran/dbt_fivetran_utils/tree/releases/v0.4.latest#optional-union-connections-defined-sources-configuration) for full configuration details.
 
-To properly incorporate all of your Stripe connections into your project's DAG:
-1. Define each of your sources in a `.yml` file in your project. Utilize the following template for the `source`-level configurations, and, **most importantly**, copy and paste the table and column-level definitions from the package's `src_stripe.yml` [file](https://github.com/fivetran/dbt_stripe/blob/main/models/staging/src_stripe.yml).
+### Disable models for non-existent sources
+This package takes into consideration that not every Stripe account utilizes the `invoice`, `invoice_line_item`, `payment_method`, `payment_method_card`, `plan`, `price`, `subscription`, `coupon`, `subscription_discount`, `transfer`, `payout`, `payout_balance_transaction`, or `credit_note` features, and allows you to disable the corresponding functionality. By default, all variables' values are assumed to be `true` with the exception of `credit_note`. Add variables for only the tables you want to disable or enable respectively:
 
 ```yml
-# a .yml file in your root project models folder
-
-version: 2
-
-sources:
-  - name: <name> # ex: Should match name in stripe_sources
-    schema: <schema_name>
-    database: <database_name>
-    loader: fivetran
-    config:
-      loaded_at_field: _fivetran_synced
-      freshness: # feel free to adjust to your liking
-        warn_after: {count: 72, period: hour}
-        error_after: {count: 168, period: hour}
-
-    tables: # copy and paste from stripe/models/staging/src_stripe.yml - see https://support.atlassian.com/bitbucket-cloud/docs/yaml-anchors/ for how to use anchors to only do so once
+# dbt_project.yml
+
+...
+vars:
+    stripe__using_invoices:                False  #Disable if you are not using the invoice and invoice_line_item tables.
+    stripe__using_payment_method:          False  #Disable if you are not using the payment_method and payment_method_card tables.
+    stripe__using_subscriptions:           False  #Disable if you are not using the subscription, subscription_item, and plan/price tables.
+    stripe__using_coupons:                 False  #Disable if you are not using coupon codes to apply discounts.
+    stripe__using_subscription_discounts:  False  #Disable if you are not using the subscription_discount table.
+    stripe__using_credit_notes:            True   #Enable if you are using the credit note tables.
+    stripe__using_transfers:               False  #Disable to turn off the transfer table temporarily.
+    stripe__using_payouts:                 False  #Disable to turn off the payout or payout_balance_transaction table temporarily.
 ```
-2. Set the `has_defined_sources` variable (scoped to the `stripe` package) to `True` in your root project, like such:
+### (Optional) Additional configurations
+<details open><summary>Expand to view configurations</summary>
+
+#### Enabling Standardized Billing Model
+This package contains the `stripe__line_item_enhanced` model which constructs a comprehensive, denormalized analytical table that enables reporting on key revenue, subscription, customer, and product metrics from your billing platform. It's designed to align with the schema of the `*__line_item_enhanced` model found in Recurly, Recharge, Stripe, Shopify, and Zuora, offering standardized reporting across various billing platforms. To see the kinds of insights this model can generate, explore example visualizations in the [Fivetran Billing Model Streamlit App](https://fivetran-billing-model.streamlit.app/). This model is enabled by default. To disable it, set the `stripe__standardized_billing_model_enabled` variable to `false` in your `dbt_project.yml`:
+
 ```yml
-# dbt_project.yml
 vars:
-  stripe:
-    has_defined_sources: true
+  stripe__standardized_billing_model_enabled: false # true by default.
 ```
+> All required sources and staging models are now bundled into this transformation package. Do not include `fivetran/stripe_source` in your `packages.yml` since this package has been deprecated.
+
 #### Considerations: Unioning Multiple Schemas
 Please note, If the source table is not found in any of the provided schemas/databases, union_data will return a completely empty table (ie limit 0) with just one string column (_dbt_source_relation). A compiler warning message will be output, highlighting that the expected source table was not found and its respective staging model is empty. The compiler warning can be turned off by the end user by setting the `fivetran__remove_empty_table_warnings` variable to True.
 ```yml
@@ -355,6 +318,14 @@ vars:
     stripe_<default_source_table_name>_identifier: your_table_name 
 ```
 
+#### Source casing for case-sensitive destinations
+By default, the package applies case-insensitive comparisons when resolving `source_relation` values. If your destination is case-sensitive and you want downstream transformations to respect the exact casing of your source database and schema names, set the following variable:
+
+```yml
+vars:
+    fivetran_using_source_casing: true
+```
+
 </details>
 
 ### (Optional) Orchestrate your models with Fivetran Transformations for dbt Core™
@@ -395,4 +366,4 @@ We highly encourage and welcome contributions to this package. Learn how to cont
 
 ## Are there any resources available?
 - If you have questions or want to reach out for help, see the [GitHub Issue](https://github.com/fivetran/dbt_stripe/issues/new/choose) section to find the right avenue of support for you.
-- If you would like to provide feedback to the dbt package team at Fivetran or would like to request a new dbt package, fill out our [Feedback Form](https://www.surveymonkey.com/r/DQ7K7WW).
+- If you would like to provide feedback to the dbt package team at Fivetran or would like to request a new dbt package, fill out our [Feedback Form](https://www.surveymonkey.com/r/DQ7K7WW).

dbt_project.yml

@@ -1,7 +1,7 @@
 config-version: 2
 name: 'stripe'
 
-version: '1.7.0'
+version: '1.8.0'
 require-dbt-version: [">=1.3.0", "<3.0.0"]
 
 models:

integration_tests/ci/test_scenarios.yml

@@ -46,4 +46,10 @@ test_scenarios:
       stripe__convert_values: true
       stripe__using_credit_notes: true
       stripe__standardized_billing_model_enabled: true
-    include_incremental: false
+    include_incremental: false
+
+  - name: "MDLS"
+    vars:
+      fivetran_using_source_casing: true
+    include_incremental: false
+    include_dbt_seed: true

integration_tests/dbt_project.yml

@@ -2,7 +2,7 @@ config-version: 2
 
 name: 'stripe_integration_tests'
 
-version: '1.7.0'
+version: '1.8.0'
 
 profile: 'integration_tests'
 
@@ -16,7 +16,6 @@ models:
 vars:
   # enable when generating docs
   # stripe__using_credit_notes: true # @docs-include
-
   stripe_schema: stripe_integrations_tests
   stripe:
     stripe_account_identifier: "account_data"
@@ -52,8 +51,8 @@ vars:
 seeds:
   +docs:
     show: False
+  +quote_columns: "{{ target.type in ('redshift','postgres') or var('fivetran_using_source_casing', false) }}"
   stripe_integration_tests:
-    +quote_columns: "{{ true if target.type in ('redshift','postgres') else false }}"
     account_data:
       +column_types:
         _fivetran_synced: timestamp
@@ -190,41 +189,41 @@ seeds:
         _fivetran_synced: timestamp
         created: timestamp
     discount_data:
-      +enabled: "{{ true if target.type not in ('snowflake', 'postgres') else false }}"
+      +enabled: "{{ target.type not in ('snowflake', 'postgres') or var('fivetran_using_source_casing', false) }}"
       +column_types:
         _fivetran_synced: timestamp
         end: timestamp
         amount: "{{ 'bigint' if target.type in ('redshift','postgres') else 'INT64' if target.type == 'bigquery' else 'bigint'}}"
         invoice_id: "{{ 'varchar(500)' if target.type in ('redshift','postgres') else 'string'}}"
     discount_data_snowflake:
       +alias: discount_data
-      +enabled: "{{ true if target.type == 'snowflake' else false }}"
+      +enabled: "{{ target.type == 'snowflake' and not var('fivetran_using_source_casing', false) }}"
       +column_types:
         _fivetran_synced: timestamp
         invoice_id: "{{ 'varchar(500)' if target.type in ('redshift','postgres') else 'string'}}"
     discount_data_postgres:
       +alias: discount_data
-      +enabled: "{{ true if target.type == 'postgres' else false }}"
+      +enabled: "{{ target.type == 'postgres' and not var('fivetran_using_source_casing', false) }}"
       +column_types:
         _fivetran_synced: timestamp
         end: timestamp
         amount: "{{ 'bigint' if target.type in ('redshift','postgres') else 'INT64' if target.type == 'bigquery' else 'bigint'}}"
         invoice_id: "{{ 'varchar(500)' if target.type in ('redshift','postgres') else 'string'}}"
     subscription_discount_data:
-      +enabled: "{{ true if target.type not in ('snowflake', 'postgres') else false }}"
+      +enabled: "{{ target.type not in ('snowflake', 'postgres') or var('fivetran_using_source_casing', false) }}"
       +column_types:
         _fivetran_synced: timestamp
         end: timestamp
         invoice_id: "{{ 'varchar(500)' if target.type in ('redshift','postgres') else 'string'}}"
     subscription_discount_data_snowflake:
       +alias: subscription_discount_data
-      +enabled: "{{ true if target.type == 'snowflake' else false }}"
+      +enabled: "{{ target.type == 'snowflake' and not var('fivetran_using_source_casing', false) }}"
       +column_types:
         _fivetran_synced: timestamp
         invoice_id: "{{ 'varchar(500)' if target.type in ('redshift','postgres') else 'string'}}"
     subscription_discount_data_postgres:
       +alias: subscription_discount_data
-      +enabled: "{{ true if target.type == 'postgres' else false }}"
+      +enabled: "{{ target.type == 'postgres' and not var('fivetran_using_source_casing', false) }}"
       +column_types:
         _fivetran_synced: timestamp
         end: timestamp
@@ -238,4 +237,4 @@ dispatch:
 flags:
   send_anonymous_usage_stats: False
 
-  use_colors: True
+  use_colors: True

macros/staging/get_discount_columns.sql

@@ -17,7 +17,7 @@
     {"name": "subscription_id", "datatype": dbt.type_string() }
 ] %}
 
-{% if target.type == 'snowflake' %}
+{% if target.type == 'snowflake' and not var('fivetran_using_source_casing', false) %}
     {{ columns.append({"name": "END", "datatype": dbt.type_string(), "quote": True, "alias": "end_at"}) }},
     {{ columns.append({"name": "START", "datatype": dbt.type_string(), "quote": True, "alias": "start_at"}) }}
 {% else %}

macros/staging/get_subscription_discount_columns.sql

@@ -11,7 +11,7 @@
     {"name": "subscription_id", "datatype": dbt.type_string()}
 ] %}
 
-{% if target.type == 'snowflake' %}
+{% if target.type == 'snowflake' and not var('fivetran_using_source_casing', false) %}
     {{ columns.append({"name": "END", "datatype": dbt.type_string(), "quote": True, "alias": "end_at"}) }},
     {{ columns.append({"name": "START", "datatype": dbt.type_string(), "quote": True, "alias": "start_at"}) }}
 {% else %}