Skip to content

Commit ee3b0bd

Browse files
fivetran-savagefivetran-joemarkiewiczgithub-actions[bot]fivetran-avinash
authored
Feature/new union macro metadata fields and variable configs (#140)
* feature/new_union_macro_metadata_fields_and_variable_configs * add_variable_configs_to_downstream_models * add_charge_medatada_fields_to_end_model * add_union_macro_files * add_union_macro_files * add_metadata_macro * update_charge_seed_column * update_integration_test_seed_column_definitions * add_seed_column_details * sdd_column_definitions * update_ci_folder * update_changelog_and_readme * update_column_definitions * add_coupon_to_yml * update_quickstart_yml * change_macro_link * add_transfers_var_config * update_readme * change_order_of_tests * add_vars_to_int_models * adjust_jinja * update_readme * restore_changelog_history * Restore changelog history * Apply suggestions from code review Co-authored-by: Joe Markiewicz <74217849+fivetran-joemarkiewicz@users.noreply.github.com> * Update README * Update CHANGELOG * Update CHANGELOG.md Co-authored-by: Joe Markiewicz <74217849+fivetran-joemarkiewicz@users.noreply.github.com> * Update CHANGELOG.md Co-authored-by: Joe Markiewicz <74217849+fivetran-joemarkiewicz@users.noreply.github.com> * Update CHANGELOG.md Co-authored-by: Joe Markiewicz <74217849+fivetran-joemarkiewicz@users.noreply.github.com> * Update macros/select_metadata_columns.sql Co-authored-by: Joe Markiewicz <74217849+fivetran-joemarkiewicz@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Joe Markiewicz <74217849+fivetran-joemarkiewicz@users.noreply.github.com> * Update README * Generate dbt docs via GitHub Actions * Apply suggestions from code review Co-authored-by: Avinash Kunnath <108772760+fivetran-avinash@users.noreply.github.com> Co-authored-by: Joe Markiewicz <74217849+fivetran-joemarkiewicz@users.noreply.github.com> * Update CHANGELOG * Apply suggestions from code review Co-authored-by: Joe Markiewicz <74217849+fivetran-joemarkiewicz@users.noreply.github.com> * Remove select_metadata_columns macro from customer_overview * Update indentation * Enhance CHANGELOG with model and Quickstart updates * Update CHANGELOG.md Co-authored-by: Avinash Kunnath <108772760+fivetran-avinash@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Avinash Kunnath <108772760+fivetran-avinash@users.noreply.github.com> * Fix indentation * update_changelog * Update CHANGELOG.md Co-authored-by: Avinash Kunnath <108772760+fivetran-avinash@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Avinash Kunnath <108772760+fivetran-avinash@users.noreply.github.com> * Generate dbt docs via GitHub Actions --------- Co-authored-by: Joe Markiewicz <74217849+fivetran-joemarkiewicz@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Avinash Kunnath <108772760+fivetran-avinash@users.noreply.github.com>
1 parent 922ae62 commit ee3b0bd

81 files changed

Lines changed: 1093 additions & 1436 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.buildkite/scripts/run_models.sh

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,9 +20,9 @@ dbt seed --target "$db" --full-refresh
2020
dbt source freshness --target "$db" || echo "...Only verifying freshness runs…"
2121
dbt run --target "$db" --full-refresh
2222
dbt test --target "$db"
23-
dbt run --vars '{stripe__using_invoices: false, stripe__using_payment_method: false, stripe__using_subscriptions: false, stripe_timezone: "America/New_York", stripe__using_subscription_history: false, stripe__using_price: false, stripe__convert_values: true, stripe__using_credit_notes: true, stripe__standardized_billing_model_enabled: false}' --target "$db"
23+
dbt run --vars '{stripe__using_payouts: false, stripe__using_transfers: false, stripe__using_invoices: false, stripe__using_payment_method: false, stripe__using_subscriptions: false, stripe_timezone: "America/New_York", stripe__using_subscription_history: false, stripe__using_price: false, stripe__convert_values: true, stripe__using_credit_notes: true, stripe__standardized_billing_model_enabled: false}' --target "$db"
2424
dbt test --target "$db"
25-
dbt run --vars '{stripe__using_invoices: true, stripe__using_payment_method: false, stripe__using_subscriptions: false, stripe_timezone: "America/New_York", stripe__using_subscription_history: false, stripe__using_price: false, stripe__convert_values: true, stripe__using_credit_notes: true, stripe__standardized_billing_model_enabled: true}' --target "$db"
25+
dbt run --vars '{stripe__using_payouts: false, stripe__using_invoices: true, stripe__using_payment_method: false, stripe__using_subscriptions: false, stripe_timezone: "America/New_York", stripe__using_subscription_history: false, stripe__using_price: false, stripe__convert_values: true, stripe__using_credit_notes: true, stripe__standardized_billing_model_enabled: true}' --target "$db"
2626
dbt test --target "$db"
2727

2828
dbt run-operation fivetran_utils.drop_schemas_automation --target "$db"

.quickstart/quickstart.yml

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,11 @@ table_variables:
1616
stripe__using_credit_notes:
1717
- credit_note_line_item
1818
- credit_note
19+
stripe__using_transfers:
20+
- transfer
21+
stripe__using_payouts:
22+
- payout
23+
- payout_balance_transaction
1924
stripe__using_coupons:
2025
- coupon
2126

@@ -47,9 +52,24 @@ public_models: [
4752
supported_vars:
4853
stripe__customer_metadata:
4954
type: dictionary
50-
description: "`CUSTOMER.metadata` properties that will be pivoted into columns in `stripe__customer_overview`"
55+
description: "`customer.metadata` properties that will be pivoted into columns in `stripe__customer_overview`, `stripe__balance_transactions`, `stripe__invoice_details`, and `stripe__subscription_details`."
5156
display_name: "Customer Metadata Properties"
5257

58+
stripe__charge_metadata:
59+
type: dictionary
60+
description: "`charge.metadata` properties that will be pivoted into columns in `stripe__balance_transactions` and `stripe__invoice_details`."
61+
display_name: "Charge Metadata Properties"
62+
63+
stripe__invoice_metadata:
64+
type: dictionary
65+
description: "`invoice.metadata` properties that will be pivoted into columns in `stripe__invoice_details` and `stripe__balance_transactions`."
66+
display_name: "Invoice Metadata Properties"
67+
68+
stripe__subscription_metadata:
69+
type: dictionary
70+
description: "`subscription.metadata` properties that will be pivoted into columns in `stripe__subscription_details`, `stripe__balance_transactions`, `stripe__invoice_line_item_details`, and `stripe__invoice_details`."
71+
display_name: "Subscription Metadata Properties"
72+
5373
stripe__convert_values:
5474
type: boolean
5575
value: false

CHANGELOG.md

Lines changed: 42 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,45 @@
1+
# dbt_stripe v1.5.0
2+
3+
## Schema Changes
4+
**2 total changes • 1 possible breaking change**
5+
| **Data Model** | **Change type** | **Old** | **New** | **Notes** |
6+
| -------------- | --------------- | ------------ | ------------ | --------- |
7+
| All models | Single-connection `source_relation` value | Empty string (`''`) | `<stripe_database>.<stripe_schema>` | |
8+
| `stg_stripe__coupon` | New column | | `metadata` | optional custom JSON field |
9+
10+
## Feature Updates
11+
- Adds configurable table variables `stripe__using_transfers` and `stripe__using_payouts`, which are set to true by default. For dbt Core users, [See the README](https://github.com/fivetran/dbt_stripe?tab=readme-ov-file#disable-models-for-non-existent-sources) for more details on how to disable these variables.
12+
- `stripe__using_transfers` disables the Transfers source and any relevant downstream components.
13+
- `stripe__using_payouts` disables both the Payouts and Payout Balance Transactions sources, along with any related downstream components.
14+
- Introduces support for the newer, more flexible unioning framework. Previously, to run the package on multiple Stripe sources at once, you could only use the `union_schemas` variable OR `union_databases` (mutually exclusive). While these setups are still supported for backwards compatibility, we recommend using `stripe_sources` instead. See the [README](https://github.com/fivetran/dbt_stripe/blob/main/README.md#option-b-union-multiple-connections) for more details.
15+
16+
```yml
17+
# dbt_project.yml
18+
19+
vars:
20+
stripe:
21+
stripe_sources:
22+
- database: connection_1_destination_name # Required
23+
schema: connection_1_schema_name # Required
24+
name: connection_1_source_name # Required only if following this step: https://github.com/fivetran/dbt_stripe/blob/main/README.md#recommended-incorporate-unioned-sources-into-dag
25+
26+
- database: connection_2_destination_name
27+
schema: connection_2_schema_name
28+
name: connection_2_source_name
29+
```
30+
- Updates end models (`stripe__balance_transactions`, `stripe__customer_overview`, `stripe__invoice_details`, `stripe__invoice_line_item_details`, `stripe__subscription_details`) to dynamically include metadata fields from staging models when metadata variables are configured. See the [README](https://github.com/fivetran/dbt_stripe/blob/main/README.md#pivoting-out-metadata-properties) for more details.
31+
32+
## Quickstart Updates
33+
- Creates table variables for sources `transfer`, `payout`, and `payout_balance_transaction`. These source tables must be selected in the Fievetran connector UI for the variables to be set to True and the dependent models to be run.
34+
- Adds `stripe__charge_metadata`, `stripe__invoice_metadata`, and `stripe__subscription_metadata` to `supported_vars` as optional properties to be pivoted into columns in various end models. These variables can be configured directly through the `supportedVars` section in the Quickstart UI.
35+
36+
## Under the Hood
37+
- Updates all tmp staging models to conditionally use either the new `stripe_union_connections` macro (when `stripe_sources` is configured) or the legacy `fivetran_utils.union_data` macro (for backward compatibility).
38+
- Updates all staging models to use the new `stripe.apply_source_relation()` macro instead of `fivetran_utils.source_relation()`.
39+
- Adds `stripe.select_metadata_columns()` macro to handle both dictionary and alias variable metadata inputs.
40+
- Adds `metadata` column to `get_coupon_columns()` macro and `coupon_data.csv` seed file.
41+
- Updates integration test seed data for customer and invoice tables.
42+
143
# dbt_stripe v1.4.0
244
[PR #138](https://github.com/fivetran/dbt_stripe/pull/138) includes the following updates:
345

@@ -423,8 +465,6 @@ Stripe passes amount-based fields, such as `amount`, `net`, and `fee`, in the sm
423465
- Updated `customer_facing_amount` to include for refunds and disputes as well
424466
- Updated `charge_id` to charge, refund, then dispute objects consecutively
425467

426-
427-
428468
## Under the Hood:
429469

430470
- Incorporated the new `fivetran_utils.drop_schemas_automation` macro into the end of each Buildkite integration test job.

0 commit comments

Comments
 (0)