Skip to content
Merged
Show file tree
Hide file tree
Changes from 24 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
8108b6d
Consolidate dbt source into transform
fivetran-catfritz Aug 13, 2025
6adef62
Q2 FY26 Automatic Package Updates (#66)
fivetran-data-model-bot Aug 13, 2025
fec0eae
manual
fivetran-jamie Aug 13, 2025
ba87972
reorganize subfolders
fivetran-jamie Aug 14, 2025
b5187bd
cofnirm schemas are good
fivetran-jamie Aug 14, 2025
3d82ad5
revert
fivetran-jamie Aug 14, 2025
8e495b7
debugging
fivetran-jamie Aug 14, 2025
490301b
try this
fivetran-jamie Aug 14, 2025
61d71c0
Update generate-docs.yml
fivetran-catfritz Aug 14, 2025
b1ddae2
Generate dbt docs via GitHub Actions
github-actions[bot] Aug 14, 2025
57f38ae
revert some stuff
fivetran-jamie Aug 14, 2025
dbfdcbc
Merge branch 'MagicBot/consolidate-source-package' of https://github.…
fivetran-jamie Aug 14, 2025
4439381
Generate dbt docs via GitHub Actions
github-actions[bot] Aug 14, 2025
898092d
readme
fivetran-jamie Aug 14, 2025
99f10bf
readme again
fivetran-jamie Aug 14, 2025
1445ad0
Apply suggestions from code review
fivetran-jamie Aug 14, 2025
260edfe
Update dbt_project.yml
fivetran-joemarkiewicz Aug 18, 2025
6294378
Update CHANGELOG.md
fivetran-joemarkiewicz Aug 18, 2025
96cc6d9
Update CHANGELOG.md
fivetran-joemarkiewicz Aug 18, 2025
45239e9
Update dbt_project.yml
fivetran-joemarkiewicz Aug 18, 2025
f4b1752
Update dbt_project.yml
fivetran-joemarkiewicz Aug 18, 2025
edb31bd
Update src_salesforce_history.yml
fivetran-joemarkiewicz Aug 18, 2025
80ad2e1
Update src_salesforce.yml
fivetran-joemarkiewicz Aug 18, 2025
6c8e79d
Generate dbt docs via GitHub Actions
github-actions[bot] Aug 18, 2025
7701e2e
Update consistency_columns.sql
fivetran-joemarkiewicz Aug 18, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .buildkite/scripts/run_models.sh
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ dbt seed --target "$db" --full-refresh
dbt source freshness --target "$db" || echo "...Only verifying freshness runs..."
dbt run --target "$db" --full-refresh
dbt test --target "$db"
dbt run --vars '{salesforce__user_role_enabled: false}' --target "$db" --full-refresh
dbt test --target "$db"
dbt run --vars '{salesforce__account_history_enabled: true, salesforce__contact_history_enabled: true, salesforce__opportunity_history_enabled: true}' --target "$db" --full-refresh
dbt test --target "$db"
dbt run --vars '{salesforce__account_history_enabled: true, salesforce__contact_history_enabled: true, salesforce__opportunity_history_enabled: true}' --target "$db"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,15 @@ Before marking this PR as "ready for review":

## PR Overview
**Package version introduced in this PR:**

-

**This PR addresses the following Issue/Feature(s):**
<!-- Add Issue # or internal ticket reference -->
-

**Summary of changes:**
<!-- 1-2 sentences describing PR changes. -->

-

### Submission Checklist
- [ ] Alignment meeting with the reviewer (if needed)
Expand All @@ -27,9 +29,10 @@ Before marking this PR as "ready for review":
- [ ] **Validation Steps:** Check for unintentional effects (e.g., add/run consistency & integrity tests)
- [ ] **Testing Instructions:** Confirm the change addresses the issue(s)
- [ ] **Focus Areas:** Complex logic or queries that need extra attention
- [ ] Merge any relevant open PRs into this PR

### Changelog
<!-- Recommend drafting changelog notes, then refining via ChatGPT using:
"Draft a changelog entry based on the following notes." -->
- [ ] Draft changelog for PR
- [ ] Final changelog for release review
- [ ] Final changelog for release review
11 changes: 6 additions & 5 deletions .github/workflows/auto-release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,12 @@ on:
pull_request:
types:
- closed
branches:
- main
- labeled

jobs:
call-workflow-passing-data:
if: github.event.pull_request.merged
release:
if: |
(github.event.pull_request.merged == true && github.event.pull_request.base.ref == 'main') ||
github.event.label.name == 'pre-release'
uses: fivetran/dbt_package_automations/.github/workflows/auto-release.yml@main
secrets: inherit
secrets: inherit
14 changes: 14 additions & 0 deletions .github/workflows/generate-docs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
name: 'generate dbt docs'
on:
pull_request:
types:
- labeled

jobs:
generate-docs:
if: github.event.label.name == 'docs:ready'
uses: fivetran/dbt_package_automations/.github/workflows/generate-docs.yml@main
secrets: inherit
with:
# This needs to be kept a long list like this, and ': public' should not be included in the last var of the list.
schema_var_name: "salesforce_schema: 'public', salesforce_history_schema"
76 changes: 70 additions & 6 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,9 +1,73 @@

target/
# dbt
**/package-lock.yml
package-lock.yml
.dbt/
dbt_modules/
dbt_packages/
logs/
keyfile.json
profiles.yml
target/
*.log

# IDE files
.idea/
.vscode/
*~
*.swp
*.swo

# Jupyter Notebook
.ipynb_checkpoints

# OS generated files
**/.DS_Store
.DS_Store
develop/
dbt_packages/
env/
.Spotlight-V100
.Trashes
._*
Thumbs.db
ehthumbs.db

# Python
*.egg
*.egg-info/
*.py[cod]
*.so
*$py.class
.Python
__pycache__/
build/
develop-eggs/
dist/
downloads/
eggs/
.env
.installed.cfg
lib/
lib64/
MANIFEST
parts/
sdist/
var/
wheels/

# Secrets and credentials
.env.*
.secrets
credentials.json
service-account.json

# Temporary files
.cache/
*.temp
*.tmp

# Virtual environments
.conda/
.env
.venv
ENV/
env/
env.bak/
venv/
venv.bak/
24 changes: 24 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,27 @@
# dbt_salesforce v2.0.0

[PR #67](https://github.com/fivetran/dbt_salesforce/pull/67) includes the following updates:

## Breaking Changes

### Source Package Consolidation
- Removed the dependency on the `fivetran/salesforce_source` package.
- All functionality from the source package has been merged into this transformation package for improved maintainability and clarity.
- If you reference `fivetran/salesforce_source` in your `packages.yml`, you must remove this dependency to avoid conflicts.
- Any source overrides referencing the `fivetran/salesforce_source` package will also need to be removed or updated to reference this package.
- Update any salesforce_source-scoped variables to be scoped to only under this package. See the [README](https://github.com/fivetran/dbt_salesforce/blob/main/README.md#change-the-build-schema) for how to configure the build schema of staging models.
- As part of the consolidation, vars are no longer used to reference staging models, and only sources are represented by vars. Staging models are now referenced directly with `ref()` in downstream models.

### dbt Fusion Compatibility Updates
- Updated package to maintain compatibility with dbt-core versions both before and after v1.10.6, which introduced a breaking change to multi-argument test syntax (e.g., `unique_combination_of_columns`).
- Temporarily removed unsupported tests to avoid errors and ensure smoother upgrades across different dbt-core versions. These tests will be reintroduced once a safe migration path is available.
- Removed all `dbt_utils.unique_combination_of_columns` tests.
- Moved `loaded_at_field: _fivetran_synced` under the `config:` block in `src_salesforce.yml` and `src_salesforce_history.yml`.

### Under the Hood
- Updated conditions in `.github/workflows/auto-release.yml`.
- Added `.github/workflows/generate-docs.yml`.

# dbt_salesforce v1.3.0

[PR #65](https://github.com/fivetran/dbt_salesforce/pull/65) includes the following updates:
Expand Down
2 changes: 1 addition & 1 deletion LICENSE
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,7 @@
same "printed page" as the copyright notice for easier
identification within third-party archives.

Copyright [yyyy] [name of copyright owner]
Copyright © 2025 Fivetran Inc.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
Expand Down
35 changes: 19 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
# Salesforce Modeling dbt Package ([Docs](https://fivetran.github.io/dbt_salesforce/))
# Salesforce dbt Package ([Docs](https://fivetran.github.io/dbt_salesforce/))

<p align="left">
<a alt="License"
href="https://github.com/fivetran/dbt_salesforce/blob/main/LICENSE">
<img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" /></a>
<a alt="dbt-core">
<img src="https://img.shields.io/badge/dbt_core™-version_>=1.3.0_<2.0.0-orange.svg" /></a>
<img src="https://img.shields.io/badge/dbt_Core™_version->=1.3.0_,<2.0.0-orange.svg" /></a>
<a alt="Maintained?">
<img src="https://img.shields.io/badge/Maintained%3F-yes-green.svg" /></a>
<a alt="PRs">
Expand All @@ -16,7 +16,7 @@
</p>

## What does this dbt package do?
- Produces modeled tables that leverage Salesforce data from [Fivetran's connector](https://fivetran.com/docs/applications/salesforce) in the format described by [this ERD](https://fivetran.com/docs/applications/salesforce#schema) and builds off the output of our [Salesforce source package](https://github.com/fivetran/dbt_salesforce_source).
- Produces modeled tables that leverage Salesforce data from [Fivetran's connector](https://fivetran.com/docs/applications/salesforce) in the format described by [this ERD](https://fivetran.com/docs/applications/salesforce#schema).
- This package also provides you with the option to leverage the history mode to gather historical records of your essential tables.

- This package enables users to:
Expand Down Expand Up @@ -77,10 +77,10 @@ Include the following salesforce package version in your `packages.yml`
```yaml
packages:
- package: fivetran/salesforce
version: [">=1.3.0", "<1.4.0"] # we recommend using ranges to capture non-breaking changes automatically
version: [">=2.0.0", "<2.1.0"] # we recommend using ranges to capture non-breaking changes automatically
```

Do NOT include the `salesforce_source` package in this file. The transformation package itself has a dependency on it and will install the source package as well.
> All required sources and staging models are now bundled into this transformation package. Do not include `fivetran/salesforce_source` in your `packages.yml` since this package has been deprecated.

### Step 3: Configure Your Variables
#### Database and Schema Variables
Expand Down Expand Up @@ -124,10 +124,10 @@ For this use case, to ensure the package runs successfully, we recommend leverag
> Note that all other end models (`salesforce__opportunity_enhanced`, `salesforce__opportunity_line_item_enhanced`, `salesforce__manager_performance`, `salesforce__owner_performance`, `salesforce__sales_snapshot`, and `salesforce__opportunity_daily_history`) will still materialize after a blanket `dbt run` but will be largely empty/null.

### (Optional) Step 4: Utilizing Salesforce History Mode records
If you have Salesforce [History Mode](https://fivetran.com/docs/using-fivetran/features#historymode) enabled for your connection, we now include support for the `account`, `contact`, and `opportunity` tables directly. These staging models from our `dbt_salesforce_source` package flow into our daily history models. This will allow you access to your historical data for these tables while taking advantage of incremental loads to help with compute.
If you have Salesforce [History Mode](https://fivetran.com/docs/using-fivetran/features#historymode) enabled for your connection, we now include support for the `account`, `contact`, and `opportunity` tables directly. These staging models flow into our daily history models. This will allow you access to your historical data for these tables while taking advantage of incremental loads to help with compute.

#### IMPORTANT: How To Update Your History Models
To ensure maximum value for these history mode models and avoid messy historical data that could come with picking and choosing which fields you bring in, **all fields in your Salesforce history mode connection are being synced into your end staging models**. That means all custom fields you picked to sync are being brought in to the final models. [See our DECISIONLOG for more details on why we are bringing in all fields](https://github.com/fivetran/dbt_salesforce_source/blob/main/DECISIONLOG.md).
To ensure maximum value for these history mode models and avoid messy historical data that could come with picking and choosing which fields you bring in, **all fields in your Salesforce history mode connection are being synced into your end staging models**. That means all custom fields you picked to sync are being brought in to the final models. [See our DECISIONLOG for more details on why we are bringing in all fields](https://github.com/fivetran/dbt_salesforce/blob/main/DECISIONLOG.md).

To update the history mode models, you must follow these steps:
1) Go to your Fivetran Salesforce History Mode connection page.
Expand Down Expand Up @@ -171,7 +171,6 @@ The History Mode models can get quite expansive since it will take in **ALL** hi
```yml
# dbt_project.yml

...
vars:
salesforce__account_history_enabled: true # False by default. Only use if you have history mode enabled and wish to view the full historical record of all your synced account fields, particularly in the daily account history model.
salesforce__contact_history_enabled: true # False by default. Only use if you have history mode enabled and wish to view the full historical record of all your synced contact fields, particularly in the daily contact history model.
Expand All @@ -183,7 +182,6 @@ By default, these models are set to bring in all your data from Salesforce Histo

We have set up where conditions in our staging models to allow you to bring in only the data you need to run in. You can set a global history filter that would apply to all of our staging history models in your `dbt_project.yml`:


```yml
vars:
global_history_start_date: 'YYYY-MM-DD' # The first `_fivetran_start` date you'd like to filter data on in all your history models.
Expand All @@ -201,21 +199,29 @@ vars:
### (Optional) Step 5: Additional Configurations
#### Change the Source Table References
Source tables are referenced using default names. If an individual source table has a different name than expected, provide the name of the table as it appears in your warehouse to the respective variable:
> IMPORTANT: See the package's source [`dbt_project.yml`](https://github.com/fivetran/dbt_salesforce_source/blob/main/dbt_project.yml) variable declarations to see the expected names.
> IMPORTANT: See the package's source [`dbt_project.yml`](https://github.com/fivetran/dbt_salesforce/blob/main/dbt_project.yml) variable declarations to see the expected names.

```yml
vars:
<package_name>_<default_source_table_name>_identifier: your_table_name
```

#### Change the Build Schema
By default, this package builds the GitHub staging models within a schema titled (<target_schema> + `_stg_salesforce`) in your target database. If this is not where you would like your GitHub staging data to be written to, add the following configuration to your root `dbt_project.yml` file:
By default, this package builds all of the Salesforce models within your `target.schema` in your target database. If this is not where you would like your Salesforce data to be written to, add the following configuration to your root `dbt_project.yml` file:

```yml
models:
salesforce_source:
+schema: my_new_schema_name # leave blank for just the target_schema
salesforce: # If you want everything written to one schema, you can scope the +schema config to the package level
salesforce:
+schema: my_new_schema_name # Will write Salesforce end models to <target_schema> + _my_new_schema_name
staging:
+schema: my_new_schema_name # Will write Salesforce staging models to <target_schema> + _my_new_schema_name
salesforce_history:
+schema: my_new_schema_name # Will write Salesforce History models to <target_schema> + _my_new_schema_name
staging:
+schema: my_new_schema_name # Will write Salesforce History staging models to <target_schema> + _my_new_schema_name
```

#### Adding Passthrough Columns
This package allows users to add additional columns to the `salesforce__opportunity_enhanced`, `salesforce__opportunity_line_item_enhanced`,`salesforce__contact_enhanced`, and any of the `daily_history` models if you have Salesforce history mode enabled. You can do this by using the below variables in your `dbt_project.yml` file. These variables allow these additional columns to be aliased (`alias`) and casted (`transform_sql`) if desired, but not required. Datatype casting is configured via a sql snippet within the `transform_sql` key. You may add the desired sql while omitting the `as field_name` at the end and your custom pass-though fields will be casted accordingly. Use the below format for declaring the respective pass-through variables.

Expand All @@ -226,7 +232,6 @@ Additionally, you may add additional columns to the staging models. For example,
```yml
# dbt_project.yml

...
vars:
salesforce__account_pass_through_columns:
- name: "salesforce__account_field"
Expand Down Expand Up @@ -288,8 +293,6 @@ This dbt package is dependent on the following dbt packages. For more informatio
> **If you have any of these dependent packages in your own `packages.yml` I highly recommend you remove them to ensure there are no package version conflicts.**
```yml
packages:
- package: fivetran/salesforce_source
version: [">=1.2.0", "<1.3.0"]
- package: fivetran/fivetran_utils
version: [">=0.4.0", "<0.5.0"]
- package: dbt-labs/dbt_utils
Expand Down
37 changes: 22 additions & 15 deletions dbt_project.yml
Original file line number Diff line number Diff line change
@@ -1,30 +1,37 @@
config-version: 2
name: 'salesforce'
version: '1.3.0'
version: '2.0.0'
require-dbt-version: [">=1.3.0", "<2.0.0"]
models:
salesforce:
+materialized: table
intermediate:
+materialized: ephemeral
Comment on lines 7 to 9

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This config doesn't look right? There's no intermediate folder in this location.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may be a FR for us to explore more in the future

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I noticed that too and noted it to make a FR later. Also all of the models get written to the target schema - would we want to explore changing that later as well?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fivetran-joemarkiewicz should i create a FR here

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that would be great. I wouldn't want to change this just now, but we should in a future update.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Created FR #68 for us to evaluate in a future update.

salesforce:
staging:
+materialized: table
salesforce_history:
staging:
+materialized: table

vars:
salesforce:
account: "{{ ref('stg_salesforce__account') }}"
opportunity: "{{ ref('stg_salesforce__opportunity') }}"
user: "{{ ref('stg_salesforce__user') }}"
user_role: "{{ ref('stg_salesforce__user_role') }}"
contact: "{{ ref('stg_salesforce__contact') }}"
lead: "{{ ref('stg_salesforce__lead') }}"
task: "{{ ref('stg_salesforce__task') }}"
event: "{{ ref('stg_salesforce__event') }}"
product_2: "{{ ref('stg_salesforce__product_2') }}"
order: "{{ ref('stg_salesforce__order') }}"
opportunity_line_item: "{{ ref('stg_salesforce__opportunity_line_item') }}"
account: "{{ source('salesforce', 'account') }}"
opportunity: "{{ source('salesforce', 'opportunity') }}"
user: "{{ source('salesforce', 'user') }}"
user_role: "{{ source('salesforce', 'user_role') }}"
contact: "{{ source('salesforce', 'contact') }}"
lead: "{{ source('salesforce', 'lead') }}"
task: "{{ source('salesforce', 'task') }}"
event: "{{ source('salesforce', 'event') }}"
product_2: "{{ source('salesforce', 'product_2') }}"
order: "{{ source('salesforce', 'order') }}"
opportunity_line_item: "{{ source('salesforce', 'opportunity_line_item') }}"

account_history: "{{ source('salesforce_history', 'account') }}"
contact_history: "{{ source('salesforce_history', 'contact') }}"
opportunity_history: "{{ source('salesforce_history', 'opportunity') }}"

account_history: "{{ ref('stg_salesforce__account_history') }}"
contact_history: "{{ ref('stg_salesforce__contact_history') }}"
opportunity_history: "{{ ref('stg_salesforce__opportunity_history') }}"

salesforce__account_pass_through_columns: []
Comment thread
fivetran-joemarkiewicz marked this conversation as resolved.
salesforce__contact_pass_through_columns: []
Expand Down
2 changes: 1 addition & 1 deletion docs/catalog.json

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/manifest.json

Large diffs are not rendered by default.

1 change: 0 additions & 1 deletion integration_tests/ci/sample.profiles.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@
# HEY! This file is used in the dbt package integrations tests with Buildkite.
# You should __NEVER__ check credentials into version control. Thanks for reading :)


integration_tests:
target: redshift
outputs:
Expand Down
Loading