Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Expand sql table resource config #2396

Open
wants to merge 4 commits into
base: devel
Choose a base branch
from

Conversation

xneg
Copy link
Contributor

@xneg xneg commented Mar 11, 2025

Description

It's possible to specify sql table using config.toml file (see docs).

[sources.sql_database.chat_message]
backend="pandas"
chunk_size=1000

[sources.sql_database.chat_message.incremental]
cursor_path="updated_at"

The problem is this setting-based approach allows only append strategy and doesn't allow to specify custom primary or merge keys.
While primary keys are reflected from source table it's not possible for views.
So this PR allows to specify write_disposition, primary_key and merge_key in config.toml file with some restrictions (see Additional Context):

[sources.sql_database.chat_message]
chunk_size=1000

[sources.sql_database.chat_message.incremental]
write_disposition.disposition = "merge"
primary_key = ["id", "updated_at"]
cursor_path="updated_at"

Additional Context

There is a problem with not total alignment between def resource(...) method's arguments and fields in config classes.
Current config system doesn't support Union types (and probably it's not possible and there is no point) but this leads to the situation that we can't use plain string values:

[sources.sql_database.chat_message.incremental]
write_disposition= "merge"
primary_key = "id"

but we have to use more complex dict or lists values:

[sources.sql_database.chat_message.incremental]
write_disposition.disposition= "merge"
primary_key = ["id"]

If you have any better ideas how to realize this opportunity I'm open to adjust my PR or implement it different way.

Copy link

netlify bot commented Mar 11, 2025

Deploy Preview for dlt-hub-docs canceled.

Name Link
🔨 Latest commit baaae97
🔍 Latest deploy log https://app.netlify.com/sites/dlt-hub-docs/deploys/67d13e99d4fd6300087893c2

@xneg xneg changed the title Expand sql table resource config feat: Expand sql table resource config Mar 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant