Skip to content

Commit 1443421

Browse files
rudolfixanuunchin
andauthored
change contribution guide (#620)
* update contributing docs * bumps uv lock * fixes some tests * Update CONTRIBUTING.md Co-authored-by: anuunchin <[email protected]> --------- Co-authored-by: anuunchin <[email protected]>
1 parent 039bbad commit 1443421

File tree

6 files changed

+49
-135
lines changed

6 files changed

+49
-135
lines changed

CONTRIBUTING.md

Lines changed: 18 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -15,24 +15,34 @@
1515
</div>
1616
<br>
1717

18+
1819
The following guide will walk you through contributing new sources or changes to existing sources
1920
and their demo pipelines and contains a troubleshooting section. Please also read
2021
[DISTRIBUTION.md](docs/DISTRIBUTION.md) to understand how our sources are distributed to the users.
21-
Refer to [BUILDING-BLOCKS.md](docs/BUILDING-BLOCKS.md) to learn about the basic building blocks of a
22-
dlt source.
2322

24-
What do you can do here:
23+
## Before you start
24+
* 🚀 We are happy to accept **bugfixes and improvements** to existing sources!
25+
* 📣 **We accept new sources but only those that cannot be easily implemented** via `REST API`, `sql_database` or `filesystem` or with **vibe coding**.
26+
- see issues for source requests!
27+
- queues and brokers like google pub sub or rabbitMQ
28+
- SAP/ERP/HEALTH
29+
- HR platforms
30+
- Graph QL
31+
32+
33+
34+
## What you can do here
2535

2636
- Contribute a change to an existing verified source or its demo pipeline: Go to the
27-
["Walktrough: Fix, improve, customize, document an existing source"](#walktrough-fix-improve-customize-document-an-existing-pipeline)
37+
["Walkthrough: Fix, improve, customize, document an existing source"](#Walkthrough-fix-improve-customize-document-an-existing-pipeline)
2838
section.
2939
- Contribute a new verified source: Go to the
30-
["Walktrough: Create and contribute a new source"](#walktrough-create-and-contribute-a-new-source)
40+
["Walkthrough: Create and contribute a new source"](#Walkthrough-create-and-contribute-a-new-source)
3141
section.
3242
- Join our slack to get support from us by following the
3343
[invitation link](https://dlthub.com/community).
3444

35-
## Walktrough: Fix, improve, customize, document an existing pipeline
45+
## Walkthrough: Fix, improve, customize, document an existing pipeline
3646

3747
In this section you will learn how to contribute changes to an existing pipeline.
3848

@@ -50,7 +60,7 @@ In this section you will learn how to contribute changes to an existing pipeline
5060
4. Run the linter and formatter to check for any problems: `make lint-code`.
5161
3. Proceed to the pull request section to [create a pull request to the main repo](#making-a-pull-request-to-the-main-repo-from-fork).
5262

53-
## Walktrough: Create and contribute a new source
63+
## Walkthrough: Create and contribute a new source
5464

5565
In this section you will learn how to contribute a new source including tests and a demo pipeline
5666
for that source. It is helpful to also read through the above section to see all the steps that are
@@ -342,7 +352,7 @@ In essence **DLT_SECRETS_TOML** is just your `secrets.toml` file and will be use
342352

343353

344354
### Sharing and obtaining source credentials, test accounts, destination access
345-
Typically we created a common test account for your source [before you started coding](#walktrough-create-and-contribute-a-new-source). This is an ideal situation - we can reuse your tests directly and can merge your work quickly.
355+
Typically we create a common test account for your source [before you started coding](#Walkthrough-create-and-contribute-a-new-source). This is an ideal situation - we can reuse your tests directly and can merge your work quickly.
346356

347357
If you contributed a source and created own credentials, test accounts or test datasets please
348358
include them in the tests or share them with `dlt` team so we can configure the CI job. If

README.md

Lines changed: 14 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -19,23 +19,29 @@ This repository contains verified dlt sources and demo pipelines for each source
1919

2020
## How to add a verified source to your dlt project
2121
`dlt` offers an `init` command that will clone and inject any source with an example pipeline from this repository into your project, setup the credentials and python dependencies. Please follow the step by step instructions in our [docs](https://dlthub.com/docs/walkthroughs/add-a-verified-source).
22+
23+
### We encourage you to hack the code
24+
Verified sources code is added to your project with the expectation that you will hack or customize code yourself - as you need it.
25+
26+
2227
## How to contact us and get help
2328
Join our slack by following the [invitation link](https://dlthub.com/community)
2429

25-
- If you have added a source and something does not work, post in the `technical-help` channel
26-
- If you want to contribute sources, customization or a fix, post in the `dlt-contributors` channel
27-
2830
## Reporting a source bug
2931
Follow this link: [bug report template](https://github.com/dlt-hub/verified-sources/issues/new?template=bug-report.md)
3032

31-
## Requesting a source implementation
32-
Follow this link: [source request template](https://github.com/dlt-hub/verified-sources/issues/new?template=request-new-source.md)
33-
3433
## How to contribute
34+
* 🚀 We are happy to accept **bugfixes and improvements** to existing sources!
35+
* 📣 **We accept new sources but only those that cannot be easily implemented** via `REST API`, `sql_database` or `filesystem` or with **vibe coding**.
36+
- see issues for source requests!
37+
- queues and brokers like google pub sub or rabbitMQ
38+
- SAP/ERP/HEALTH
39+
- HR platforms
40+
- Graph QL
41+
* 📣 **Before starting, announce your PR** with [source request template](https://github.com/dlt-hub/verified-sources/issues/new?template=bug-report.md)
42+
3543
Find step by step instruction as well as troubleshooting help in [CONTRIBUTING.md](CONTRIBUTING.md).
3644

37-
## Building blocks
38-
Find a list of the building blocks of dlt with links to our official documentation in [BUILDING-BLOCKS.md](docs/BUILDING-BLOCKS.md).
3945

4046
## Building with LLMs
4147
See README [`ai`](ai/README.md) folder for details.

docs/BUILDING-BLOCKS.md

Lines changed: 0 additions & 35 deletions
This file was deleted.

sources/pipedrive/__init__.py

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,6 @@ def pipedrive_source(
3434
Args:
3535
pipedrive_api_key: https://pipedrive.readme.io/docs/how-to-find-the-api-token
3636
since_timestamp: Starting timestamp for incremental loading. By default complete history is loaded on first run.
37-
incremental: Enable or disable incremental loading.
3837
3938
Returns resources:
4039
custom_fields_mapping
@@ -175,7 +174,7 @@ def parsed_mapping(
175174

176175

177176
@dlt.resource(primary_key="id", write_disposition="merge")
178-
def leads(
177+
def leads(
179178
pipedrive_api_key: str = dlt.secrets.value,
180179
update_time: dlt.sources.incremental[str] = dlt.sources.incremental(
181180
"update_time", "1970-01-01 00:00:00"

tests/inbox/test_inbox.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -111,7 +111,7 @@ def test_parse_pdf() -> None:
111111
def pdf_to_text(file_items: Sequence[FileItemDict]) -> Iterator[Dict[str, Any]]:
112112
# extract data from PDF page by page
113113
for file_item in file_items:
114-
with file_item.open() as file:
114+
with file_item.open(compression="disable") as file:
115115
reader = PdfReader(file)
116116
for page_no in range(len(reader.pages)):
117117
# add page content to file item

0 commit comments

Comments
 (0)