Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(Do Not Merge) Source S3: Bump Resource Allocation For Check & Discover #53204

Closed
wants to merge 8 commits into from
Closed

(Do Not Merge) Source S3: Bump Resource Allocation For Check & Discover #53204

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
b9abe99
Source S3: Bump Resource Allocation For Check & Discover
agarctfi Feb 6, 2025
1903e06
Update job name
agarctfi Feb 6, 2025
a0c69ed
add resourceRequirements for check_connection
agarctfi Feb 6, 2025
0779410
update maxSecondsBetweenMessages
agarctfi Feb 6, 2025
b916942
update resourceRequirements
agarctfi Feb 6, 2025
936a009
Update resource limits on dest side
agarctfi Feb 6, 2025
936ee4b
add snowflake
agarctfi Feb 6, 2025
5400814
remove discover resourceRequirements from destinations
agarctfi Feb 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions airbyte-integrations/connectors/destination-redshift/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,8 +84,12 @@ data:
jobSpecific:
- jobType: sync
resourceRequirements:
memory_limit: 1Gi
memory_request: 1Gi
memory_limit: 4Gi
memory_request: 4Gi
- jobType: discover_schema
resourceRequirements:
memory_limit: 4Gi
memory_request: 4Gi
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you mean to do this on the source? Destinations do not run discover schema.

Also this has infra implications doing this large of a bump. Specifically for SME customers who run on smaller nodes.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry to drag you into this, but @colesnodgrass did you say we strip these requirements for SME? If so, this would not affect SME.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will remove this one from the destinations, thanks! We won't be releasing it; we'll just be testing it on the cloud through dev images since these specific users have been blocked for quite some time.

supportLevel: certified
supportsDbt: true
supportsRefreshes: true
Expand Down
8 changes: 6 additions & 2 deletions airbyte-integrations/connectors/destination-snowflake/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,12 @@ data:
jobSpecific:
- jobType: sync
resourceRequirements:
memory_limit: 2Gi
memory_request: 2Gi
memory_limit: 4Gi
memory_request: 4Gi
- jobType: discover_schema
resourceRequirements:
memory_limit: 4Gi
memory_request: 4Gi
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto.

supportLevel: certified
supportsDbt: true
supportsRefreshes: true
Expand Down
13 changes: 12 additions & 1 deletion airbyte-integrations/connectors/source-s3/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,20 @@ data:
connectorSubtype: file
connectorType: source
definitionId: 69589781-7828-43c5-9f63-8925b1c1ccc2
dockerImageTag: 4.12.1
dockerImageTag: 4.12.2
dockerRepository: airbyte/source-s3
documentationUrl: https://docs.airbyte.com/integrations/sources/s3
resourceRequirements:
jobSpecific:
- jobType: check_connection
resourceRequirements:
memory_limit: 4096mi
memory_request: 4096mi
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checks should be have very low resource requirements. I'd consider this a source bug.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my understanding, Source S3, could be an outlier given the size of the file the user has on their source.

- jobType: discover_schema
resourceRequirements:
memory_limit: 4096mi
memory_request: 4096mi

githubIssueLabel: source-s3
icon: s3.svg
license: ELv2
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-s3/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ requires = [ "poetry-core>=1.0.0",]
build-backend = "poetry.core.masonry.api"

[tool.poetry]
version = "4.12.1"
version = "4.12.2"
name = "source-s3"
description = "Source implementation for S3."
authors = [ "Airbyte <[email protected]>",]
Expand Down
Loading