(Do Not Merge) Source S3: Bump Resource Allocation For Check & Discover #53204
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
octavia-squidington-iii
added
area/connectors
agarctfi
changed the title
| - jobType: discover_schema | ||
| resourceRequirements: | ||
| memory_limit: 4Gi | ||
| memory_request: 4Gi |
There was a problem hiding this comment.
Did you mean to do this on the source? Destinations do not run discover schema.
Also this has infra implications doing this large of a bump. Specifically for SME customers who run on smaller nodes.
There was a problem hiding this comment.
Sorry to drag you into this, but @colesnodgrass did you say we strip these requirements for SME? If so, this would not affect SME.
There was a problem hiding this comment.
I will remove this one from the destinations, thanks! We won't be releasing it; we'll just be testing it on the cloud through dev images since these specific users have been blocked for quite some time.
| - jobType: discover_schema | ||
| resourceRequirements: | ||
| memory_limit: 4Gi | ||
| memory_request: 4Gi |
| - jobType: check_connection | ||
| resourceRequirements: | ||
| memory_limit: 4096mi | ||
| memory_request: 4096mi |
There was a problem hiding this comment.
Checks should be have very low resource requirements. I'd consider this a source bug.
There was a problem hiding this comment.
From my understanding, Source S3, could be an outlier given the size of the file the user has on their source.
|
It doesn't seem like this solved the issue so closing this PR for now |
What
For: https://github.com/airbytehq/oncall/issues/6959
Users have a lot of files, causing a timeout during schema discovery.
How
We bumped the memory and CPU limits to see if this would allow discover to finish. We will make a dev image from this and pin their connection.
Review guide
User Impact
Can this PR be safely reverted and rolled back?