feat: update internal packages to Python 3.11 #52664

devin-ai-integration · 2025-01-30T20:07:06Z

Description

Updates all internal Python packages to use Python 3.11 and replaces deprecated distutils usage.

Changes

Update Python version from 3.10 to 3.11 in all internal packages' pyproject.toml files
Update python_versions in airbyte_ci configuration sections
Replace distutils.util.strtobool with custom implementation in airbyte-python-cdk
Preserve specific version constraints (e.g., <3.12) where applicable

Updated Packages

connector_ops
connectors_qa
ci_credentials
metadata_service/lib
metadata_service/orchestrator
pipelines
common_utils
connectors_insights
auto_merge
erd
live-tests
connector-acceptance-test

Link to Devin run

https://app.devin.ai/sessions/7b158d1965bd493db84ea04078974ce6

Pre-merge checklist

All Python versions updated to 3.11 in pyproject.toml files
All airbyte_ci configurations updated to use Python 3.11
Specific version constraints preserved where needed
distutils.util.strtobool replaced with custom implementation

- Update Python version from 3.10 to 3.11 in all internal packages - Update python_versions in airbyte_ci configuration - Preserve specific version constraints where applicable Co-Authored-By: [email protected] <[email protected]>

devin-ai-integration · 2025-01-30T20:07:10Z

🤖 Devin AI Engineer

Original prompt from [email protected]:

@Devin can you make a PR to make a new major version of our Python connector base-image, 4.0, based on Python 3.11 instead of 3.10? AND a separate pull request that would switch all our internal packages (think airbyte-ci) to use python 3.11 in their pyproject.toml files, and in testing harnesses, and make sure the dagger testing containers come with python 3.11 as well.

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

Address comments on this PR. Add "(aside)" to your comment to have me ignore it.
Look at CI failures and help fix them

⚙️ Control Options:

Disable automatic comment and CI monitoring

vercel · 2025-01-30T20:07:11Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jan 31, 2025 5:53am

Co-Authored-By: [email protected] <[email protected]>

…i action Co-Authored-By: [email protected] <[email protected]>

Co-Authored-By: [email protected] <[email protected]>

devin-ai-integration · 2025-01-30T20:55:44Z

Devin is currently unreachable - the session may have died.

natikgadzhi · 2025-01-30T21:23:25Z

.github/workflows/upload-metadata-files.yml

@@ -19,7 +19,9 @@ jobs:
        if: steps.changed-files.outputs.any_changed == 'true'


@devin-ai-integration you should also update the "Setup Python 3.10" to 3.11

…1 compatibility Co-Authored-By: [email protected] <[email protected]>

natikgadzhi · 2025-01-30T21:45:20Z

airbyte-integrations/bases/connector-acceptance-test/pyproject.toml

@@ -32,8 +32,8 @@ requests-mock = "~=1.9.3"
 pytest-mock = "~=3.6.1"
 pendulum = "*"
 pytest-cov = "~=3.0.0"
-hypothesis = "*"
-hypothesis-jsonschema = "*"
+hypothesis = "~=6.84.3"


Can you pin with ^6.84.3, and leave a comment for why it's important?

Co-Authored-By: [email protected] <[email protected]>

…h versions Co-Authored-By: [email protected] <[email protected]>

natikgadzhi

Okay, OVERALL this looks good to me. I expect to see a couple of packages failing known bad tests here, but overall I will try and push this in if the CI is happy, and will communicate this widely so we know what to revert if shit hits the fan.

natikgadzhi · 2025-01-30T22:25:46Z

.github/actions/install-airbyte-ci/action.yml

        token: ${{ inputs.github_token }}
+        cache: "pip"


I don't know if these are strictly necessary here — but I don't mind? /shrug

natikgadzhi · 2025-01-31T02:34:06Z

airbyte-ci/connectors/auto_merge/README.md

@@ -50,6 +50,9 @@ have been merged.

 ## Changelog

+### 0.1.5


Lol good catch — we have 0.1.4 out without the changelog entry.

natikgadzhi · 2025-01-31T02:48:02Z

Do not merge this without me, I will look into the failures myself localy. ci_credentials is expected, but the other two, I'd like to fix up

Co-Authored-By: [email protected] <[email protected]>

natikgadzhi · 2025-01-31T03:15:53Z

/format-fix

Format-fix job started... Check job output.

🟦 Job completed successfully (no changes).

This reverts commit 5d78a3a.

Co-Authored-By: [email protected] <[email protected]>

This reverts commit f1a8a63.

This reverts commit 77d4151.

Co-Authored-By: [email protected] <[email protected]>

natikgadzhi · 2025-01-31T03:53:25Z

/format-fix

Format-fix job started... Check job output.

✅ Changes applied successfully. (7b33ba2)

natikgadzhi · 2025-01-31T05:47:49Z

Alright, I expect that the only failing test should be the ci_credentials one, and if that's true, I will merge this in the morning.

This reverts commit e743cd4.

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: [email protected] <[email protected]> Co-authored-by: Natik Gadzhi <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]>

feat: update internal packages to Python 3.11

527e0c1

- Update Python version from 3.10 to 3.11 in all internal packages - Update python_versions in airbyte_ci configuration - Preserve specific version constraints where applicable Co-Authored-By: [email protected] <[email protected]>

devin-ai-integration bot requested review from a team and aaronsteers January 30, 2025 20:07

octavia-squidington-iii added the community label Jan 30, 2025

vercel bot deployed to Preview January 30, 2025 20:12 View deployment

devin-ai-integration bot and others added 4 commits January 30, 2025 20:19

ci: update Python version to 3.11 in GitHub Actions workflows

665b7c6

Co-Authored-By: [email protected] <[email protected]>

ci: update Python version to 3.11 in GitHub Actions install-airbyte-c…

763c0ec

…i action Co-Authored-By: [email protected] <[email protected]>

ci: update Python setup in GitHub Actions workflows

24de05b

Co-Authored-By: [email protected] <[email protected]>

fix: add missing exceptiongroup dependency to pipelines package

23489c9

Co-Authored-By: [email protected] <[email protected]>

vercel bot deployed to Preview January 30, 2025 20:38 View deployment

natikgadzhi reviewed Jan 30, 2025

View reviewed changes

fix: pin hypothesis and hypothesis-jsonschema versions for Python 3.1…

48bb08e

…1 compatibility Co-Authored-By: [email protected] <[email protected]>

natikgadzhi reviewed Jan 30, 2025

View reviewed changes

vercel bot deployed to Preview January 30, 2025 21:50 View deployment

chore: update poetry.lock files for Python 3.11

519e0e7

Co-Authored-By: [email protected] <[email protected]>

vercel bot deployed to Preview January 30, 2025 21:58 View deployment

Merge branch 'master' into devin/1738267380-py311-internal

b54e1f4

vercel bot deployed to Preview January 30, 2025 22:17 View deployment

devin-ai-integration bot mentioned this pull request Jan 31, 2025

ci: update internal packages to Python 3.11 #52674

Closed

4 tasks

natikgadzhi closed this Jan 31, 2025

natikgadzhi reopened this Jan 31, 2025

chore: resolve merge conflicts with master

ad61b13

Co-Authored-By: [email protected] <[email protected]>

vercel bot deployed to Preview January 31, 2025 02:28 View deployment

ci: update Python version requirement from 3.10 to 3.11 and bump patc…

c111bf9

…h versions Co-Authored-By: [email protected] <[email protected]>

natikgadzhi approved these changes Jan 31, 2025

View reviewed changes

vercel bot deployed to Preview January 31, 2025 02:37 View deployment

devin-ai-integration bot and others added 2 commits January 31, 2025 03:03

fix: update test_publish.py to handle both spec.json and spec.cloud.json

b014d60

Co-Authored-By: [email protected] <[email protected]>

fix: update dependencies and fix test failures for Python 3.11

5d78a3a

Co-Authored-By: [email protected] <[email protected]>

vercel bot deployed to Preview January 31, 2025 03:09 View deployment

natikgadzhi and others added 4 commits January 30, 2025 19:17

ignore erd typing

ae0400b

Revert "fix: update dependencies and fix test failures for Python 3.11"

94932df

This reverts commit 5d78a3a.

fix: resolve merge conflicts and update Python dependencies

77d4151

Co-Authored-By: [email protected] <[email protected]>

fix: regenerate poetry.lock for Python 3.11 compatibility

f1a8a63

Co-Authored-By: [email protected] <[email protected]>

vercel bot deployed to Preview January 31, 2025 03:34 View deployment

natikgadzhi and others added 4 commits January 30, 2025 19:46

maybe fix CAT tests

b0cdb74

Revert "fix: regenerate poetry.lock for Python 3.11 compatibility"

30bc938

This reverts commit f1a8a63.

Revert "fix: resolve merge conflicts and update Python dependencies"

966898f

This reverts commit 77d4151.

chore: regenerate poetry.lock for Python 3.11 upgrade

c533a91

Co-Authored-By: [email protected] <[email protected]>

chore: auto-fix lint and format issues

7b33ba2

vercel bot deployed to Preview January 31, 2025 04:05 View deployment

bump deps for cats so they pass tests

b9788e6

vercel bot deployed to Preview January 31, 2025 04:21 View deployment

fix CATs tests

a335845

vercel bot deployed to Preview January 31, 2025 05:36 View deployment

fix pipelines tests

588c778

vercel bot deployed to Preview January 31, 2025 05:53 View deployment

natikgadzhi merged commit e743cd4 into master Jan 31, 2025
38 of 40 checks passed

natikgadzhi deleted the devin/1738267380-py311-internal branch January 31, 2025 17:55

natikgadzhi added a commit that referenced this pull request Jan 31, 2025

Revert "feat: update internal packages to Python 3.11 (#52664)"

d9b7ca8

This reverts commit e743cd4.

natikgadzhi mentioned this pull request Jan 31, 2025

Revert "feat: update internal packages to Python 3.11" #52692

Merged

natikgadzhi restored the devin/1738267380-py311-internal branch January 31, 2025 19:54

natikgadzhi mentioned this pull request Jan 31, 2025

feat(airbyte-ci): update tools to Python 3.11 #52695

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: update internal packages to Python 3.11 #52664

feat: update internal packages to Python 3.11 #52664

devin-ai-integration bot commented Jan 30, 2025 •

edited by natikgadzhi

Loading

devin-ai-integration bot commented Jan 30, 2025 •

edited by natikgadzhi

Loading

vercel bot commented Jan 30, 2025 •

edited

Loading

devin-ai-integration bot commented Jan 30, 2025

natikgadzhi Jan 30, 2025

natikgadzhi Jan 30, 2025

natikgadzhi left a comment

natikgadzhi Jan 30, 2025

natikgadzhi Jan 31, 2025

natikgadzhi commented Jan 31, 2025

natikgadzhi commented Jan 31, 2025 •

edited by github-actions bot

Loading

natikgadzhi commented Jan 31, 2025 •

edited by github-actions bot

Loading

natikgadzhi commented Jan 31, 2025

		@@ -19,7 +19,9 @@ jobs:
		if: steps.changed-files.outputs.any_changed == 'true'

feat: update internal packages to Python 3.11 #52664

feat: update internal packages to Python 3.11 #52664

Conversation

devin-ai-integration bot commented Jan 30, 2025 • edited by natikgadzhi Loading

Description

Changes

Updated Packages

Link to Devin run

Pre-merge checklist

devin-ai-integration bot commented Jan 30, 2025 • edited by natikgadzhi Loading

🤖 Devin AI Engineer

vercel bot commented Jan 30, 2025 • edited Loading

devin-ai-integration bot commented Jan 30, 2025

natikgadzhi Jan 30, 2025

Choose a reason for hiding this comment

natikgadzhi Jan 30, 2025

Choose a reason for hiding this comment

natikgadzhi left a comment

Choose a reason for hiding this comment

natikgadzhi Jan 30, 2025

Choose a reason for hiding this comment

natikgadzhi Jan 31, 2025

Choose a reason for hiding this comment

natikgadzhi commented Jan 31, 2025

natikgadzhi commented Jan 31, 2025 • edited by github-actions bot Loading

natikgadzhi commented Jan 31, 2025 • edited by github-actions bot Loading

natikgadzhi commented Jan 31, 2025

devin-ai-integration bot commented Jan 30, 2025 •

edited by natikgadzhi

Loading

devin-ai-integration bot commented Jan 30, 2025 •

edited by natikgadzhi

Loading

vercel bot commented Jan 30, 2025 •

edited

Loading

natikgadzhi commented Jan 31, 2025 •

edited by github-actions bot

Loading

natikgadzhi commented Jan 31, 2025 •

edited by github-actions bot

Loading