-
Notifications
You must be signed in to change notification settings - Fork 25
feat: new CLI: airbyte-cdk image build
#504
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
167 commits
Select commit
Hold shift + click to select a range
3de3edf
add skeleton implementation, imported from airbyte-pytest
aaronsteers 3b88490
tidy imports
aaronsteers 1250472
fix some imports
aaronsteers 9dec6e3
Merge branch 'main' into aj/feat/mini-cat-test-suites
aaronsteers 11d89f6
working test of tests
aaronsteers 42d8a4e
remove old file
aaronsteers c8e7722
misc fixes
aaronsteers 6248994
allow subclasses of abstractsource
aaronsteers a0f8ed5
Merge remote-tracking branch 'origin/main' into aj/feat/mini-cat-test…
aaronsteers fcfe697
fix test handling for expected errors
aaronsteers 03f3360
Auto-fix lint and format issues
460c40a
remove unused code
aaronsteers 8918bfe
fix tests
aaronsteers 83344f3
use pep440 style dynamic versioning
aaronsteers 76a92a9
update ci job (temporarily run on all pushes)
aaronsteers e6911e5
Merge remote-tracking branch 'origin/main' into aj/feat/mini-cat-test…
aaronsteers 039eef0
Merge remote-tracking branch 'origin/main' into aj/feat/mini-cat-test…
aaronsteers 5658441
Update .github/workflows/pypi_publish.yml
aaronsteers 310ea68
fix fixture resource paths
aaronsteers 1e74eef
lint fixes
aaronsteers c9ad3bd
fix imports
aaronsteers f8b581e
fix lint issues
aaronsteers 4a05f46
fix lint and typing
aaronsteers e58dfad
[cherry-pick-me][chore]: resolve pytest warnings undeclared marks, an…
aaronsteers 418dcb9
fix relative path logic
aaronsteers 8d75d0d
ruff fix
aaronsteers c3a32ae
[cherry-pick-me][fix]: dataclasses constructor break with __test__ me…
aaronsteers 0897b83
fix paths
aaronsteers b282e5d
[cherry-pick-me]: use kw args
aaronsteers 9a8b5eb
Merge branch 'main' into aj/feat/mini-cat-test-suites
aaronsteers d5aaf3c
[cherry-pick-me]: kw args
aaronsteers 7a00ff4
[cherry-pick-me]: use kw args for Test* dataclasses
aaronsteers 5dff39e
Merge remote-tracking branch 'origin/main' into aj/feat/mini-cat-test…
aaronsteers 3160691
fix relative resource paths
aaronsteers 015ced9
fix imports
aaronsteers e2a9fb9
fix more tests
aaronsteers bfc1943
format fix
aaronsteers f82b76a
[cherry-pick-me][chore]: resolve pytest warnings undeclared marks, an…
aaronsteers f091b53
[cherry-pick-me][fix]: dataclasses constructor break with __test__ me…
aaronsteers 12afc3d
[cherry-pick-me]: use kw args
aaronsteers cca55f7
[cherry-pick-me]: kw args
aaronsteers 4f7a84e
[cherry-pick-me]: use kw args for Test* dataclasses
aaronsteers e24594b
Merge branch 'devin/1744436819-cherry-pick-test-fixes' into aj/feat/m…
aaronsteers 9389cc5
reduce code needed for inheritance
aaronsteers e4cae20
fix type hint
aaronsteers 18ff2e8
Use ClassVar[bool] for __test__ instead of kw_only=True
devin-ai-integration[bot] 40fddcc
Use ClassVar[bool] for __test__ attributes instead of kw_only=True
devin-ai-integration[bot] e2c69df
Update more classes to use ClassVar[bool] for __test__ attributes
devin-ai-integration[bot] e79e901
Apply ruff formatting
devin-ai-integration[bot] c26345e
Fix remaining files with ruff formatting
devin-ai-integration[bot] eb0f643
addl cleanup
aaronsteers 7a750d6
Merge branch 'devin/1744436819-cherry-pick-test-fixes' into aj/feat/m…
aaronsteers b8407e5
Merge branch 'main' into devin/1744436819-cherry-pick-test-fixes
aaronsteers 88acba3
Merge branch 'main' into aj/feat/mini-cat-test-suites
aaronsteers 4b6c53e
poe lock
aaronsteers f0bea0c
remove unrelated changes
aaronsteers 8c91964
Merge branch 'devin/1744436819-cherry-pick-test-fixes' into aj/feat/m…
aaronsteers 4f84c14
revert unnecessary changes
aaronsteers d0aba9f
Merge branch 'devin/1744436819-cherry-pick-test-fixes' into aj/feat/m…
aaronsteers 1468a48
clean up naming, remove unused
aaronsteers 2812e7d
tidy up pr
aaronsteers c4e6655
clean up IConnector interface
aaronsteers f6d6ccd
ruff fix
aaronsteers d20d91f
poe lock
aaronsteers b363b52
Merge branch 'main' into aj/feat/mini-cat-test-suites
aaronsteers b045532
clean up test module structure and pytest hooks
aaronsteers dc95a71
remove extra pytest files
aaronsteers a36d6a9
add usage docs
aaronsteers da442a9
clean up
aaronsteers 623df9c
new cli
aaronsteers c65d832
feat: Add build command to CDK CLI
devin-ai-integration[bot] bd7a765
fix: Update formatting for build command files
devin-ai-integration[bot] dd1f040
add airbyte-cdk cli scaffold
aaronsteers 0209c1e
add comment
aaronsteers 2a32e27
feat: Add Click-based CLI interface
devin-ai-integration[bot] 653bda1
chore: Update poetry.lock
devin-ai-integration[bot] 9510f2d
fix: Add type annotations to CLI functions
devin-ai-integration[bot] 84d649b
feat: Always build for both AMD64 and ARM64 architectures
devin-ai-integration[bot] f3e1743
fix: Format code with ruff
devin-ai-integration[bot] 2bc8848
fix: Address PR comments - remove infer_connector_language, use build…
devin-ai-integration[bot] ca1bbbe
fix: Update Dockerignore to use * pattern and fix formatting
devin-ai-integration[bot] 373f263
fix: Format all files with ruff
devin-ai-integration[bot] 430dd8d
docs: Move build.md content to __init__.py docstring
devin-ai-integration[bot] fe946ac
docs: Update __init__.py docstring with build command documentation
devin-ai-integration[bot] af3e56e
Delete airbyte_cdk/cli/entrypoint/run.py
aaronsteers 7b6dc5d
fix: Clean up duplicate code in _run.py and update imports
devin-ai-integration[bot] 2b1db11
refactor: Move build functionality to utils/docker and metadata model…
devin-ai-integration[bot] 4d45bab
fix: Remove unsupported --ignorefile flag and implement proper cleanu…
devin-ai-integration[bot] 7e68925
fix: Add fallback to single platform build when multi-platform not av…
devin-ai-integration[bot] df4b315
fix: Update dockerignore to include source_* directories
devin-ai-integration[bot] 38c7166
add helper functions: find_connector_root(), find_connector_root_from…
aaronsteers 83eaca9
enhancement: Improve build process to match airbyte-ci's approach wit…
devin-ai-integration[bot] 789f109
fix: Apply ruff formatting to fix CI checks
devin-ai-integration[bot] c371030
misc fixes (working now)
aaronsteers 8892914
delete unused devtools.py file
aaronsteers ba09434
fix: Add run_command function to build.py and update imports
devin-ai-integration[bot] b25671b
fix: Add run_command function to build.py
devin-ai-integration[bot] de07f9d
clean up cli module structure
aaronsteers a330cb7
update cli entrypoint
aaronsteers 14b1385
add image command stub
aaronsteers 1b45851
add --version support
aaronsteers 67e4aca
Merge remote-tracking branch 'origin/main' into aj/feat/add-standard-…
aaronsteers 79b11dc
ruff format
aaronsteers aec8602
chore: `poetry add rich` (transitive dependency)
aaronsteers ea848b0
remove click declaration (replaced by rich-click)
aaronsteers 4fc4932
typing fixes
aaronsteers 0cc217e
fix deptry issues
aaronsteers d7ee4f1
Merge branch 'aj/feat/add-standard-tests-cli' into devin/1744841809-a…
aaronsteers e1c7c13
delete unnecessay
aaronsteers ba0ab6b
move docstring
aaronsteers 61fdced
fully move model to models.connector_metadata
aaronsteers f293e7e
clean up
aaronsteers 7daa7f0
fix docstring
aaronsteers 4ba7e5c
clean up cli module
aaronsteers 49c9045
revert format update
aaronsteers d9c1ced
lean into pydantic model
aaronsteers 3103d0c
clean up
aaronsteers 52546be
clean up
aaronsteers 63cbc9b
fix: Address PR feedback - fix typo in __all__, update docstring, and…
devin-ai-integration[bot] c8f91cf
fix: Correct CLI entry point in pyproject.toml
devin-ai-integration[bot] 71652ef
Apply suggestions from code review
aaronsteers 78b173c
add-back the help text
aaronsteers 598d454
refactor, tidy up
aaronsteers c4f9eee
pull in refactoring from downstream pr
aaronsteers 5f025ba
add manifest help text
aaronsteers 6fdf2c6
feat: add secrets fetch command to CDK CLI (#494)
devin-ai-integration[bot] db52db6
apply suggestion (owner-only read/write)
aaronsteers 4d524b6
ensure we print the absolute path so secrets don't get lost
aaronsteers 72514ac
allow imports outside of cwd
aaronsteers 8cdf876
remove stale files from other PR
aaronsteers 032d5e7
remove stale declarative module
aaronsteers 7de9ab7
fix imports and such
aaronsteers a28e63b
tidy up cli help text declarations
aaronsteers 1b47886
Merge branch 'main' into aj/feat/add-standard-tests-cli
aaronsteers bcf5d9f
Merge branch 'aj/feat/add-standard-tests-cli' into devin/1744841809-a…
aaronsteers 871cb71
refactor docker implementation
aaronsteers 42af39d
toggle default arch, fix return value for verify step
aaronsteers 964a215
better err handling
aaronsteers d7f9c7b
clean up
aaronsteers 3e9183b
tidy docstring
aaronsteers 6ea0623
ruff fix
aaronsteers ea5fd85
poe lock
aaronsteers 0229421
fix mypy
aaronsteers 297a98c
fix base image
aaronsteers f7b19e1
use uv instead of pip for installs
aaronsteers 7a87b62
use rich click
aaronsteers f7e4ddf
refactor, use build args
aaronsteers e45e19b
Merge branch 'main' into aj/feat/add-standard-tests-cli
aaronsteers bf7ce53
Auto-commit Resolving dependencies... changes
f323849
improve language detection
aaronsteers 51301d4
remove java image option
aaronsteers e40f3a1
Merge branch 'main' into aj/feat/add-standard-tests-cli
aaronsteers c12242d
mypy fix
aaronsteers 86e0a60
clean up arch implementation
aaronsteers b0ba703
Auto-fix lint and format issues
4f383fc
chore: add optional 'dev' dependency
aaronsteers b012495
Auto-commit Resolving dependencies... changes
9ea97fa
Merge branch 'main' into aj/feat/add-standard-tests-cli
aaronsteers fb51377
Merge branch 'aj/feat/add-standard-tests-cli' into aj/feat/add-cli-bu…
aaronsteers 7d93966
Merge branch 'main' into aj/feat/add-cli-build-command
aaronsteers 02e7ce1
fix multi-docker tag
aaronsteers 3981729
remove unused import
aaronsteers eb8cea9
reorder docker build args
aaronsteers 53dbe55
add error handling for missing metadata.yml file
aaronsteers dcbc7cc
improve err handling for connector image build
aaronsteers 8341969
add exception handling for tag errors
aaronsteers 3d73ded
ruff format fix
aaronsteers File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1 +1,9 @@ | ||
# Copyright (c) 2024 Airbyte, Inc., all rights reserved. | ||
# Copyright (c) 2025 Airbyte, Inc., all rights reserved. | ||
"""The `airbyte-cdk.cli` module provides command-line interfaces for the Airbyte CDK. | ||
|
||
As of now, it includes the following CLI entry points: | ||
|
||
- `airbyte-cdk`: Commands for working with connectors. | ||
- `source-declarative-manifest`: Directly invoke the declarative manifests connector. | ||
|
||
""" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,97 @@ | ||
"""Models to represent the structure of a `metadata.yaml` file.""" | ||
|
||
from __future__ import annotations | ||
|
||
from enum import Enum | ||
from pathlib import Path | ||
|
||
import yaml | ||
from pydantic import BaseModel, Field | ||
|
||
|
||
class ConnectorLanguage(str, Enum): | ||
"""Connector implementation language.""" | ||
|
||
PYTHON = "python" | ||
JAVA = "java" | ||
LOW_CODE = "low-code" | ||
MANIFEST_ONLY = "manifest-only" | ||
UNKNOWN = "unknown" | ||
|
||
|
||
class ConnectorBuildOptions(BaseModel): | ||
"""Connector build options from metadata.yaml.""" | ||
|
||
model_config = {"extra": "allow"} | ||
|
||
baseImage: str | None = Field( | ||
None, | ||
description="Base image to use for building the connector", | ||
) | ||
path: str | None = Field( | ||
None, | ||
description="Path to the connector code within the repository", | ||
) | ||
|
||
|
||
class ConnectorMetadata(BaseModel): | ||
"""Connector metadata from metadata.yaml.""" | ||
|
||
model_config = {"extra": "allow"} | ||
|
||
dockerRepository: str = Field(..., description="Docker repository for the connector image") | ||
dockerImageTag: str = Field(..., description="Docker image tag for the connector") | ||
|
||
tags: list[str] = Field( | ||
default=[], | ||
description="List of tags for the connector", | ||
) | ||
|
||
@property | ||
def language(self) -> ConnectorLanguage: | ||
"""Get the connector language.""" | ||
for tag in self.tags: | ||
if tag.startswith("language:"): | ||
language = tag.split(":", 1)[1] | ||
if language == "python": | ||
return ConnectorLanguage.PYTHON | ||
elif language == "java": | ||
return ConnectorLanguage.JAVA | ||
elif language == "low-code": | ||
return ConnectorLanguage.LOW_CODE | ||
elif language == "manifest-only": | ||
return ConnectorLanguage.MANIFEST_ONLY | ||
|
||
return ConnectorLanguage.UNKNOWN | ||
|
||
connectorBuildOptions: ConnectorBuildOptions | None = Field( | ||
None, description="Options for building the connector" | ||
) | ||
|
||
|
||
class MetadataFile(BaseModel): | ||
"""Represents the structure of a metadata.yaml file.""" | ||
|
||
model_config = {"extra": "allow"} | ||
|
||
data: ConnectorMetadata = Field(..., description="Connector metadata") | ||
|
||
@classmethod | ||
def from_file( | ||
cls, | ||
file_path: Path, | ||
) -> MetadataFile: | ||
"""Load metadata from a YAML file.""" | ||
if not file_path.exists(): | ||
raise FileNotFoundError(f"Metadata file not found: {file_path!s}") | ||
|
||
metadata_content = file_path.read_text() | ||
metadata_dict = yaml.safe_load(metadata_content) | ||
|
||
if not metadata_dict or "data" not in metadata_dict: | ||
raise ValueError( | ||
"Invalid metadata format: missing 'data' field in YAML file '{file_path!s}'" | ||
) | ||
|
||
metadata_file = MetadataFile.model_validate(metadata_dict) | ||
return metadata_file |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.