Skip to content

refactor: connectors can launch themselves; deprecate: AirbyteEntrypoint and launch(), add SerDe methods on critical protocol classes #562

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 15 commits into
base: main
Choose a base branch
from

Conversation

aaronsteers
Copy link
Contributor

@aaronsteers aaronsteers commented May 21, 2025

Request: Early Feedback.

Goal here is to detangle the tightly-coupled interdependencies:

  • AirbyteEntrypoint class
  • launch() method
  • Connector and Source classes themselves.

Enumerated changes:

  • parse_cli_args() made a dedicated helper function. Moved to its own utils module, ensuring no circular dependencies between the airbyte_entrypoint module and the connector class modules.
  • All Connector classes inherit class method called launch_with_cli_args()
  • The launch() function is deprecated (but not removed).
    • Internally, launch() it is now implemented via source.launch_with_cli_args() Calling it will emit a deprecation warning.
  • Goal is to fully deprecate and remove AirbyteEntrypoint (although probably not in this iteration).

Context

I finally accepted taking on this work, as it is needed to unblock:

The problem is: the AirbyteEntrypoint currently gatekeeps access to the connector class itself. And AirbyteEntrypoint doesn't have flexibility into calling results into memory vs pushing results to STDOUT.

Related: the airbyte_cdk.test.entrypoint_output.EntrypointOutput in some ways tries to mitigate this, but it has to in some cases fully re-invent the AirbyteEntrypoint implementation.

And since neither of these gives any flexibility in modes of execution or control of RAM overflow, this PR consolidates the execution to have fewer parts, with the (hopefully) the right interface layers to accomplish execution with minimal wheel-reinventing for callers.

@aaronsteers aaronsteers changed the title refactor: connectors can launch themselves; deprecate: AirbyteEntrypoint and launch() refactor: connectors can launch themselves; deprecate: AirbyteEntrypoint and launch(), add SerDe methods on critical protocol classes May 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant