Skip to content

feat(metadata-service): Add schema support for smoke test scenarios #69072

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: aj/fix-typed-ast-metadata-service
Choose a base branch
from
Open

feat(metadata-service): Add schema support for smoke test scenarios #69072

wants to merge 2 commits into from

Conversation

@aaronsteers
Copy link
Collaborator

@aaronsteers aaronsteers commented Oct 29, 2025 •
edited
Loading

Builds on:

What

Add JSON schema support for the new smoke tests configuration format introduced in airbytehq/airbyte-python-cdk#775.

This enables connectors to define smoke test scenarios in their metadata.yaml files under data.connectorTestSuitesOptions.

How

  1. Created SmokeTestScenario.yaml - New schema file defining the structure for individual smoke test scenarios with the following properties:

    • name (required): Name of the test scenario
    • config_file: Path to config file
    • config_settings: Dictionary to override config
    • expect_failure: Whether the test should fail
    • only_streams: Streams to include
    • exclude_streams: Streams to exclude
    • suggested_streams_only: Whether to limit to suggested streams
    • configured_catalog_path: Path to pre-configured catalog

  2. Updated ConnectorTestSuiteOptions.yaml - Added "smokeTests" to the suite enum and added a scenarios property (array of SmokeTestScenario)

  3. Regenerated models - Ran poetry run poe generate-models to generate Python models and bundled JSON schema

Review guide

⚠️ Key areas to review:

  1. SmokeTestScenario.yaml - Verify field names and types match the CDK's ConnectorTestScenario model in scenario.py

    • Note: suggested_streams field was intentionally excluded as it's derived from top-level metadata, not user input

  2. ConnectorTestSuiteOptions.yaml - The scenarios field is optional for all suite types but only applicable for smokeTests. Consider if JSON schema should enforce this with conditional validation (e.g., if/then schema keywords)

  3. Generated files side effects - The regeneration process made incidental changes to other files:

    • Fixed type: const → type: string in StreamBreakingChangeScope
    • Refactored ConnectorIPCOptions structure
    • These weren't explicitly intended; verify they're safe

  4. Bundled schema - Check ConnectorMetadataDefinitionV0.json has the smoke test scenario definition properly inlined

  5. Integration testing - This PR only tests schema generation. Should be tested with:

    • An actual connector metadata.yaml using smokeTests
    • Verification that CDK can parse the metadata
    • Validation behavior for invalid configurations

User Impact

Connector developers can now define smoke test scenarios in their metadata.yaml files:

data:
  connectorTestSuitesOptions:
    - suite: smokeTests
      scenarios:
        - name: default
          config_file: secrets/config_oauth.json
          config_settings:
            start_date: "2025-01-01T00:00:00Z"
          only_streams:
            - users

The metadata service will validate these configurations against the schema.

Can this PR be safely reverted and rolled back?

  • YES 💚

This is purely additive - adds a new optional schema structure without modifying existing functionality.

Link to Devin run: https://app.devin.ai/sessions/2df71103d1fd4177a9691557c003b6ee
Requested by: AJ Steers ([email protected]) / @aaronsteers
Related: airbytehq/airbyte-python-cdk#775

@devin-ai-integration
Copy link
Contributor

devin-ai-integration bot commented Oct 29, 2025

Original prompt from AJ Steers 
Received message in Slack channel #ask-devin-ai:

@Devin - look at the YAML JSON schema files adjacent to this one: airbyte-ci/connectors/metadata_service/lib/metadata_service/models/src/ActorDefinitionResourceRequirements.yaml

We want these files to make a coherent JSON schema, independently if easy to do, but more importantly together with cross references. However it seems we need a ton of manual coding to merge them into a single JSON scheme of file reference that can be used to validate yaml and JSON files.

Are we missing an important spec component here, for instance should we writing refs differently?
Thread URL: https://airbytehq-team.slack.com/archives/C08BHPUMEPJ/p1761603745188709?thread_ts=1761603745.188709

@devin-ai-integration
Copy link
Contributor

devin-ai-integration bot commented Oct 29, 2025

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@devin-ai-integration
Copy link
Contributor

devin-ai-integration bot commented Oct 29, 2025

Devin is archived and cannot be woken up. Please unarchive Devin if you want to continue using it.

6 similar comments
@devin-ai-integration
Copy link
Contributor

devin-ai-integration bot commented Oct 29, 2025

Devin is archived and cannot be woken up. Please unarchive Devin if you want to continue using it.

@devin-ai-integration
Copy link
Contributor

devin-ai-integration bot commented Oct 29, 2025

Devin is archived and cannot be woken up. Please unarchive Devin if you want to continue using it.

@devin-ai-integration
Copy link
Contributor

devin-ai-integration bot commented Oct 29, 2025

Devin is archived and cannot be woken up. Please unarchive Devin if you want to continue using it.

@devin-ai-integration
Copy link
Contributor

devin-ai-integration bot commented Oct 29, 2025

Devin is archived and cannot be woken up. Please unarchive Devin if you want to continue using it.

@devin-ai-integration
Copy link
Contributor

devin-ai-integration bot commented Oct 29, 2025

Devin is archived and cannot be woken up. Please unarchive Devin if you want to continue using it.

@devin-ai-integration
Copy link
Contributor

devin-ai-integration bot commented Oct 29, 2025

Devin is archived and cannot be woken up. Please unarchive Devin if you want to continue using it.

@github-actions
Copy link
Contributor

github-actions bot commented Oct 29, 2025

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

Helpful Resources

PR Slash Commands

Airbyte Maintainers (that's you!) can execute the following slash commands on your PR:

  • /format-fix - Fixes most formatting issues.
  • /bump-version - Bumps connector versions.
    • You can specify a custom changelog by passing changelog. Example: /bump-version changelog="My cool update"
    • Leaving the changelog arg blank will auto-populate the changelog from the PR title.
  • /run-cat-tests - Runs legacy CAT tests (Connector Acceptance Tests)
  • /build-connector-images - Builds and publishes a pre-release docker image for the modified connector(s).
  • JVM connectors:
    • /update-connector-cdk-version connector=<CONNECTOR_NAME> - Updates the specified connector to the latest CDK version.
      Example: /update-connector-cdk-version connector=destination-bigquery
    • /bump-bulk-cdk-version type=patch changelog='foo' - Bump the Bulk CDK's version. type can be major/minor/patch.
  • Python connectors:
    • /poe connector source-example lock - Run the Poe lock task on the source-example connector, committing the results back to the branch.
    • /poe source example lock - Alias for /poe connector source-example lock.
    • /poe source example use-cdk-branch my/branch - Pin the source-example CDK reference to the branch name specified.
    • /poe source example use-cdk-latest - Update the source-example CDK dependency to the latest available version.

📝 Edit this welcome message.

@aaronsteers aaronsteers changed the base branch from master to aj/fix-typed-ast-metadata-service October 29, 2025 07:16
@aaronsteers aaronsteers requested a review from a team as a code owner October 29, 2025 07:16
@devin-ai-integration @aaronsteers
feat(metadata-service): Add schema support for smoke test scenarios
af7ab8f 
Add JSON schema definitions to support the new smoke tests configuration
format introduced in airbytehq/airbyte-python-cdk#775.

Changes:
- Created SmokeTestScenario.yaml schema defining the structure for individual
  smoke test scenarios with properties: name, config_file, config_settings,
  expect_failure, only_streams, exclude_streams, suggested_streams_only,
  and configured_catalog_path
- Updated ConnectorTestSuiteOptions.yaml to add 'smokeTests' to the suite enum
  and added 'scenarios' property to support the new smoke test format
- Regenerated Python models and bundled JSON schema

This enables connectors to define smoke tests in their metadata.yaml file under
data.connectorTestSuitesOptions with the following structure:

  connectorTestSuitesOptions:
    - suite: smokeTests
      scenarios:
        - name: default
          config_file: secrets/config_oauth.json
          config_settings:
            start_date: "2025-01-01T00:00:00Z"
          only_streams:
            - users

Related: airbytehq/airbyte-python-cdk#775
Co-Authored-By: AJ Steers <[email protected]>
@devin-ai-integration devin-ai-integration bot force-pushed the devin/1761721670-add-smoke-tests-schema branch from c6c225b to af7ab8f Compare October 29, 2025 07:22
@aaronsteers aaronsteers mentioned this pull request Oct 29, 2025
Open
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dbgold17 dbgold17 Awaiting requested review from dbgold17

@clnoll clnoll Awaiting requested review from clnoll

Assignees

@aaronsteers aaronsteers

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@aaronsteers