diff --git a/airbyte-integrations/connectors/source-klaviyo/.coveragerc b/airbyte-integrations/connectors/source-klaviyo/.coveragerc deleted file mode 100644 index f75d1e84fd28f..0000000000000 --- a/airbyte-integrations/connectors/source-klaviyo/.coveragerc +++ /dev/null @@ -1,3 +0,0 @@ -[run] -omit = - source_klaviyo/run.py diff --git a/airbyte-integrations/connectors/source-klaviyo/README.md b/airbyte-integrations/connectors/source-klaviyo/README.md index 1a876c45272d3..0eea868131d50 100644 --- a/airbyte-integrations/connectors/source-klaviyo/README.md +++ b/airbyte-integrations/connectors/source-klaviyo/README.md @@ -1,49 +1,22 @@ # Klaviyo source connector -This is the repository for the Klaviyo source connector, written in Python. -For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.com/integrations/sources/klaviyo). +This directory contains the manifest-only connector for `source-klaviyo`. +This _manifest-only_ connector is not a Python package on its own, as it runs inside of the base `source-declarative-manifest` image. -## Local development - -### Prerequisites - -- Python (~=3.9) -- Poetry (~=1.7) - installation instructions [here](https://python-poetry.org/docs/#installation) - -### Installing the connector - -From this connector directory, run: - -```bash -poetry install --with dev -``` - -### Create credentials +For information about how to configure and use this connector within Airbyte, see [the connector's full documentation](https://docs.airbyte.com/integrations/sources/klaviyo). -**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.com/integrations/sources/klaviyo) -to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_klaviyo/spec.yaml` file. -Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information. -See `sample_files/sample_config.json` for a sample config file. - -### Locally running the connector - -``` -poetry run source-klaviyo spec -poetry run source-klaviyo check --config secrets/config.json -poetry run source-klaviyo discover --config secrets/config.json -poetry run source-klaviyo read --config secrets/config.json --catalog integration_tests/configured_catalog.json -``` +## Local development -### Running unit tests +We recommend using the Connector Builder to edit this connector. +Using either Airbyte Cloud or your local Airbyte OSS instance, navigate to the **Builder** tab and select **Import a YAML**. +Then select the connector's `manifest.yaml` file to load the connector into the Builder. You're now ready to make changes to the connector! -To run unit tests locally, from the connector directory run: - -``` -poetry run pytest unit_tests -``` +If you prefer to develop locally, you can follow the instructions below. ### Building the docker image +You can build any manifest-only connector with `airbyte-ci`: + 1. Install [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md) 2. Run the following command to build the docker image: @@ -53,18 +26,24 @@ airbyte-ci connectors --name=source-klaviyo build An image will be available on your host with the tag `airbyte/source-klaviyo:dev`. +### Creating credentials + +**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.com/integrations/sources/klaviyo) +to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `spec` object in the connector's `manifest.yaml` file. +Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information. + ### Running as a docker container -Then run any of the connector commands as follows: +Then run any of the standard source connector commands: -``` +```bash docker run --rm airbyte/source-klaviyo:dev spec docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-klaviyo:dev check --config /secrets/config.json docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-klaviyo:dev discover --config /secrets/config.json docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-klaviyo:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json ``` -### Running our CI test suite +### Running the CI test suite You can run our full test suite locally using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md): @@ -72,33 +51,15 @@ You can run our full test suite locally using [`airbyte-ci`](https://github.com/ airbyte-ci connectors --name=source-klaviyo test ``` -### Customizing acceptance Tests - -Customize `acceptance-test-config.yml` file to configure acceptance tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information. -If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py. - -### Dependency Management - -All of your dependencies should be managed via Poetry. -To add a new dependency, run: - -```bash -poetry add -``` - -Please commit the changes to `pyproject.toml` and `poetry.lock` files. - ## Publishing a new version of the connector -You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what? - -1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=source-klaviyo test` -2. Bump the connector version (please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors)): - - bump the `dockerImageTag` value in in `metadata.yaml` - - bump the `version` value in `pyproject.toml` -3. Make sure the `metadata.yaml` content is up to date. +If you want to contribute changes to `source-klaviyo`, here's how you can do that: +1. Make your changes locally, or load the connector's manifest into Connector Builder and make changes there. +2. Make sure your changes are passing our test suite with `airbyte-ci connectors --name=source-klaviyo test` +3. Bump the connector version (please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors)): + - bump the `dockerImageTag` value in in `metadata.yaml` 4. Make sure the connector documentation and its changelog is up to date (`docs/integrations/sources/klaviyo.md`). 5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention). 6. Pat yourself on the back for being an awesome contributor. 7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master. -8. Once your PR is merged, the new version of the connector will be automatically published to Docker Hub and our connector registry. +8. Once your PR is merged, the new version of the connector will be automatically published to Docker Hub and our connector registry. \ No newline at end of file diff --git a/airbyte-integrations/connectors/source-klaviyo/acceptance-test-config.yml b/airbyte-integrations/connectors/source-klaviyo/acceptance-test-config.yml index c1b121625086d..9f017cc565de1 100644 --- a/airbyte-integrations/connectors/source-klaviyo/acceptance-test-config.yml +++ b/airbyte-integrations/connectors/source-klaviyo/acceptance-test-config.yml @@ -34,6 +34,6 @@ acceptance_tests: timeout_seconds: 7200 spec: tests: - - spec_path: source_klaviyo/spec.json + - spec_path: manifest.yaml connector_image: airbyte/source-klaviyo:dev test_strictness_level: high diff --git a/airbyte-integrations/connectors/source-klaviyo/components.py b/airbyte-integrations/connectors/source-klaviyo/components.py new file mode 100644 index 0000000000000..fae11544492cb --- /dev/null +++ b/airbyte-integrations/connectors/source-klaviyo/components.py @@ -0,0 +1,322 @@ +# +# Copyright (c) 2024 Airbyte, Inc., all rights reserved. +# +import logging +from abc import ABC +from dataclasses import dataclass +from typing import Any, Dict, Iterable, Mapping, Optional, Union + +import dpath +import requests +from requests.exceptions import InvalidURL + +from airbyte_cdk.sources.declarative.declarative_stream import DeclarativeStream +from airbyte_cdk.sources.declarative.extractors.dpath_extractor import DpathExtractor +from airbyte_cdk.sources.declarative.interpolation.interpolated_string import InterpolatedString +from airbyte_cdk.sources.declarative.migrations.state_migration import StateMigration +from airbyte_cdk.sources.declarative.requesters.error_handlers import DefaultErrorHandler +from airbyte_cdk.sources.declarative.requesters.error_handlers.backoff_strategies import WaitTimeFromHeaderBackoffStrategy +from airbyte_cdk.sources.declarative.transformations import RecordTransformation +from airbyte_cdk.sources.message.repository import InMemoryMessageRepository +from airbyte_cdk.sources.streams.call_rate import APIBudget +from airbyte_cdk.sources.streams.http.error_handlers import BackoffStrategy, ErrorHandler, HttpStatusErrorHandler +from airbyte_cdk.sources.streams.http.error_handlers.default_error_mapping import DEFAULT_ERROR_MAPPING +from airbyte_cdk.sources.streams.http.error_handlers.response_models import ErrorResolution, FailureType, ResponseAction +from airbyte_cdk.sources.streams.http.http_client import HttpClient +from airbyte_cdk.sources.types import Config, StreamSlice, StreamState + + +ARCHIVED_EMAIL = {"archived": "true", "campaign_type": "email"} +NOT_ARCHIVED_EMAIL = {"archived": "false", "campaign_type": "email"} + +ARCHIVED = {"archived": "true"} +NOT_ARCHIVED = {"archived": "false"} + +DEFAULT_START_DATE = "2012-01-01T00:00:00Z" + + +class ArchivedToPerPartitionStateMigration(StateMigration, ABC): + """ + Updates old format state to new per partitioned format. + Partitions: [{archived: True}, {archived: False}] + Default built in airbyte cdk migration will recognise only top-level field cursor value(updated_at), + but for partition {archived: True} source should use cursor value from archived object. + + Example input state: + { + "updated_at": "2020-10-10T00:00:00+00:00", + "archived": { + "updated_at": "2021-10-10T00:00:00+00:00" + } + } + + Example output state: + { + "partition":{ "archived":"true" }, + "cursor":{ "updated_at":"2021-10-10T00:00:00+00:00" } + } + { + "partition":{ "archived":"false" }, + "cursor":{ "updated_at":"2020-10-10T00:00:00+00:00" } + } + """ + + declarative_stream: DeclarativeStream + config: Config + + def __init__(self, declarative_stream: DeclarativeStream, config: Config): + self._config = config + self.declarative_stream = declarative_stream + self._cursor = declarative_stream.incremental_sync + self._parameters = declarative_stream.parameters + self._cursor_field = InterpolatedString.create(self._cursor.cursor_field, parameters=self._parameters).eval(self._config) + + def get_archived_cursor_value(self, stream_state: Mapping[str, Any]): + return stream_state.get("archived", {}).get(self._cursor.cursor_field, self._config.get("start_date", DEFAULT_START_DATE)) + + def get_not_archived_cursor_value(self, stream_state: Mapping[str, Any]): + return stream_state.get(self._cursor.cursor_field, self._config.get("start_date", DEFAULT_START_DATE)) + + def should_migrate(self, stream_state: Mapping[str, Any]) -> bool: + return bool("states" not in stream_state and stream_state) + + def migrate(self, stream_state: Mapping[str, Any]) -> Mapping[str, Any]: + if not self.should_migrate(stream_state): + return stream_state + is_archived_updated_at = self.get_archived_cursor_value(stream_state) + is_not_archived_updated_at = self.get_not_archived_cursor_value(stream_state) + + migrated_stream_state = { + "states": [ + {"partition": ARCHIVED, "cursor": {self._cursor.cursor_field: is_archived_updated_at}}, + {"partition": NOT_ARCHIVED, "cursor": {self._cursor.cursor_field: is_not_archived_updated_at}}, + ] + } + return migrated_stream_state + + +class CampaignsStateMigration(ArchivedToPerPartitionStateMigration): + """ + Campaigns stream has 2 partition field: archived and campaign_type(email, sms). + Previous API version didn't return sms in campaigns output so we need to migrate only email partition. + + Example input state: + { + "updated_at": "2020-10-10T00:00:00+00:00", + "archived": { + "updated_at": "2021-10-10T00:00:00+00:00" + } + } + Example output state: + { + "partition":{ "archived":"true","campaign_type":"email" }, + "cursor":{ "updated_at":"2021-10-10T00:00:00+00:00" } + } + { + "partition":{ "archived":"false","campaign_type":"email" }, + "cursor":{ "updated_at":"2020-10-10T00:00:00+00:00" } + } + """ + + def migrate(self, stream_state: Mapping[str, Any]) -> Mapping[str, Any]: + if not self.should_migrate(stream_state): + return stream_state + is_archived_updated_at = self.get_archived_cursor_value(stream_state) + is_not_archived_updated_at = self.get_not_archived_cursor_value(stream_state) + + migrated_stream_state = { + "states": [ + {"partition": ARCHIVED_EMAIL, "cursor": {self._cursor.cursor_field: is_archived_updated_at}}, + {"partition": NOT_ARCHIVED_EMAIL, "cursor": {self._cursor.cursor_field: is_not_archived_updated_at}}, + ] + } + return migrated_stream_state + + +class CampaignsDetailedTransformation(RecordTransformation): + """ + Campaigns detailed stream fetches detailed campaigns info: + estimated_recipient_count: integer + campaign_messages: list of objects. + + To get this data CampaignsDetailedTransformation makes extra API requests: + https://a.klaviyo.com/api/campaign-recipient-estimations/{campaign_id} + https://developers.klaviyo.com/en/v2024-10-15/reference/get_messages_for_campaign + """ + + config: Config + + api_revision = "2024-10-15" + url_base = "https://a.klaviyo.com/api/" + name = "campaigns_detailed" + max_retries = 5 + max_time = 60 * 10 + + def __init__(self, config: Config, **kwargs): + self.logger = logging.getLogger("airbyte") + self.config = config + self._api_key = self.config["api_key"] + self._http_client = HttpClient( + name=self.name, + logger=self.logger, + error_handler=self.get_error_handler(), + api_budget=APIBudget(policies=[]), + backoff_strategy=self.get_backoff_strategy(), + message_repository=InMemoryMessageRepository(), + ) + + def transform( + self, + record: Dict[str, Any], + config: Optional[Config] = None, + stream_state: Optional[StreamState] = None, + stream_slice: Optional[StreamSlice] = None, + ) -> None: + self._set_recipient_count(record) + self._set_campaign_message(record) + + def _set_recipient_count(self, record: Mapping[str, Any]) -> None: + campaign_id = record["id"] + _, recipient_count_response = self._http_client.send_request( + url=f"{self.url_base}campaign-recipient-estimations/{campaign_id}", + request_kwargs={}, + headers=self.request_headers(), + http_method="GET", + ) + record["estimated_recipient_count"] = ( + recipient_count_response.json().get("data", {}).get("attributes", {}).get("estimated_recipient_count", 0) + ) + + def _set_campaign_message(self, record: Mapping[str, Any]) -> None: + messages_link = record.get("relationships", {}).get("campaign-messages", {}).get("links", {}).get("related") + if messages_link: + _, campaign_message_response = self._http_client.send_request( + url=messages_link, request_kwargs={}, headers=self.request_headers(), http_method="GET" + ) + record["campaign_messages"] = campaign_message_response.json().get("data") + + def get_backoff_strategy(self) -> BackoffStrategy: + return WaitTimeFromHeaderBackoffStrategy(header="Retry-After", max_waiting_time_in_seconds=self.max_time, parameters={}, config={}) + + def request_headers(self): + return { + "Accept": "application/json", + "Revision": self.api_revision, + "Authorization": f"Klaviyo-API-Key {self._api_key}", + } + + def get_error_handler(self) -> ErrorHandler: + error_mapping = DEFAULT_ERROR_MAPPING | { + 404: ErrorResolution(ResponseAction.IGNORE, FailureType.config_error, "Resource not found. Ignoring.") + } + + return HttpStatusErrorHandler(logger=self.logger, error_mapping=error_mapping, max_retries=self.max_retries) + + +@dataclass +class KlaviyoIncludedFieldExtractor(DpathExtractor): + def extract_records(self, response: requests.Response) -> Iterable[Mapping[str, Any]]: + # Evaluate and retrieve the extraction paths + evaluated_field_paths = [field_path.eval(self.config) for field_path in self._field_path] + target_records = self.extract_records_by_path(response, evaluated_field_paths) + included_relations = list(self.extract_records_by_path(response, ["included"])) + + # Update target records with included records + updated_records = self.update_target_records_with_included(target_records, included_relations) + yield from updated_records + + @staticmethod + def update_target_records_with_included( + target_records: Iterable[Mapping[str, Any]], included_relations: Iterable[Mapping[str, Any]] + ) -> Iterable[Mapping[str, Any]]: + for target_record in target_records: + target_relationships = target_record.get("relationships", {}) + + for included_relation in included_relations: + included_relation_attributes = included_relation.get("attributes", {}) + included_relation_type = included_relation["type"] + included_relation_id = included_relation["id"] + + target_relationship_id = target_relationships.get(included_relation_type, {}).get("data", {}).get("id") + + if included_relation_id == target_relationship_id: + target_relationships[included_relation_type]["data"].update(included_relation_attributes) + + yield target_record + + def extract_records_by_path(self, response: requests.Response, field_paths: list = None) -> Iterable[Mapping[str, Any]]: + try: + response_body = response.json() + except Exception as e: + raise Exception(f"Failed to parse response body as JSON: {e}") + + # Extract data from the response body based on the provided field paths + if not field_paths: + extracted_data = response_body + else: + field_path_str = "/".join(field_paths) # Convert list of field paths to a single string path for dpath + if "*" in field_path_str: + extracted_data = dpath.values(response_body, field_path_str) + else: + extracted_data = dpath.get(response_body, field_path_str, default=[]) + + # Yield extracted data as individual records + if isinstance(extracted_data, list): + yield from extracted_data + elif extracted_data: + yield extracted_data + else: + yield from [] + + +class KlaviyoErrorHandler(DefaultErrorHandler): + def interpret_response(self, response_or_exception: Optional[Union[requests.Response, Exception]]) -> ErrorResolution: + """ + We have seen `[Errno -3] Temporary failure in name resolution` a couple of times on two different connections + (1fed2ede-2d33-4543-85e3-7d6e5736075d and 1b276f7d-358a-4fe3-a437-6747fd780eed). Retrying the requests on later syncs is working + which makes it sound like a transient issue. + """ + if isinstance(response_or_exception, InvalidURL): + return ErrorResolution( + response_action=ResponseAction.RETRY, + failure_type=FailureType.transient_error, + error_message="source-klaviyo has faced a temporary DNS resolution issue. Retrying...", + ) + return super().interpret_response(response_or_exception) + + +class PerPartitionToSingleStateMigration(StateMigration): + """ + Transforms the input state for per-partitioned streams from the legacy format to the low-code format. + The cursor field and partition ID fields are automatically extracted from the stream's DatetimebasedCursor and SubstreamPartitionRouter. + + Example input state: + { + "partition": {"event_id": "13506132"}, + "cursor": {"datetime": "2120-10-10 00:00:00+00:00"} + } + Example output state: + { + "datetime": "2120-10-10 00:00:00+00:00" + } + """ + + declarative_stream: DeclarativeStream + config: Config + + def __init__(self, declarative_stream: DeclarativeStream, config: Config): + self._config = config + self.declarative_stream = declarative_stream + self._cursor = declarative_stream.incremental_sync + self._parameters = declarative_stream.parameters + self._cursor_field = InterpolatedString.create(self._cursor.cursor_field, parameters=self._parameters).eval(self._config) + + def should_migrate(self, stream_state: Mapping[str, Any]) -> bool: + return "states" in stream_state + + def migrate(self, stream_state: Mapping[str, Any]) -> Mapping[str, Any]: + if not self.should_migrate(stream_state): + return stream_state + + min_state = min(stream_state.get("states"), key=lambda state: state["cursor"][self._cursor_field]) + return min_state.get("cursor") diff --git a/airbyte-integrations/connectors/source-klaviyo/main.py b/airbyte-integrations/connectors/source-klaviyo/main.py deleted file mode 100644 index bda84b0215c4b..0000000000000 --- a/airbyte-integrations/connectors/source-klaviyo/main.py +++ /dev/null @@ -1,9 +0,0 @@ -# -# Copyright (c) 2023 Airbyte, Inc., all rights reserved. -# - -from source_klaviyo.run import run - - -if __name__ == "__main__": - run() diff --git a/airbyte-integrations/connectors/source-klaviyo/manifest.yaml b/airbyte-integrations/connectors/source-klaviyo/manifest.yaml new file mode 100644 index 0000000000000..89bbc394dc944 --- /dev/null +++ b/airbyte-integrations/connectors/source-klaviyo/manifest.yaml @@ -0,0 +1,3345 @@ +version: 6.33.4 + +type: DeclarativeSource + +check: + type: CheckStream + stream_names: + - metrics + +definitions: + streams: + profiles: + type: DeclarativeStream + name: profiles + primary_key: + - id + retriever: + type: SimpleRetriever + requester: + $ref: "#/definitions/base_requester" + path: profiles + http_method: GET + request_parameters: + additional-fields[profile]: >- + {{ 'predictive_analytics' if not + config['disable_fetching_predictive_analytics'] else '' }} + filter: >- + greater-than(updated,{{ + stream_interval.start_time }}) + sort: "updated" + request_headers: + Accept: application/json + Revision: "2024-10-15" + error_handler: + type: CustomErrorHandler + class_name: >- + source_declarative_manifest.components.KlaviyoErrorHandler + backoff_strategies: + - type: WaitTimeFromHeader + header: Retry-After + response_filters: + - type: HttpResponseFilter + action: RATE_LIMITED + http_codes: + - 429 + - type: HttpResponseFilter + action: FAIL + http_codes: + - 401 + - 403 + failure_type: config_error + error_message: >- + Please provide a valid API key and make sure it has + permissions to read specified streams. + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: + - data + schema_normalization: Default + paginator: + type: DefaultPaginator + page_token_option: + type: RequestPath + page_size_option: + type: RequestOption + field_name: page[size] + inject_into: request_parameter + pagination_strategy: + type: CursorPagination + page_size: 100 + cursor_value: "{{ response.get('links', {}).get('next') }}" + incremental_sync: + type: DatetimeBasedCursor + cursor_field: updated + cursor_datetime_formats: + - "%Y-%m-%dT%H:%M:%S.%f%z" + - "%Y-%m-%dT%H:%M:%S%z" + - "%Y-%m-%d %H:%M:%S%z" + datetime_format: "%Y-%m-%dT%H:%M:%S%z" + start_datetime: + type: MinMaxDatetime + datetime: "{{ config.get('start_date', '2012-01-01T00:00:00Z') }}" + datetime_format: "%Y-%m-%dT%H:%M:%S%z" + transformations: + - type: AddFields + fields: + - path: + - updated + value: "{{ record.get('attributes', {}).get('updated') }}" + schema_loader: + type: InlineSchemaLoader + schema: + $ref: "#/schemas/profiles" + global_exclusions: + type: DeclarativeStream + name: global_exclusions + primary_key: + - id + retriever: + type: SimpleRetriever + requester: + $ref: "#/definitions/base_requester" + path: profiles + http_method: GET + request_parameters: + additional-fields[profile]: subscriptions + request_headers: + Accept: application/json + Revision: "2024-10-15" + error_handler: + type: CustomErrorHandler + class_name: >- + source_declarative_manifest.components.KlaviyoErrorHandler + backoff_strategies: + - type: WaitTimeFromHeader + header: Retry-After + response_filters: + - type: HttpResponseFilter + action: RATE_LIMITED + http_codes: + - 429 + - type: HttpResponseFilter + action: FAIL + http_codes: + - 401 + - 403 + failure_type: config_error + error_message: >- + Please provide a valid API key and make sure it has + permissions to read specified streams. + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: + - data + record_filter: + type: RecordFilter + condition: >- + {{ + record['attributes']['subscriptions']['email']['marketing']['suppression'] + }} + paginator: + type: DefaultPaginator + page_token_option: + type: RequestPath + page_size_option: + type: RequestOption + field_name: page[size] + inject_into: request_parameter + pagination_strategy: + type: CursorPagination + page_size: 100 + cursor_value: "{{ response.get('links', {}).get('next') }}" + incremental_sync: + type: DatetimeBasedCursor + cursor_field: updated + cursor_datetime_formats: + - "%Y-%m-%dT%H:%M:%S.%f%z" + - "%Y-%m-%dT%H:%M:%S%z" + - "%Y-%m-%d %H:%M:%S%z" + datetime_format: "%Y-%m-%dT%H:%M:%S%z" + start_datetime: + type: MinMaxDatetime + datetime: "{{ config.get('start_date', '2012-01-01T00:00:00Z') }}" + datetime_format: "%Y-%m-%dT%H:%M:%S%z" + transformations: + - type: AddFields + fields: + - path: + - updated + value: "{{ record.get('attributes', {}).get('updated') }}" + - type: AddFields + fields: + - path: + - attributes + - subscriptions + - email + - marketing + - suppressions + value: >- + {{ + record['attributes']['subscriptions']['email']['marketing']['suppression'] + }} + - type: RemoveFields + field_pointers: + - - attributes + - subscriptions + - email + - marketing + - suppression + schema_loader: + type: InlineSchemaLoader + schema: + $ref: "#/schemas/global_exclusions" + events: + type: DeclarativeStream + name: events + primary_key: + - id + retriever: + type: SimpleRetriever + requester: + $ref: "#/definitions/base_requester" + path: events + http_method: GET + request_parameters: + fields[metric]: name,created,updated,integration + include: metric,attributions + filter: >- + greater-or-equal(datetime,{{ + stream_interval.start_time }}),less-or-equal(datetime,{{ stream_interval.end_time }}) + sort: "datetime" + request_headers: + Accept: application/json + Revision: "2024-10-15" + error_handler: + type: CustomErrorHandler + class_name: >- + source_declarative_manifest.components.KlaviyoErrorHandler + backoff_strategies: + - type: WaitTimeFromHeader + header: Retry-After + response_filters: + - type: HttpResponseFilter + action: RATE_LIMITED + http_codes: + - 429 + - type: HttpResponseFilter + action: FAIL + http_codes: + - 401 + - 403 + failure_type: config_error + error_message: >- + Please provide a valid API key and make sure it has + permissions to read specified streams. + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: + - data + paginator: + type: DefaultPaginator + page_token_option: + type: RequestPath + pagination_strategy: + type: CursorPagination + cursor_value: "{{ response.get('links', {}).get('next') }}" + incremental_sync: + type: DatetimeBasedCursor + cursor_field: datetime + cursor_datetime_formats: + - "%Y-%m-%dT%H:%M:%S.%f%z" + - "%Y-%m-%dT%H:%M:%S%z" + - "%Y-%m-%d %H:%M:%S%z" + datetime_format: "%Y-%m-%dT%H:%M:%S%z" + start_datetime: + type: MinMaxDatetime + datetime: "{{ config.get('start_date', '2012-01-01T00:00:00Z') }}" + datetime_format: "%Y-%m-%dT%H:%M:%S%z" + transformations: + - type: AddFields + fields: + - path: + - datetime + value: "{{ record.get('attributes', {}).get('datetime') }}" + schema_loader: + type: InlineSchemaLoader + schema: + $ref: "#/schemas/events" + events_detailed: + type: DeclarativeStream + state_migrations: + - type: CustomStateMigration + class_name: >- + source_declarative_manifest.components.PerPartitionToSingleStateMigration + name: events_detailed + primary_key: + - id + retriever: + type: SimpleRetriever + requester: + $ref: "#/definitions/base_requester" + path: events + http_method: GET + request_parameters: + include: metric,attributions + fields[metric]: name + filter: >- + greater-than(datetime,{{ + stream_interval.start_time }}) + sort: "datetime" + request_headers: + Accept: application/json + Revision: "2024-10-15" + error_handler: + type: CustomErrorHandler + class_name: >- + source_declarative_manifest.components.KlaviyoErrorHandler + backoff_strategies: + - type: WaitTimeFromHeader + header: Retry-After + response_filters: + - type: HttpResponseFilter + action: RATE_LIMITED + http_codes: + - 429 + - type: HttpResponseFilter + action: FAIL + http_codes: + - 401 + - 403 + failure_type: config_error + error_message: >- + Please provide a valid API key and make sure it has + permissions to read specified streams. + record_selector: + type: RecordSelector + extractor: + type: CustomRecordExtractor + class_name: >- + source_declarative_manifest.components.KlaviyoIncludedFieldExtractor + field_path: + - data + paginator: + type: DefaultPaginator + page_token_option: + type: RequestPath + pagination_strategy: + type: CursorPagination + cursor_value: "{{ response.get('links', {}).get('next') }}" + incremental_sync: + type: DatetimeBasedCursor + cursor_field: "datetime" + cursor_datetime_formats: + - "%Y-%m-%dT%H:%M:%S.%f%z" + - "%Y-%m-%dT%H:%M:%S%z" + - "%Y-%m-%d %H:%M:%S%z" + datetime_format: "%Y-%m-%dT%H:%M:%S%z" + start_datetime: + type: MinMaxDatetime + datetime: "{{ config.get('start_date', '2012-01-01T00:00:00Z') }}" + datetime_format: "%Y-%m-%dT%H:%M:%S%z" + transformations: + - type: AddFields + fields: + - path: + - datetime + value: "{{ record.get('attributes', {}).get('datetime') }}" + schema_loader: + type: InlineSchemaLoader + schema: + $ref: "#/schemas/events_detailed" + email_templates: + type: DeclarativeStream + name: email_templates + primary_key: + - id + retriever: + type: SimpleRetriever + requester: + $ref: "#/definitions/base_requester" + path: templates + http_method: GET + request_parameters: + filter: >- + greater-than(updated,{{ + stream_interval.start_time }}) + sort: "updated" + request_headers: + Accept: application/json + Revision: "2024-10-15" + error_handler: + type: CustomErrorHandler + class_name: >- + source_declarative_manifest.components.KlaviyoErrorHandler + backoff_strategies: + - type: WaitTimeFromHeader + header: Retry-After + response_filters: + - type: HttpResponseFilter + action: RATE_LIMITED + http_codes: + - 429 + - type: HttpResponseFilter + action: FAIL + http_codes: + - 401 + - 403 + failure_type: config_error + error_message: >- + Please provide a valid API key and make sure it has + permissions to read specified streams. + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: + - data + paginator: + type: DefaultPaginator + page_token_option: + type: RequestPath + pagination_strategy: + type: CursorPagination + cursor_value: "{{ response.get('links', {}).get('next') }}" + incremental_sync: + type: DatetimeBasedCursor + cursor_field: updated + cursor_datetime_formats: + - "%Y-%m-%dT%H:%M:%S.%f%z" + - "%Y-%m-%dT%H:%M:%S%z" + - "%Y-%m-%d %H:%M:%S%z" + datetime_format: "%Y-%m-%dT%H:%M:%S%z" + start_datetime: + type: MinMaxDatetime + datetime: "{{ config.get('start_date', '2012-01-01T00:00:00Z') }}" + datetime_format: "%Y-%m-%dT%H:%M:%S%z" + transformations: + - type: AddFields + fields: + - path: + - updated + value: "{{ record.get('attributes', {}).get('updated') }}" + schema_loader: + type: InlineSchemaLoader + schema: + $ref: "#/schemas/email_templates" + campaigns: + type: DeclarativeStream + state_migrations: + - type: CustomStateMigration + class_name: >- + source_declarative_manifest.components.CampaignsStateMigration + name: campaigns + primary_key: + - id + retriever: + type: SimpleRetriever + requester: + $ref: "#/definitions/base_requester" + path: campaigns + http_method: GET + request_parameters: + filter: >- + and(greater-or-equal(updated_at,{{ + stream_interval.start_time }}),less-or-equal(updated_at,{{ stream_interval.end_time + }}),equals(messages.channel,'{{ stream_partition['campaign_type'] + }}'),equals(archived,{{ stream_partition['archived'] }})) + sort: "updated_at" + request_headers: + Accept: application/json + Revision: "2024-10-15" + error_handler: + type: CustomErrorHandler + class_name: >- + source_declarative_manifest.components.KlaviyoErrorHandler + backoff_strategies: + - type: WaitTimeFromHeader + header: Retry-After + response_filters: + - type: HttpResponseFilter + action: RATE_LIMITED + http_codes: + - 429 + - type: HttpResponseFilter + action: FAIL + http_codes: + - 401 + - 403 + failure_type: config_error + error_message: >- + Please provide a valid API key and make sure it has + permissions to read specified streams. + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: + - data + paginator: + type: DefaultPaginator + page_token_option: + type: RequestPath + pagination_strategy: + type: CursorPagination + cursor_value: "{{ response.get('links', {}).get('next') }}" + partition_router: + - type: ListPartitionRouter + values: + - sms + - email + cursor_field: campaign_type + - type: ListPartitionRouter + values: + - "true" + - "false" + cursor_field: archived + incremental_sync: + type: DatetimeBasedCursor + cursor_field: updated_at + cursor_datetime_formats: + - "%Y-%m-%dT%H:%M:%S.%f%z" + - "%Y-%m-%dT%H:%M:%S%z" + - "%Y-%m-%d %H:%M:%S%z" + datetime_format: "%Y-%m-%dT%H:%M:%S%z" + start_datetime: + type: MinMaxDatetime + datetime: "{{ config.get('start_date', '2012-01-01T00:00:00Z') }}" + datetime_format: "%Y-%m-%dT%H:%M:%S%z" + transformations: + - type: AddFields + fields: + - path: + - updated_at + value: "{{ record.get('attributes', {}).get('updated_at') }}" + - type: AddFields + fields: + - path: + - attributes + - channel + value: "{{ stream_partition['campaign_type'] }}" + schema_loader: + type: InlineSchemaLoader + schema: + $ref: "#/schemas/campaigns" + campaigns_detailed: + type: DeclarativeStream + state_migrations: + - type: CustomStateMigration + class_name: >- + source_declarative_manifest.components.CampaignsStateMigration + name: campaigns_detailed + primary_key: + - id + retriever: + type: SimpleRetriever + requester: + $ref: "#/definitions/base_requester" + path: campaigns + http_method: GET + request_parameters: + filter: >- + and(greater-or-equal(updated_at,{{ + stream_interval.start_time }}),less-or-equal(updated_at,{{ stream_interval.end_time + }}),equals(messages.channel,'{{ stream_partition['campaign_type'] + }}'),equals(archived,{{ stream_partition['archived'] }})) + sort: "updated_at" + request_headers: + Accept: application/json + Revision: "2024-10-15" + error_handler: + type: CustomErrorHandler + class_name: >- + source_declarative_manifest.components.KlaviyoErrorHandler + backoff_strategies: + - type: WaitTimeFromHeader + header: Retry-After + response_filters: + - type: HttpResponseFilter + action: RATE_LIMITED + http_codes: + - 429 + - type: HttpResponseFilter + action: FAIL + http_codes: + - 401 + - 403 + failure_type: config_error + error_message: >- + Please provide a valid API key and make sure it has + permissions to read specified streams. + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: + - data + paginator: + type: DefaultPaginator + page_token_option: + type: RequestPath + pagination_strategy: + type: CursorPagination + cursor_value: "{{ response.get('links', {}).get('next') }}" + partition_router: + - type: ListPartitionRouter + values: + - sms + - email + cursor_field: campaign_type + - type: ListPartitionRouter + values: + - "true" + - "false" + cursor_field: archived + incremental_sync: + type: DatetimeBasedCursor + cursor_field: updated_at + cursor_datetime_formats: + - "%Y-%m-%dT%H:%M:%S.%f%z" + - "%Y-%m-%dT%H:%M:%S%z" + - "%Y-%m-%d %H:%M:%S%z" + datetime_format: "%Y-%m-%dT%H:%M:%S%z" + start_datetime: + type: MinMaxDatetime + datetime: "{{ config.get('start_date', '2012-01-01T00:00:00Z') }}" + datetime_format: "%Y-%m-%dT%H:%M:%S%z" + transformations: + - type: AddFields + fields: + - type: AddedFieldDefinition + path: + - updated_at + value: "{{ record.get('attributes', {}).get('updated_at') }}" + - type: CustomTransformation + class_name: >- + source_declarative_manifest.components.CampaignsDetailedTransformation + schema_loader: + type: InlineSchemaLoader + schema: + $ref: "#/schemas/campaigns_detailed" + flows: + type: DeclarativeStream + state_migrations: + - type: CustomStateMigration + class_name: >- + source_declarative_manifest.components.ArchivedToPerPartitionStateMigration + name: flows + primary_key: + - id + retriever: + type: SimpleRetriever + requester: + $ref: "#/definitions/base_requester" + path: flows + http_method: GET + request_parameters: + filter: >- + and(greater-or-equal(updated,{{ + stream_interval.start_time }}),less-or-equal(updated,{{ stream_interval.end_time + }}),equals(archived,{{ stream_partition['archived'] }})) + sort: "updated" + request_headers: + Accept: application/json + Revision: "2024-10-15" + error_handler: + type: CustomErrorHandler + class_name: >- + source_declarative_manifest.components.KlaviyoErrorHandler + backoff_strategies: + - type: WaitTimeFromHeader + header: Retry-After + response_filters: + - type: HttpResponseFilter + action: RATE_LIMITED + http_codes: + - 429 + - type: HttpResponseFilter + action: FAIL + http_codes: + - 401 + - 403 + failure_type: config_error + error_message: >- + Please provide a valid API key and make sure it has + permissions to read specified streams. + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: + - data + paginator: + type: DefaultPaginator + page_token_option: + type: RequestPath + pagination_strategy: + type: CursorPagination + cursor_value: "{{ response.get('links', {}).get('next') }}" + partition_router: + type: ListPartitionRouter + values: + - "true" + - "false" + cursor_field: archived + incremental_sync: + type: DatetimeBasedCursor + cursor_field: updated + cursor_datetime_formats: + - "%Y-%m-%dT%H:%M:%S.%f%z" + - "%Y-%m-%dT%H:%M:%S%z" + - "%Y-%m-%d %H:%M:%S%z" + datetime_format: "%Y-%m-%dT%H:%M:%S%z" + start_datetime: + type: MinMaxDatetime + datetime: "{{ config.get('start_date', '2012-01-01T00:00:00Z') }}" + datetime_format: "%Y-%m-%dT%H:%M:%S%z" + transformations: + - type: AddFields + fields: + - path: + - updated + value: "{{ record.get('attributes', {}).get('updated') }}" + schema_loader: + type: InlineSchemaLoader + schema: + $ref: "#/schemas/flows" + metrics: + type: DeclarativeStream + name: metrics + primary_key: + - id + retriever: + type: SimpleRetriever + requester: + $ref: "#/definitions/base_requester" + path: metrics + http_method: GET + request_headers: + Accept: application/json + Revision: "2024-10-15" + error_handler: + type: CustomErrorHandler + class_name: >- + source_declarative_manifest.components.KlaviyoErrorHandler + backoff_strategies: + - type: WaitTimeFromHeader + header: Retry-After + response_filters: + - type: HttpResponseFilter + action: RATE_LIMITED + http_codes: + - 429 + - type: HttpResponseFilter + action: FAIL + http_codes: + - 401 + - 403 + failure_type: config_error + error_message: >- + Please provide a valid API key and make sure it has + permissions to read specified streams. + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: + - data + paginator: + type: DefaultPaginator + page_token_option: + type: RequestPath + pagination_strategy: + type: CursorPagination + cursor_value: "{{ response.get('links', {}).get('next') }}" + incremental_sync: + type: DatetimeBasedCursor + cursor_field: updated + datetime_format: "%Y-%m-%dT%H:%M:%S%z" + start_datetime: "{{ config.get('start_date', '2012-01-01T00:00:00Z') }}" + is_client_side_incremental: true + transformations: + - type: AddFields + fields: + - path: + - updated + value: "{{ record.get('attributes', {}).get('updated') }}" + schema_loader: + type: InlineSchemaLoader + schema: + $ref: "#/schemas/metrics" + lists: + type: DeclarativeStream + name: lists + primary_key: + - id + retriever: + type: SimpleRetriever + requester: + $ref: "#/definitions/base_requester" + path: lists + http_method: GET + request_headers: + Accept: application/json + Revision: "2024-10-15" + error_handler: + type: CustomErrorHandler + class_name: >- + source_declarative_manifest.components.KlaviyoErrorHandler + backoff_strategies: + - type: WaitTimeFromHeader + header: Retry-After + response_filters: + - type: HttpResponseFilter + action: RATE_LIMITED + http_codes: + - 429 + - type: HttpResponseFilter + action: FAIL + http_codes: + - 401 + - 403 + failure_type: config_error + error_message: >- + Please provide a valid API key and make sure it has + permissions to read specified streams. + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: + - data + paginator: + type: DefaultPaginator + page_token_option: + type: RequestPath + pagination_strategy: + type: CursorPagination + cursor_value: "{{ response.get('links', {}).get('next') }}" + incremental_sync: + type: DatetimeBasedCursor + cursor_field: updated + datetime_format: "%Y-%m-%dT%H:%M:%S%z" + start_datetime: "{{ config.get('start_date', '2012-01-01T00:00:00Z') }}" + is_client_side_incremental: true + transformations: + - type: AddFields + fields: + - path: + - updated + value: "{{ record.get('attributes', {}).get('updated') }}" + schema_loader: + type: InlineSchemaLoader + schema: + $ref: "#/schemas/lists" + lists_detailed: + type: DeclarativeStream + name: lists_detailed + primary_key: + - id + retriever: + type: SimpleRetriever + requester: + $ref: "#/definitions/base_requester" + path: lists/{{ stream_slice.id }} + http_method: GET + request_parameters: + additional-fields[list]: profile_count + request_headers: + Accept: application/json + Revision: "2024-10-15" + error_handler: + type: CustomErrorHandler + class_name: >- + source_declarative_manifest.components.KlaviyoErrorHandler + backoff_strategies: + - type: WaitTimeFromHeader + header: Retry-After + response_filters: + - type: HttpResponseFilter + action: RATE_LIMITED + http_codes: + - 429 + - type: HttpResponseFilter + action: FAIL + http_codes: + - 401 + - 403 + failure_type: config_error + error_message: >- + Please provide a valid API key and make sure it has + permissions to read specified streams. + record_selector: + type: RecordSelector + extractor: + type: DpathExtractor + field_path: + - data + paginator: + type: DefaultPaginator + page_token_option: + type: RequestPath + pagination_strategy: + type: CursorPagination + cursor_value: "{{ response.get('links', {}).get('next') }}" + partition_router: + type: SubstreamPartitionRouter + parent_stream_configs: + - type: ParentStreamConfig + parent_key: id + partition_field: id + stream: + $ref: "#/definitions/streams/lists" + incremental_sync: + type: DatetimeBasedCursor + cursor_field: updated + datetime_format: "%Y-%m-%dT%H:%M:%S%z" + start_datetime: "{{ config.get('start_date', '2012-01-01T00:00:00Z') }}" + is_client_side_incremental: true + transformations: + - type: AddFields + fields: + - path: + - updated + value: "{{ record.get('attributes', {}).get('updated') }}" + schema_loader: + type: InlineSchemaLoader + schema: + $ref: "#/schemas/lists_detailed" + base_requester: + type: HttpRequester + url_base: https://a.klaviyo.com/api/ + authenticator: + type: ApiKeyAuthenticator + api_token: Klaviyo-API-Key {{ config['api_key'] }} + inject_into: + type: RequestOption + field_name: Authorization + inject_into: header + +streams: + # Incremental streams + - $ref: "#/definitions/streams/profiles" + - $ref: "#/definitions/streams/global_exclusions" + - $ref: "#/definitions/streams/events" + - $ref: "#/definitions/streams/events_detailed" + - $ref: "#/definitions/streams/email_templates" + - $ref: "#/definitions/streams/campaigns" + - $ref: "#/definitions/streams/campaigns_detailed" + - $ref: "#/definitions/streams/flows" + # Semi-Incremental streams + - $ref: "#/definitions/streams/metrics" + - $ref: "#/definitions/streams/lists" + - $ref: "#/definitions/streams/lists_detailed" + +spec: + type: Spec + connection_specification: + type: object + $schema: http://json-schema.org/draft-07/schema# + required: + - api_key + properties: + api_key: + type: string + description: >- + Klaviyo API Key. See our docs + if you need help finding this key. + title: Api Key + airbyte_secret: true + order: 0 + start_date: + type: string + description: >- + UTC date and time in the format 2017-01-25T00:00:00Z. Any data before + this date will not be replicated. This field is optional - if not + provided, all data will be replicated. + title: Start Date + pattern: ^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$ + examples: + - "2017-01-25T00:00:00Z" + format: date-time + order: 1 + disable_fetching_predictive_analytics: + type: boolean + description: >- + Certain streams like the profiles stream can retrieve predictive + analytics data from Klaviyo's API. However, at high volume, this can + lead to service availability issues on the API which can be improved + by not fetching this field. WARNING: Enabling this setting will stop + the "predictive_analytics" column from being populated in your + downstream destination. + title: Disable Fetching Predictive Analytics + order: 2 + num_workers: + type: integer + description: >- + The number of worker threads to use for the sync. The performance + upper boundary is based on the limit of your Klaviyo plan. More info + about the rate limit plan tiers can be found on Klaviyo's API docs. + title: Number of concurrent workers + minimum: 1 + maximum: 50 + default: 10 + examples: + - 1 + - 2 + - 3 + order: 3 + additionalProperties: true + +metadata: + autoImportSchema: + profiles: false + global_exclusions: false + events: false + events_detailed: false + email_templates: false + campaigns: false + campaigns_detailed: false + flows: false + metrics: false + lists: false + lists_detailed: false + yamlComponents: + streams: + profiles: + - errorHandler + global_exclusions: + - errorHandler + events: + - errorHandler + events_detailed: + - errorHandler + - recordSelector + email_templates: + - errorHandler + campaigns: + - errorHandler + campaigns_detailed: + - errorHandler + - transformations + flows: + - errorHandler + metrics: + - errorHandler + - incrementalSync + lists: + - errorHandler + - incrementalSync + lists_detailed: + - errorHandler + - incrementalSync + global: + - authenticator + testedStreams: + profiles: + streamHash: 7d27c2aee801ec7d0038722136c6b7e06b14a9ed + hasResponse: true + responsesAreSuccessful: true + hasRecords: true + primaryKeysArePresent: true + primaryKeysAreUnique: true + global_exclusions: + streamHash: 7e7633526c2855390903d6e60973bb13b23272d7 + hasResponse: true + responsesAreSuccessful: true + hasRecords: true + primaryKeysArePresent: true + primaryKeysAreUnique: true + events: + streamHash: af0180236001cbacc0788046bdc916026e1f82f6 + hasResponse: true + responsesAreSuccessful: true + hasRecords: false + primaryKeysArePresent: true + primaryKeysAreUnique: true + email_templates: + streamHash: 4c12cf304ffe3cd0fcaba1479498ad19c18c6f32 + hasResponse: true + responsesAreSuccessful: true + hasRecords: true + primaryKeysArePresent: true + primaryKeysAreUnique: true + metrics: + streamHash: 96e06644c47a223a29c85dc4318ec5f7da1cc414 + hasResponse: true + responsesAreSuccessful: true + hasRecords: true + primaryKeysArePresent: true + primaryKeysAreUnique: true + lists: + streamHash: 9edcccbf069463bf70bdc40db756e0f81eba032b + hasResponse: true + responsesAreSuccessful: true + hasRecords: true + primaryKeysArePresent: true + primaryKeysAreUnique: true + lists_detailed: + streamHash: 34e4a9f1fb0c879b915d8558d67feb887d76f8e5 + hasResponse: true + responsesAreSuccessful: false + hasRecords: true + primaryKeysArePresent: true + primaryKeysAreUnique: true + assist: {} + +schemas: + profiles: + type: object + $schema: http://json-schema.org/draft-07/schema# + additionalProperties: true + properties: + type: + type: + - "null" + - string + attributes: + type: + - "null" + - object + additionalProperties: true + properties: + anonymous_id: + type: + - "null" + - string + created: + type: + - "null" + - string + format: date-time + email: + type: + - "null" + - string + external_id: + type: + - "null" + - string + first_name: + type: + - "null" + - string + image: + type: + - "null" + - string + last_event_date: + type: + - "null" + - string + format: date-time + last_name: + type: + - "null" + - string + locale: + type: + - "null" + - string + location: + type: + - "null" + - object + properties: + address1: + type: + - "null" + - string + address2: + type: + - "null" + - string + city: + type: + - "null" + - string + country: + type: + - "null" + - string + ip: + type: + - "null" + - string + latitude: + oneOf: + - type: "null" + - type: number + - type: string + longitude: + oneOf: + - type: "null" + - type: number + - type: string + region: + type: + - "null" + - string + timezone: + type: + - "null" + - string + zip: + type: + - "null" + - string + organization: + type: + - "null" + - string + phone_number: + type: + - "null" + - string + predictive_analytics: + type: + - "null" + - object + properties: + average_days_between_orders: + type: + - "null" + - number + average_order_value: + type: + - "null" + - number + churn_probability: + type: + - "null" + - number + expected_date_of_next_order: + type: + - "null" + - string + historic_clv: + type: + - "null" + - number + historic_number_of_orders: + type: + - "null" + - integer + predicted_clv: + type: + - "null" + - number + predicted_number_of_orders: + type: + - "null" + - number + total_clv: + type: + - "null" + - number + properties: + type: + - "null" + - object + additionalProperties: true + subscriptions: + type: + - "null" + - object + properties: + email: + type: + - "null" + - object + properties: + marketing: + type: + - "null" + - object + properties: + can_receive_email_marketing: + type: boolean + consent: + type: string + consent_timestamp: + type: + - "null" + - string + format: date-time + custom_method_detail: + type: + - "null" + - string + double_optin: + type: + - "null" + - boolean + last_updated: + type: + - "null" + - string + format: date-time + list_suppressions: + type: + - "null" + - array + items: + type: + - "null" + - object + properties: + list_id: + type: string + reason: + type: string + timestamp: + type: string + format: date-time + method: + type: + - "null" + - string + method_detail: + type: + - "null" + - string + suppressions: + type: + - "null" + - array + items: + type: + - "null" + - object + properties: + reason: + type: string + timestamp: + type: string + format: date-time + timestamp: + type: + - "null" + - string + format: date-time + mobile_push: + type: + - "null" + - object + properties: + marketing: + type: + - "null" + - object + properties: + can_receive_sms_marketing: + type: + - "null" + - boolean + consent: + type: + - "null" + - string + consent_timestamp: + type: + - "null" + - string + format: date-time + sms: + type: + - "null" + - object + properties: + marketing: + type: + - "null" + - object + properties: + can_receive_sms_marketing: + type: + - "null" + - boolean + consent: + type: + - "null" + - string + consent_timestamp: + type: + - "null" + - string + format: date-time + last_updated: + type: + - "null" + - string + format: date-time + method: + type: + - "null" + - string + method_detail: + type: + - "null" + - string + timestamp: + type: + - "null" + - string + format: date-time + transactional: + type: + - "null" + - object + properties: + can_receive_sms_marketing: + type: + - "null" + - boolean + consent: + type: + - "null" + - string + consent_timestamp: + type: + - "null" + - string + format: date-time + last_updated: + type: + - "null" + - string + format: date-time + method: + type: + - "null" + - string + method_detail: + type: + - "null" + - string + timestamp: + type: + - "null" + - string + format: date-time + title: + type: + - "null" + - string + updated: + type: + - "null" + - string + format: date-time + id: + type: string + links: + type: + - "null" + - object + properties: + self: + type: + - "null" + - string + relationships: + type: + - "null" + - object + properties: + lists: + type: + - "null" + - object + properties: + links: + type: + - "null" + - object + properties: + related: + type: + - "null" + - string + self: + type: + - "null" + - string + segments: + type: + - "null" + - object + properties: + links: + type: + - "null" + - object + properties: + related: + type: + - "null" + - string + self: + type: + - "null" + - string + segments: + type: + - "null" + - object + updated: + type: + - "null" + - string + format: date-time + global_exclusions: + type: object + $schema: http://json-schema.org/draft-07/schema# + additionalProperties: true + properties: + type: + type: + - "null" + - string + attributes: + type: + - "null" + - object + additionalProperties: true + properties: + anonymous_id: + type: + - "null" + - string + created: + type: + - "null" + - string + format: date-time + email: + type: + - "null" + - string + external_id: + type: + - "null" + - string + first_name: + type: + - "null" + - string + image: + type: + - "null" + - string + last_event_date: + type: + - "null" + - string + format: date-time + last_name: + type: + - "null" + - string + locale: + type: + - "null" + - string + location: + type: + - "null" + - object + properties: + address1: + type: + - "null" + - string + address2: + type: + - "null" + - string + city: + type: + - "null" + - string + country: + type: + - "null" + - string + ip: + type: + - "null" + - string + latitude: + oneOf: + - type: "null" + - type: number + - type: string + longitude: + oneOf: + - type: "null" + - type: number + - type: string + region: + type: + - "null" + - string + timezone: + type: + - "null" + - string + zip: + type: + - "null" + - string + organization: + type: + - "null" + - string + phone_number: + type: + - "null" + - string + predictive_analytics: + type: + - "null" + - object + properties: + average_days_between_orders: + type: + - "null" + - number + average_order_value: + type: + - "null" + - number + churn_probability: + type: + - "null" + - number + expected_date_of_next_order: + type: + - "null" + - string + historic_clv: + type: + - "null" + - number + historic_number_of_orders: + type: + - "null" + - integer + predicted_clv: + type: + - "null" + - number + predicted_number_of_orders: + type: + - "null" + - number + total_clv: + type: + - "null" + - number + properties: + type: + - "null" + - object + additionalProperties: true + subscriptions: + type: + - "null" + - object + properties: + email: + type: + - "null" + - object + properties: + marketing: + type: + - "null" + - object + properties: + can_receive_email_marketing: + type: boolean + consent: + type: string + consent_timestamp: + type: + - "null" + - string + format: date-time + custom_method_detail: + type: + - "null" + - string + double_optin: + type: + - "null" + - boolean + last_updated: + type: + - "null" + - string + format: date-time + list_suppressions: + type: + - "null" + - array + items: + type: + - "null" + - object + properties: + list_id: + type: string + reason: + type: string + timestamp: + type: string + format: date-time + method: + type: + - "null" + - string + method_detail: + type: + - "null" + - string + suppressions: + type: + - "null" + - array + items: + type: + - "null" + - object + properties: + reason: + type: string + timestamp: + type: string + format: date-time + timestamp: + type: + - "null" + - string + format: date-time + mobile_push: + type: + - "null" + - object + properties: + marketing: + type: + - "null" + - object + properties: + can_receive_sms_marketing: + type: + - "null" + - boolean + consent: + type: + - "null" + - string + consent_timestamp: + type: + - "null" + - string + format: date-time + sms: + type: + - "null" + - object + properties: + marketing: + type: + - "null" + - object + properties: + can_receive_sms_marketing: + type: + - "null" + - boolean + consent: + type: + - "null" + - string + consent_timestamp: + type: + - "null" + - string + format: date-time + last_updated: + type: + - "null" + - string + format: date-time + method: + type: + - "null" + - string + method_detail: + type: + - "null" + - string + timestamp: + type: + - "null" + - string + format: date-time + transactional: + type: + - "null" + - object + properties: + can_receive_sms_marketing: + type: + - "null" + - boolean + consent: + type: + - "null" + - string + consent_timestamp: + type: + - "null" + - string + format: date-time + last_updated: + type: + - "null" + - string + format: date-time + method: + type: + - "null" + - string + method_detail: + type: + - "null" + - string + timestamp: + type: + - "null" + - string + format: date-time + title: + type: + - "null" + - string + updated: + type: + - "null" + - string + format: date-time + id: + type: string + links: + type: + - "null" + - object + properties: + self: + type: + - "null" + - string + relationships: + type: + - "null" + - object + properties: + lists: + type: + - "null" + - object + properties: + links: + type: + - "null" + - object + properties: + related: + type: + - "null" + - string + self: + type: + - "null" + - string + segments: + type: + - "null" + - object + properties: + links: + type: + - "null" + - object + properties: + related: + type: + - "null" + - string + self: + type: + - "null" + - string + segments: + type: + - "null" + - object + updated: + type: + - "null" + - string + format: date-time + events: + type: object + $schema: http://json-schema.org/draft-07/schema# + additionalProperties: true + properties: + type: + type: string + attributes: + type: + - "null" + - object + properties: + datetime: + type: string + format: date-time + event_properties: + type: + - "null" + - object + additionalProperties: true + timestamp: + type: integer + uuid: + type: string + datetime: + type: string + format: date-time + id: + type: string + links: + type: + - "null" + - object + properties: + self: + type: string + relationships: + type: + - "null" + - object + properties: + attributions: + type: + - "null" + - object + properties: + data: + type: + - "null" + - array + items: + type: + - "null" + - object + properties: + type: + type: string + id: + type: string + links: + type: + - "null" + - object + additionalProperties: true + properties: + related: + type: string + self: + type: string + metric: + type: + - "null" + - object + properties: + data: + type: + - "null" + - object + properties: + type: + type: string + id: + type: string + links: + type: + - "null" + - object + additionalProperties: true + properties: + related: + type: string + self: + type: string + profile: + type: + - "null" + - object + properties: + data: + type: + - "null" + - object + properties: + type: + type: string + id: + type: string + links: + type: + - "null" + - object + additionalProperties: true + properties: + related: + type: string + self: + type: string + events_detailed: + type: object + $schema: http://json-schema.org/draft-07/schema# + additionalProperties: true + properties: + type: + type: string + attributes: + type: + - "null" + - object + properties: + datetime: + type: string + format: date-time + event_properties: + type: + - "null" + - object + additionalProperties: true + timestamp: + type: integer + uuid: + type: string + datetime: + type: string + format: date-time + id: + type: string + links: + type: + - "null" + - object + properties: + self: + type: string + relationships: + type: + - "null" + - object + properties: + metric: + type: + - "null" + - object + properties: + data: + type: + - "null" + - object + properties: + type: + type: string + id: + type: string + name: + type: string + links: + type: + - "null" + - object + additionalProperties: true + properties: + related: + type: string + self: + type: string + profile: + type: + - "null" + - object + properties: + data: + type: + - "null" + - object + properties: + type: + type: string + id: + type: string + links: + type: + - "null" + - object + additionalProperties: true + properties: + related: + type: string + self: + type: string + email_templates: + type: object + $schema: http://json-schema.org/draft-07/schema# + additionalProperties: true + properties: + type: + type: string + attributes: + type: + - "null" + - object + additionalProperties: true + properties: + company_id: + type: + - "null" + - string + created: + type: + - "null" + - string + format: date-time + editor_type: + type: + - "null" + - string + html: + type: string + name: + type: string + text: + type: + - "null" + - string + updated: + type: + - "null" + - string + format: date-time + id: + type: string + links: + type: + - "null" + - object + additionalProperties: true + properties: + self: + type: string + updated: + type: + - "null" + - string + format: date-time + campaigns: + type: object + $schema: http://json-schema.org/draft-07/schema# + additionalProperties: true + properties: + type: + type: string + attributes: + type: + - "null" + - object + additionalProperties: true + properties: + archived: + type: boolean + audiences: + type: + - "null" + - object + additionalProperties: true + properties: + excluded: + type: + - "null" + - array + items: + type: + - "null" + - string + included: + type: + - "null" + - array + items: + type: + - "null" + - string + channel: + type: string + created_at: + type: + - "null" + - string + format: date-time + message: + type: string + name: + type: string + scheduled_at: + type: + - "null" + - string + format: date-time + send_options: + type: + - "null" + - object + properties: + ignore_unsubscribes: + type: + - "null" + - boolean + use_smart_sending: + type: + - "null" + - boolean + send_strategy: + type: + - "null" + - object + additionalProperties: true + properties: + method: + type: string + options_static: + type: + - "null" + - object + properties: + datetime: + type: string + airbyte_type: timestamp_without_timezone + format: date-time + is_local: + type: + - "null" + - boolean + send_past_recipients_immediately: + type: + - "null" + - boolean + options_sto: + type: + - "null" + - object + properties: + date: + type: string + format: date + options_throttled: + type: + - "null" + - object + properties: + datetime: + type: string + airbyte_type: timestamp_without_timezone + format: date-time + throttle_percentage: + type: integer + send_time: + type: + - "null" + - string + format: date-time + status: + type: string + tracking_options: + type: + - "null" + - object + additionalProperties: true + properties: + add_tracking_params: + type: + - "null" + - boolean + is_add_utm: + type: + - "null" + - boolean + is_tracking_clicks: + type: + - "null" + - boolean + is_tracking_opens: + type: + - "null" + - boolean + utm_params: + type: + - "null" + - array + items: + type: + - "null" + - object + properties: + name: + type: string + value: + type: string + updated_at: + type: + - "null" + - string + format: date-time + id: + type: string + links: + type: + - "null" + - object + additionalProperties: true + properties: + self: + type: string + relationships: + type: + - "null" + - object + additionalProperties: true + properties: + campaign-messages: + type: + - "null" + - object + properties: + data: + type: array + items: + type: + - "null" + - object + properties: + type: + type: string + id: + type: string + links: + type: + - "null" + - object + properties: + related: + type: string + self: + type: string + tags: + type: + - "null" + - object + properties: + data: + type: array + items: + type: + - "null" + - object + properties: + type: + type: string + id: + type: string + links: + type: + - "null" + - object + properties: + related: + type: string + self: + type: string + updated_at: + type: + - "null" + - string + format: date-time + campaigns_detailed: + type: object + $schema: http://json-schema.org/draft-07/schema# + additionalProperties: true + properties: + type: + type: string + attributes: + type: + - "null" + - object + additionalProperties: true + properties: + archived: + type: boolean + audiences: + type: + - "null" + - object + additionalProperties: true + properties: + excluded: + type: + - "null" + - array + items: + type: + - "null" + - string + included: + type: + - "null" + - array + items: + type: + - "null" + - string + channel: + type: string + created_at: + type: + - "null" + - string + format: date-time + message: + type: string + name: + type: string + scheduled_at: + type: + - "null" + - string + format: date-time + send_options: + type: + - "null" + - object + properties: + ignore_unsubscribes: + type: + - "null" + - boolean + use_smart_sending: + type: + - "null" + - boolean + send_strategy: + type: + - "null" + - object + additionalProperties: true + properties: + method: + type: string + options_static: + type: + - "null" + - object + properties: + datetime: + type: string + airbyte_type: timestamp_without_timezone + format: date-time + is_local: + type: + - "null" + - boolean + send_past_recipients_immediately: + type: + - "null" + - boolean + options_sto: + type: + - "null" + - object + properties: + date: + type: string + format: date + options_throttled: + type: + - "null" + - object + properties: + datetime: + type: string + airbyte_type: timestamp_without_timezone + format: date-time + throttle_percentage: + type: integer + send_time: + type: + - "null" + - string + format: date-time + status: + type: string + tracking_options: + type: + - "null" + - object + additionalProperties: true + properties: + add_tracking_params: + type: + - "null" + - boolean + is_add_utm: + type: + - "null" + - boolean + is_tracking_clicks: + type: + - "null" + - boolean + is_tracking_opens: + type: + - "null" + - boolean + utm_params: + type: + - "null" + - array + items: + type: + - "null" + - object + properties: + name: + type: string + value: + type: string + updated_at: + type: + - "null" + - string + format: date-time + campaign_messages: + type: + - "null" + - array + items: + type: + - "null" + - object + properties: + type: + type: + - "null" + - string + attributes: + type: + - "null" + - object + properties: + channel: + type: + - "null" + - string + content: + type: + - "null" + - object + properties: + bcc_email: + type: + - "null" + - string + cc_email: + type: + - "null" + - string + from_email: + type: + - "null" + - string + from_label: + type: + - "null" + - string + preview_text: + type: + - "null" + - string + reply_to_email: + type: + - "null" + - string + subject: + type: + - "null" + - string + created_at: + type: + - "null" + - string + airbyte_type: timestamp_without_timezone + format: date-time + label: + type: + - "null" + - string + render_options: + type: + - "null" + - object + properties: + add_info_link: + type: + - "null" + - boolean + add_opt_out_language: + type: + - "null" + - boolean + add_org_prefix: + type: + - "null" + - boolean + shorten_links: + type: + - "null" + - boolean + send_times: + type: + - "null" + - array + items: + type: + - "null" + - object + properties: + datetime: + type: + - "null" + - string + format: date-time + is_local: + type: + - "null" + - boolean + updated_at: + type: + - "null" + - string + airbyte_type: timestamp_without_timezone + format: date-time + id: + type: + - "null" + - string + links: + type: + - "null" + - object + properties: + self: + type: + - "null" + - string + relationships: + type: + - "null" + - object + properties: + campaign: + type: + - "null" + - object + properties: + data: + type: + - "null" + - object + properties: + type: + type: + - "null" + - string + id: + type: + - "null" + - string + links: + type: + - "null" + - object + properties: + related: + type: + - "null" + - string + self: + type: + - "null" + - string + template: + type: + - "null" + - object + properties: + data: + type: + - "null" + - object + properties: + type: + type: + - "null" + - string + id: + type: + - "null" + - string + links: + type: + - "null" + - object + properties: + related: + type: + - "null" + - string + self: + type: + - "null" + - string + estimated_recipient_count: + type: + - "null" + - integer + id: + type: string + links: + type: + - "null" + - object + additionalProperties: true + properties: + self: + type: string + relationships: + type: + - "null" + - object + additionalProperties: true + properties: + campaign-messages: + type: + - "null" + - object + properties: + data: + type: array + items: + type: + - "null" + - object + properties: + type: + type: string + id: + type: string + links: + type: + - "null" + - object + properties: + related: + type: string + self: + type: string + tags: + type: + - "null" + - object + properties: + data: + type: array + items: + type: + - "null" + - object + properties: + type: + type: string + id: + type: string + links: + type: + - "null" + - object + properties: + related: + type: string + self: + type: string + updated_at: + type: + - "null" + - string + format: date-time + flows: + type: object + $schema: http://json-schema.org/draft-07/schema# + additionalProperties: true + properties: + type: + type: string + attributes: + type: + - "null" + - object + additionalProperties: true + properties: + archived: + type: boolean + created: + type: string + format: date-time + name: + type: string + status: + type: string + trigger_type: + type: string + updated: + type: string + format: date-time + id: + type: string + links: + type: + - "null" + - object + additionalProperties: true + properties: + self: + type: string + relationships: + type: + - "null" + - object + additionalProperties: true + properties: + flow-actions: + type: + - "null" + - object + properties: + data: + type: array + items: + type: + - "null" + - object + additionalProperties: true + properties: + type: + type: string + id: + type: string + links: + type: + - "null" + - object + properties: + related: + type: string + self: + type: string + tags: + type: + - "null" + - object + properties: + data: + type: array + items: + type: + - "null" + - object + properties: + type: + type: string + id: + type: string + links: + type: + - "null" + - object + properties: + related: + type: string + self: + type: string + updated: + type: string + format: date-time + metrics: + type: object + $schema: http://json-schema.org/draft-07/schema# + additionalProperties: true + properties: + type: + type: string + attributes: + type: + - "null" + - object + properties: + created: + type: string + format: date-time + integration: + type: + - "null" + - object + additionalProperties: true + name: + type: string + updated: + type: string + format: date-time + id: + type: string + links: + type: + - "null" + - object + additionalProperties: true + properties: + self: + type: string + relationships: + type: + - "null" + - object + properties: + flow-triggers: + type: + - "null" + - object + properties: + data: + type: + - "null" + - object + properties: + type: + type: + - "null" + - string + id: + type: + - "null" + - string + links: + type: + - "null" + - object + properties: + related: + type: + - "null" + - string + self: + type: + - "null" + - string + updated: + type: string + format: date-time + lists: + type: object + $schema: http://json-schema.org/draft-07/schema# + additionalProperties: true + properties: + type: + type: string + attributes: + type: + - "null" + - object + properties: + created: + type: + - "null" + - string + format: date-time + name: + type: string + opt_in_process: + type: + - "null" + - string + updated: + type: + - "null" + - string + format: date-time + id: + type: string + links: + type: + - "null" + - object + additionalProperties: true + properties: + self: + type: string + relationships: + type: + - "null" + - object + additionalProperties: true + properties: + flow-triggers: + type: + - "null" + - object + properties: + data: + type: + - "null" + - object + properties: + type: + type: + - "null" + - string + id: + type: + - "null" + - string + links: + type: + - "null" + - object + properties: + related: + type: + - "null" + - string + self: + type: + - "null" + - string + profiles: + type: + - "null" + - object + properties: + links: + type: + - "null" + - object + properties: + related: + type: string + self: + type: string + tags: + type: + - "null" + - object + properties: + data: + type: array + items: + type: + - "null" + - object + properties: + type: + type: string + id: + type: string + links: + type: + - "null" + - object + properties: + related: + type: string + self: + type: string + updated: + type: + - "null" + - string + format: date-time + lists_detailed: + type: object + $schema: http://json-schema.org/draft-07/schema# + additionalProperties: true + properties: + type: + type: string + attributes: + type: + - "null" + - object + properties: + created: + type: + - "null" + - string + format: date-time + name: + type: string + opt_in_process: + type: + - "null" + - string + profile_count: + type: + - "null" + - integer + updated: + type: + - "null" + - string + format: date-time + id: + type: string + links: + type: + - "null" + - object + additionalProperties: true + properties: + self: + type: string + relationships: + type: + - "null" + - object + additionalProperties: true + properties: + flow-triggers: + type: + - "null" + - object + properties: + data: + type: + - "null" + - object + properties: + type: + type: + - "null" + - string + id: + type: + - "null" + - string + links: + type: + - "null" + - object + properties: + related: + type: + - "null" + - string + self: + type: + - "null" + - string + profiles: + type: + - "null" + - object + properties: + links: + type: + - "null" + - object + properties: + related: + type: string + self: + type: string + tags: + type: + - "null" + - object + properties: + data: + type: array + items: + type: + - "null" + - object + properties: + type: + type: string + id: + type: string + links: + type: + - "null" + - object + properties: + related: + type: string + self: + type: string + updated: + type: + - "null" + - string + format: date-time + +api_budget: + type: HTTPAPIBudget + # Each policy here uses a MovingWindowCallRatePolicy with two rates: + # one for burst (per-second) and one for steady (per-minute). + policies: + # Campaigns and Campaigns Detailed + - type: MovingWindowCallRatePolicy + rates: + - limit: 10 # burst: 10 calls per second + interval: PT1S + - limit: 150 # steady: 150 calls per minute + interval: PT1M + matchers: + - method: GET + url_path_pattern: "^/api/campaigns($|/)" # matches '/campaigns' + # Flows + - type: MovingWindowCallRatePolicy + rates: + - limit: 3 # burst: 3 calls per second + interval: PT1S + - limit: 60 # steady: 60 calls per minute + interval: PT1M + matchers: + - method: GET + url_path_pattern: "^/api/flows($|/)" # matches '/flows' + # Profiles (and global_exclusions share the same endpoint) + - type: MovingWindowCallRatePolicy + rates: + - limit: 10 # burst: 10 calls per second + interval: PT1S + - limit: 150 # steady: 150 calls per minute + interval: PT1M + matchers: + - method: GET + url_path_pattern: "^/api/profiles($|/)" # matches '/profiles' (exact or with trailing slash/extra) + # Events (and events_detailed share the same endpoint) + - type: MovingWindowCallRatePolicy + rates: + - limit: 350 # burst: 350 calls per second + interval: PT1S + - limit: 3500 # steady: 3500 calls per minute + interval: PT1M + matchers: + - method: GET + url_path_pattern: "^/api/events($|/)" # matches '/events' and '/events_detailed' if using same endpoint + # Email Templates + - type: MovingWindowCallRatePolicy + rates: + - limit: 10 # burst: 10 calls per second + interval: PT1S + - limit: 150 # steady: 150 calls per minute + interval: PT1M + matchers: + - method: GET + url_path_pattern: "^/api/templates($|/)" # matches '/templates' + # Metrics + - type: MovingWindowCallRatePolicy + rates: + - limit: 10 # burst: 10 calls per second + interval: PT1S + - limit: 150 # steady: 150 calls per minute + interval: PT1M + matchers: + - method: GET + url_path_pattern: "^/api/metrics($|/)" + # Lists (the parent endpoint for lists streams) + - type: MovingWindowCallRatePolicy + rates: + - limit: 75 # burst: 75 calls per second + interval: PT1S + - limit: 700 # steady: 700 calls per minute + interval: PT1M + matchers: + - method: GET + url_path_pattern: "^/api/lists$" # exactly '/lists' + # Lists Detailed (uses a different URL path – note the extra segment) + - type: MovingWindowCallRatePolicy + rates: + - limit: 1 # burst: 1 call per second + interval: PT1S + - limit: 15 # steady: 15 calls per minute + interval: PT1M + matchers: + - method: GET + url_path_pattern: "^/api/lists/" # matches any path beginning with '/lists/' (e.g. '/lists/123') + params: + "additional-fields[list]": "profile_count" # Other API budget settings: + status_codes_for_ratelimit_hit: [429] + +# Klaviyo's rate limiting is different by endpoints: +# - XS: 1/s burst; 15/m steady +# - S: 3/s burst; 60/m steady +# - M: 10/s burst; 150/m steady +# - L: 75/s burst; 700/m steady +# - XL: 350/s burst; 3500/m steady +# - `lists_detailed`: 1/s burst, 15/m steady (Rate limits when using the additional-fields[list]=profile_count parameter in your API request) + +# As of 2024-11-11, we have the following streams: +# | Stream | Endpoint | Klaviyo Rate Limit Size | Source Concurrency Between Streams | Source Concurrency Within Stream | Source Max Number of Threads Sharing Rate Limits | | +#|-------------------|----------------------------------------------------------------------|-------------------------|------------------------------------|---------------------------------------------------|------------------------------------------------------------------|---------------------------------------------------------------------------------| +#| profiles | https://developers.klaviyo.com/en/v2023-02-22/reference/get_profiles | M | Yes, shared with global_exclusions | No as `step` is not defined in `incremental_sync` | 2 | With other streams (global_exclusions), not within stream as `step` not defined | +#| global_exclusions | https://developers.klaviyo.com/en/v2023-02-22/reference/get_profiles | M | Yes, shared with profiles | No as `step` is not defined in `incremental_sync` | 2 | With other streams (profiles), not within stream as `step` not defined | +#| events | https://developers.klaviyo.com/en/reference/get_events | XL | Yes, shared with events_detailed | Yes | number of steps for events + number of steps for events_detailed | With other streams (events_detailed) and within stream as sliced on `datetime` | +#| events_detailed | https://developers.klaviyo.com/en/reference/get_events | XL | Yes, shared with events | Yes | number of steps for events + number of steps for events_detailed | With other streams (events) and within stream as sliced on `datetime` | +#| email_templates | https://developers.klaviyo.com/en/reference/get_templates | M | None | No as `step` is not defined in `incremental_sync` | 1 | None | +#| metrics | https://developers.klaviyo.com/en/reference/get_metrics | M | None | No as `step` is not defined in `incremental_sync` | 1 | None | +#| lists | https://developers.klaviyo.com/en/reference/get_lists | L | Yes, shared with lists_detailed | No as `step` is not defined in `incremental_sync` | 2 | With other streams (lists_detailed), not within stream as `step` not defined | +#| lists_detailed | https://developers.klaviyo.com/en/reference/get_lists | 1/s | Yes, shared with lists | No as `step` is not defined in `incremental_sync` | 1 | With other streams (lists), not within stream as `step` not defined | +# Note: As of 2024-11-11, `metrics`, `lists` and `lists_detailed` are not supported by the Concurrent CDK as they do client side-filtering. + +# Based on the above, the only threads that allow for slicing and hence might perform more concurrent HTTP requests are `events` and `events_detailed`. There are no slicing for the others and hence the concurrency is limited by the number of streams querying the same endpoint. Given that the event endpoint is XL, we will set a default concurrency to 10. + +concurrency_level: + type: ConcurrencyLevel + default_concurrency: "{{ config.get('num_workers', 25) }}" + max_concurrency: 50 diff --git a/airbyte-integrations/connectors/source-klaviyo/metadata.yaml b/airbyte-integrations/connectors/source-klaviyo/metadata.yaml index 4488a8022ff73..8c3dfa0e5dc3b 100644 --- a/airbyte-integrations/connectors/source-klaviyo/metadata.yaml +++ b/airbyte-integrations/connectors/source-klaviyo/metadata.yaml @@ -7,8 +7,8 @@ data: connectorType: source definitionId: 95e8cffd-b8c4-4039-968e-d32fb4a69bde connectorBuildOptions: - baseImage: docker.io/airbyte/python-connector-base:4.0.0@sha256:d9894b6895923b379f3006fa251147806919c62b7d9021b5cd125bb67d7bbe22 - dockerImageTag: 2.13.0 + baseImage: docker.io/airbyte/source-declarative-manifest:6.33.7@sha256:d30897ff117abcd185b369400fe1f4074b1043e2352cb1c8b03033d56c90742f + dockerImageTag: 2.14.0 dockerRepository: airbyte/source-klaviyo githubIssueLabel: source-klaviyo icon: klaviyo.svg @@ -17,7 +17,7 @@ data: name: Klaviyo remoteRegistries: pypi: - enabled: true + enabled: false packageName: airbyte-source-klaviyo registryOverrides: cloud: @@ -42,8 +42,8 @@ data: upgradeDeadline: "2023-11-30" documentationUrl: https://docs.airbyte.com/integrations/sources/klaviyo tags: - - language:python - cdk:low-code + - language:manifest-only ab_internal: sl: 300 ql: 400 diff --git a/airbyte-integrations/connectors/source-klaviyo/pyproject.toml b/airbyte-integrations/connectors/source-klaviyo/pyproject.toml deleted file mode 100644 index 058a1f463970f..0000000000000 --- a/airbyte-integrations/connectors/source-klaviyo/pyproject.toml +++ /dev/null @@ -1,37 +0,0 @@ -[build-system] -requires = [ "poetry-core>=1.0.0",] -build-backend = "poetry.core.masonry.api" - -[tool.poetry] -version = "2.13.0" -name = "source-klaviyo" -description = "Source implementation for Klaviyo." -authors = [ "Airbyte ",] -license = "MIT" -readme = "README.md" -documentation = "https://docs.airbyte.com/integrations/sources/klaviyo" -homepage = "https://airbyte.com" -repository = "https://github.com/airbytehq/airbyte" -[[tool.poetry.packages]] -include = "source_klaviyo" - -[tool.poetry.dependencies] -python = "^3.10,<3.12" -airbyte_cdk = "^6" -pendulum = "<3.0.0" - -[tool.poetry.scripts] -source-klaviyo = "source_klaviyo.run:run" - -[tool.poetry.group.dev.dependencies] -pytest = "^6.1" -pytest-mock = "^3.12.0" -requests-mock = "^1.9.3" -freezegun = "*" - -[tool.poe] -include = [ - # Shared tasks definition file(s) can be imported here. - # Run `poe` or `poe --help` to see the list of available tasks. - "${POE_GIT_DIR}/poe-tasks/poetry-connector-tasks.toml", -] diff --git a/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/__init__.py b/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/__init__.py deleted file mode 100644 index 53739124631fe..0000000000000 --- a/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/__init__.py +++ /dev/null @@ -1,7 +0,0 @@ -# -# Copyright (c) 2023 Airbyte, Inc., all rights reserved. -# - -from .source import SourceKlaviyo - -__all__ = ["SourceKlaviyo"] diff --git a/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/components/archived_to_per_partition_state_migration.py b/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/components/archived_to_per_partition_state_migration.py deleted file mode 100644 index b40322388b07c..0000000000000 --- a/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/components/archived_to_per_partition_state_migration.py +++ /dev/null @@ -1,117 +0,0 @@ -# -# Copyright (c) 2024 Airbyte, Inc., all rights reserved. -# -from abc import ABC -from typing import Any, Mapping - -from airbyte_cdk.sources.declarative.declarative_stream import DeclarativeStream -from airbyte_cdk.sources.declarative.interpolation.interpolated_string import InterpolatedString -from airbyte_cdk.sources.declarative.migrations.state_migration import StateMigration -from airbyte_cdk.sources.types import Config - - -ARCHIVED_EMAIL = {"archived": "true", "campaign_type": "email"} -NOT_ARCHIVED_EMAIL = {"archived": "false", "campaign_type": "email"} - -ARCHIVED = {"archived": "true"} -NOT_ARCHIVED = {"archived": "false"} - -DEFAULT_START_DATE = "2012-01-01T00:00:00Z" - - -class ArchivedToPerPartitionStateMigration(StateMigration, ABC): - """ - Updates old format state to new per partitioned format. - Partitions: [{archived: True}, {archived: False}] - Default built in airbyte cdk migration will recognise only top-level field cursor value(updated_at), - but for partition {archived: True} source should use cursor value from archived object. - - Example input state: - { - "updated_at": "2020-10-10T00:00:00+00:00", - "archived": { - "updated_at": "2021-10-10T00:00:00+00:00" - } - } - - Example output state: - { - "partition":{ "archived":"true" }, - "cursor":{ "updated_at":"2021-10-10T00:00:00+00:00" } - } - { - "partition":{ "archived":"false" }, - "cursor":{ "updated_at":"2020-10-10T00:00:00+00:00" } - } - """ - - declarative_stream: DeclarativeStream - config: Config - - def __init__(self, declarative_stream: DeclarativeStream, config: Config): - self._config = config - self.declarative_stream = declarative_stream - self._cursor = declarative_stream.incremental_sync - self._parameters = declarative_stream.parameters - self._cursor_field = InterpolatedString.create(self._cursor.cursor_field, parameters=self._parameters).eval(self._config) - - def get_archived_cursor_value(self, stream_state: Mapping[str, Any]): - return stream_state.get("archived", {}).get(self._cursor.cursor_field, self._config.get("start_date", DEFAULT_START_DATE)) - - def get_not_archived_cursor_value(self, stream_state: Mapping[str, Any]): - return stream_state.get(self._cursor.cursor_field, self._config.get("start_date", DEFAULT_START_DATE)) - - def should_migrate(self, stream_state: Mapping[str, Any]) -> bool: - return bool("states" not in stream_state and stream_state) - - def migrate(self, stream_state: Mapping[str, Any]) -> Mapping[str, Any]: - if not self.should_migrate(stream_state): - return stream_state - is_archived_updated_at = self.get_archived_cursor_value(stream_state) - is_not_archived_updated_at = self.get_not_archived_cursor_value(stream_state) - - migrated_stream_state = { - "states": [ - {"partition": ARCHIVED, "cursor": {self._cursor.cursor_field: is_archived_updated_at}}, - {"partition": NOT_ARCHIVED, "cursor": {self._cursor.cursor_field: is_not_archived_updated_at}}, - ] - } - return migrated_stream_state - - -class CampaignsStateMigration(ArchivedToPerPartitionStateMigration): - """ - Campaigns stream has 2 partition field: archived and campaign_type(email, sms). - Previous API version didn't return sms in campaigns output so we need to migrate only email partition. - - Example input state: - { - "updated_at": "2020-10-10T00:00:00+00:00", - "archived": { - "updated_at": "2021-10-10T00:00:00+00:00" - } - } - Example output state: - { - "partition":{ "archived":"true","campaign_type":"email" }, - "cursor":{ "updated_at":"2021-10-10T00:00:00+00:00" } - } - { - "partition":{ "archived":"false","campaign_type":"email" }, - "cursor":{ "updated_at":"2020-10-10T00:00:00+00:00" } - } - """ - - def migrate(self, stream_state: Mapping[str, Any]) -> Mapping[str, Any]: - if not self.should_migrate(stream_state): - return stream_state - is_archived_updated_at = self.get_archived_cursor_value(stream_state) - is_not_archived_updated_at = self.get_not_archived_cursor_value(stream_state) - - migrated_stream_state = { - "states": [ - {"partition": ARCHIVED_EMAIL, "cursor": {self._cursor.cursor_field: is_archived_updated_at}}, - {"partition": NOT_ARCHIVED_EMAIL, "cursor": {self._cursor.cursor_field: is_not_archived_updated_at}}, - ] - } - return migrated_stream_state diff --git a/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/components/campaign_details_transformations.py b/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/components/campaign_details_transformations.py deleted file mode 100644 index 2bfa96368ac75..0000000000000 --- a/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/components/campaign_details_transformations.py +++ /dev/null @@ -1,94 +0,0 @@ -# Copyright (c) 2024 Airbyte, Inc., all rights reserved. - -import logging -from typing import Any, Dict, Mapping, Optional - -from airbyte_cdk.sources.declarative.requesters.error_handlers.backoff_strategies import WaitTimeFromHeaderBackoffStrategy -from airbyte_cdk.sources.declarative.transformations import RecordTransformation -from airbyte_cdk.sources.message.repository import InMemoryMessageRepository -from airbyte_cdk.sources.streams.call_rate import APIBudget -from airbyte_cdk.sources.streams.http.error_handlers import BackoffStrategy, ErrorHandler, HttpStatusErrorHandler -from airbyte_cdk.sources.streams.http.error_handlers.default_error_mapping import DEFAULT_ERROR_MAPPING -from airbyte_cdk.sources.streams.http.error_handlers.response_models import ErrorResolution, FailureType, ResponseAction -from airbyte_cdk.sources.streams.http.http_client import HttpClient -from airbyte_cdk.sources.types import Config, StreamSlice, StreamState - - -class CampaignsDetailedTransformation(RecordTransformation): - """ - Campaigns detailed stream fetches detailed campaigns info: - estimated_recipient_count: integer - campaign_messages: list of objects. - - To get this data CampaignsDetailedTransformation makes extra API requests: - https://a.klaviyo.com/api/campaign-recipient-estimations/{campaign_id} - https://developers.klaviyo.com/en/v2024-10-15/reference/get_messages_for_campaign - """ - - config: Config - - api_revision = "2024-10-15" - url_base = "https://a.klaviyo.com/api/" - name = "campaigns_detailed" - max_retries = 5 - max_time = 60 * 10 - - def __init__(self, config: Config, **kwargs): - self.logger = logging.getLogger("airbyte") - self.config = config - self._api_key = self.config["api_key"] - self._http_client = HttpClient( - name=self.name, - logger=self.logger, - error_handler=self.get_error_handler(), - api_budget=APIBudget(policies=[]), - backoff_strategy=self.get_backoff_strategy(), - message_repository=InMemoryMessageRepository(), - ) - - def transform( - self, - record: Dict[str, Any], - config: Optional[Config] = None, - stream_state: Optional[StreamState] = None, - stream_slice: Optional[StreamSlice] = None, - ) -> None: - self._set_recipient_count(record) - self._set_campaign_message(record) - - def _set_recipient_count(self, record: Mapping[str, Any]) -> None: - campaign_id = record["id"] - _, recipient_count_response = self._http_client.send_request( - url=f"{self.url_base}campaign-recipient-estimations/{campaign_id}", - request_kwargs={}, - headers=self.request_headers(), - http_method="GET", - ) - record["estimated_recipient_count"] = ( - recipient_count_response.json().get("data", {}).get("attributes", {}).get("estimated_recipient_count", 0) - ) - - def _set_campaign_message(self, record: Mapping[str, Any]) -> None: - messages_link = record.get("relationships", {}).get("campaign-messages", {}).get("links", {}).get("related") - if messages_link: - _, campaign_message_response = self._http_client.send_request( - url=messages_link, request_kwargs={}, headers=self.request_headers(), http_method="GET" - ) - record["campaign_messages"] = campaign_message_response.json().get("data") - - def get_backoff_strategy(self) -> BackoffStrategy: - return WaitTimeFromHeaderBackoffStrategy(header="Retry-After", max_waiting_time_in_seconds=self.max_time, parameters={}, config={}) - - def request_headers(self): - return { - "Accept": "application/json", - "Revision": self.api_revision, - "Authorization": f"Klaviyo-API-Key {self._api_key}", - } - - def get_error_handler(self) -> ErrorHandler: - error_mapping = DEFAULT_ERROR_MAPPING | { - 404: ErrorResolution(ResponseAction.IGNORE, FailureType.config_error, "Resource not found. Ignoring.") - } - - return HttpStatusErrorHandler(logger=self.logger, error_mapping=error_mapping, max_retries=self.max_retries) diff --git a/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/components/included_fields_extractor.py b/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/components/included_fields_extractor.py deleted file mode 100644 index e69d904b263f5..0000000000000 --- a/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/components/included_fields_extractor.py +++ /dev/null @@ -1,67 +0,0 @@ -# -# Copyright (c) 2024 Airbyte, Inc., all rights reserved. -# - -from dataclasses import dataclass -from typing import Any, Iterable, Mapping - -import dpath -import requests - -from airbyte_cdk.sources.declarative.extractors.dpath_extractor import DpathExtractor - - -@dataclass -class KlaviyoIncludedFieldExtractor(DpathExtractor): - def extract_records(self, response: requests.Response) -> Iterable[Mapping[str, Any]]: - # Evaluate and retrieve the extraction paths - evaluated_field_paths = [field_path.eval(self.config) for field_path in self._field_path] - target_records = self.extract_records_by_path(response, evaluated_field_paths) - included_relations = list(self.extract_records_by_path(response, ["included"])) - - # Update target records with included records - updated_records = self.update_target_records_with_included(target_records, included_relations) - yield from updated_records - - @staticmethod - def update_target_records_with_included( - target_records: Iterable[Mapping[str, Any]], included_relations: Iterable[Mapping[str, Any]] - ) -> Iterable[Mapping[str, Any]]: - for target_record in target_records: - target_relationships = target_record.get("relationships", {}) - - for included_relation in included_relations: - included_relation_attributes = included_relation.get("attributes", {}) - included_relation_type = included_relation["type"] - included_relation_id = included_relation["id"] - - target_relationship_id = target_relationships.get(included_relation_type, {}).get("data", {}).get("id") - - if included_relation_id == target_relationship_id: - target_relationships[included_relation_type]["data"].update(included_relation_attributes) - - yield target_record - - def extract_records_by_path(self, response: requests.Response, field_paths: list = None) -> Iterable[Mapping[str, Any]]: - try: - response_body = response.json() - except Exception as e: - raise Exception(f"Failed to parse response body as JSON: {e}") - - # Extract data from the response body based on the provided field paths - if not field_paths: - extracted_data = response_body - else: - field_path_str = "/".join(field_paths) # Convert list of field paths to a single string path for dpath - if "*" in field_path_str: - extracted_data = dpath.values(response_body, field_path_str) - else: - extracted_data = dpath.get(response_body, field_path_str, default=[]) - - # Yield extracted data as individual records - if isinstance(extracted_data, list): - yield from extracted_data - elif extracted_data: - yield extracted_data - else: - yield from [] diff --git a/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/components/klaviyo_error_handler.py b/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/components/klaviyo_error_handler.py deleted file mode 100644 index 411cb965fc935..0000000000000 --- a/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/components/klaviyo_error_handler.py +++ /dev/null @@ -1,26 +0,0 @@ -# Copyright (c) 2024 Airbyte, Inc., all rights reserved. - -from typing import Optional, Union - -import requests -from requests.exceptions import InvalidURL - -from airbyte_cdk.models import FailureType -from airbyte_cdk.sources.declarative.requesters.error_handlers import DefaultErrorHandler -from airbyte_cdk.sources.streams.http.error_handlers import ErrorResolution, ResponseAction - - -class KlaviyoErrorHandler(DefaultErrorHandler): - def interpret_response(self, response_or_exception: Optional[Union[requests.Response, Exception]]) -> ErrorResolution: - """ - We have seen `[Errno -3] Temporary failure in name resolution` a couple of times on two different connections - (1fed2ede-2d33-4543-85e3-7d6e5736075d and 1b276f7d-358a-4fe3-a437-6747fd780eed). Retrying the requests on later syncs is working - which makes it sound like a transient issue. - """ - if isinstance(response_or_exception, InvalidURL): - return ErrorResolution( - response_action=ResponseAction.RETRY, - failure_type=FailureType.transient_error, - error_message="source-klaviyo has faced a temporary DNS resolution issue. Retrying...", - ) - return super().interpret_response(response_or_exception) diff --git a/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/components/per_partition_state_migration.py b/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/components/per_partition_state_migration.py deleted file mode 100644 index 560ea8158dcd7..0000000000000 --- a/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/components/per_partition_state_migration.py +++ /dev/null @@ -1,47 +0,0 @@ -# -# Copyright (c) 2024 Airbyte, Inc., all rights reserved. -# - -from typing import Any, Mapping - -from airbyte_cdk.sources.declarative.declarative_stream import DeclarativeStream -from airbyte_cdk.sources.declarative.interpolation.interpolated_string import InterpolatedString -from airbyte_cdk.sources.declarative.migrations.state_migration import StateMigration -from airbyte_cdk.sources.types import Config - - -class PerPartitionToSingleStateMigration(StateMigration): - """ - Transforms the input state for per-partitioned streams from the legacy format to the low-code format. - The cursor field and partition ID fields are automatically extracted from the stream's DatetimebasedCursor and SubstreamPartitionRouter. - - Example input state: - { - "partition": {"event_id": "13506132"}, - "cursor": {"datetime": "2120-10-10 00:00:00+00:00"} - } - Example output state: - { - "datetime": "2120-10-10 00:00:00+00:00" - } - """ - - declarative_stream: DeclarativeStream - config: Config - - def __init__(self, declarative_stream: DeclarativeStream, config: Config): - self._config = config - self.declarative_stream = declarative_stream - self._cursor = declarative_stream.incremental_sync - self._parameters = declarative_stream.parameters - self._cursor_field = InterpolatedString.create(self._cursor.cursor_field, parameters=self._parameters).eval(self._config) - - def should_migrate(self, stream_state: Mapping[str, Any]) -> bool: - return "states" in stream_state - - def migrate(self, stream_state: Mapping[str, Any]) -> Mapping[str, Any]: - if not self.should_migrate(stream_state): - return stream_state - - min_state = min(stream_state.get("states"), key=lambda state: state["cursor"][self._cursor_field]) - return min_state.get("cursor") diff --git a/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/manifest.yaml b/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/manifest.yaml deleted file mode 100644 index 5336019f6006c..0000000000000 --- a/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/manifest.yaml +++ /dev/null @@ -1,2153 +0,0 @@ -version: 6.33.0 -type: DeclarativeSource - -definitions: - # Authenticator - authenticator: - type: ApiKeyAuthenticator - api_token: "Klaviyo-API-Key {{ config['api_key'] }}" - inject_into: - type: RequestOption - field_name: "Authorization" - inject_into: header - - # Requester - requester: - type: HttpRequester - url_base: "https://a.klaviyo.com/api/" - authenticator: "#/definitions/authenticator" - http_method: GET - error_handler: - type: CustomErrorHandler - class_name: source_klaviyo.components.klaviyo_error_handler.KlaviyoErrorHandler - backoff_strategies: - - type: WaitTimeFromHeader - header: "Retry-After" - response_filters: - - type: HttpResponseFilter - action: RATE_LIMITED - http_codes: [429] - - type: HttpResponseFilter - action: FAIL - http_codes: [401, 403] - failure_type: config_error - error_message: Please provide a valid API key and make sure it has permissions to read specified streams. - request_headers: - Accept: "application/json" - Revision: "2024-10-15" - - # Selector - selector: - type: RecordSelector - extractor: - type: DpathExtractor - field_path: ["data"] - - # Paginator - cursor_pagination_strategy: - type: CursorPagination - cursor_value: "{{ response.get('links', {}).get('next') }}" - - paginator: - type: DefaultPaginator - pagination_strategy: "#/definitions/cursor_pagination_strategy" - page_token_option: - type: RequestPath - - # Retrievers - base_retriever: - type: SimpleRetriever - record_selector: "#/definitions/selector" - requester: "#/definitions/requester" - paginator: "#/definitions/paginator" - - profiles_retriever: - $ref: "#/definitions/base_retriever" - paginator: - $ref: "#/definitions/paginator" - pagination_strategy: - $ref: "#/definitions/cursor_pagination_strategy" - page_size: 100 - page_size_option: - type: RequestOption - field_name: "page[size]" - inject_into: request_parameter - requester: - $ref: "#/definitions/requester" - request_headers: - Accept: "application/json" - Revision: "2024-10-15" - request_parameters: - "additional-fields[profile]": "{{ 'predictive_analytics' if not config['disable_fetching_predictive_analytics'] else '' }}" - - # Base streams - base_stream: - type: DeclarativeStream - primary_key: "id" - transformations: - - type: AddFields - fields: - - type: AddedFieldDefinition - path: ["updated"] - value: "{{ record.get('attributes', {}).get('updated') }}" - - base_semi_incremental_stream: - $ref: "#/definitions/base_stream" - retriever: "#/definitions/base_retriever" - incremental_sync: - type: DatetimeBasedCursor - cursor_field: "updated" - datetime_format: "%Y-%m-%dT%H:%M:%S%z" - start_datetime: "{{ config.get('start_date', '2012-01-01T00:00:00Z') }}" - is_client_side_incremental: true - - # Incremental streams - profiles_stream: - # Docs: https://developers.klaviyo.com/en/v2024-10-15/reference/get_profiles - name: "profiles" - $ref: "#/definitions/base_stream" - incremental_sync: - type: DatetimeBasedCursor - cursor_field: "updated" - start_datetime: "{{ config.get('start_date', '2012-01-01T00:00:00Z') }}" - datetime_format: "%Y-%m-%dT%H:%M:%S%z" - cursor_datetime_formats: - - "%Y-%m-%dT%H:%M:%S.%f%z" - - "%Y-%m-%dT%H:%M:%S%z" - - "%Y-%m-%d %H:%M:%S%z" - schema_loader: - type: InlineSchemaLoader - schema: "#/definitions/profiles_schema" - retriever: - $ref: "#/definitions/profiles_retriever" - requester: - $ref: "#/definitions/profiles_retriever/requester" - request_parameters: - $ref: "#/definitions/profiles_retriever/requester/request_parameters" - "filter": "greater-than({{ parameters.cursor_field }},{{ stream_interval.start_time }})" - "sort": "{{ parameters.cursor_field }}" - record_selector: - $ref: "#/definitions/selector" - schema_normalization: Default - $parameters: - path: "profiles" - cursor_field: "updated" - - global_exclusions_stream: - # Docs: https://developers.klaviyo.com/en/v2024-10-15/reference/get_profiles - name: "global_exclusions" - $ref: "#/definitions/profiles_stream" - schema_loader: - type: InlineSchemaLoader - schema: "#/definitions/global_exclusions_schema" - retriever: - $ref: "#/definitions/profiles_retriever" - requester: - $ref: "#/definitions/requester" - request_parameters: - "additional-fields[profile]": "subscriptions" - record_selector: - $ref: "#/definitions/selector" - record_filter: - type: RecordFilter - condition: "{{ record['attributes']['subscriptions']['email']['marketing']['suppression'] }}" - transformations: - - type: AddFields - fields: - - type: AddedFieldDefinition - path: ["updated"] - value: "{{ record.get('attributes', {}).get('updated') }}" - - type: AddedFieldDefinition - path: - [ - "attributes", - "subscriptions", - "email", - "marketing", - "suppressions", - ] - value: "{{ record['attributes']['subscriptions']['email']['marketing']['suppression'] }}" - - type: RemoveFields - field_pointers: - [["attributes", "subscriptions", "email", "marketing", "suppression"]] - - events_stream: - # Docs: https://developers.klaviyo.com/en/reference/get_events - name: "events" - $ref: "#/definitions/base_stream" - incremental_sync: - type: DatetimeBasedCursor - cursor_field: "datetime" - start_datetime: "{{ config.get('start_date', '2012-01-01T00:00:00Z') }}" - datetime_format: "%Y-%m-%dT%H:%M:%S%z" - cursor_datetime_formats: - - "%Y-%m-%dT%H:%M:%S.%f%z" - - "%Y-%m-%dT%H:%M:%S%z" - - "%Y-%m-%d %H:%M:%S%z" - step: P7D - cursor_granularity: PT1S - retriever: - $ref: "#/definitions/base_retriever" - requester: - $ref: "#/definitions/requester" - request_parameters: - "fields[metric]": "name,created,updated,integration" - "include": "metric,attributions" - "filter": "greater-or-equal({{ parameters.cursor_field }},{{ stream_interval.start_time }}),less-or-equal({{ parameters.cursor_field }},{{ stream_interval.end_time }})" - "sort": "{{ parameters.cursor_field }}" - schema_loader: - type: InlineSchemaLoader - schema: "#/definitions/events_schema" - transformations: - - type: AddFields - fields: - - type: AddedFieldDefinition - path: ["datetime"] - value: "{{ record.get('attributes', {}).get('datetime') }}" - $parameters: - path: "events" - cursor_field: "datetime" - - email_templates_stream: - # Docs: https://developers.klaviyo.com/en/reference/get_templates - name: "email_templates" - $ref: "#/definitions/base_stream" - incremental_sync: - type: DatetimeBasedCursor - cursor_field: "updated" - start_datetime: "{{ config.get('start_date', '2012-01-01T00:00:00Z') }}" - datetime_format: "%Y-%m-%dT%H:%M:%S%z" - cursor_datetime_formats: - - "%Y-%m-%dT%H:%M:%S.%f%z" - - "%Y-%m-%dT%H:%M:%S%z" - - "%Y-%m-%d %H:%M:%S%z" - schema_loader: - type: InlineSchemaLoader - schema: "#/definitions/email_templates_schema" - retriever: - $ref: "#/definitions/base_retriever" - requester: - $ref: "#/definitions/requester" - request_parameters: - "filter": "greater-than({{ parameters.cursor_field }},{{ stream_interval.start_time }})" - "sort": "{{ parameters.cursor_field }}" - $parameters: - path: "templates" - cursor_field: "updated" - - campaigns_stream: - # Docs: https://developers.klaviyo.com/en/v2024-10-15/reference/get_campaigns - name: "campaigns" - $ref: "#/definitions/base_stream" - incremental_sync: - type: DatetimeBasedCursor - cursor_field: "updated_at" - start_datetime: "{{ config.get('start_date', '2012-01-01T00:00:00Z') }}" - datetime_format: "%Y-%m-%dT%H:%M:%S%z" - cursor_datetime_formats: - - "%Y-%m-%dT%H:%M:%S.%f%z" - - "%Y-%m-%dT%H:%M:%S%z" - - "%Y-%m-%d %H:%M:%S%z" - step: P60D - cursor_granularity: PT1S - retriever: - $ref: "#/definitions/base_retriever" - requester: - $ref: "#/definitions/requester" - request_parameters: - "filter": "and(greater-or-equal({{ parameters.cursor_field }},{{ stream_interval.start_time }}),less-or-equal({{ parameters.cursor_field }},{{ stream_interval.end_time }}),equals(messages.channel,'{{ stream_partition['campaign_type'] }}'),equals(archived,{{ stream_partition['archived'] }}))" - "sort": "{{ parameters.cursor_field }}" - partition_router: - - type: ListPartitionRouter - values: - - sms - - email - cursor_field: campaign_type - - type: ListPartitionRouter - values: - - "true" - - "false" - cursor_field: archived - state_migrations: - - type: CustomStateMigration - class_name: source_klaviyo.components.archived_to_per_partition_state_migration.CampaignsStateMigration - schema_loader: - type: InlineSchemaLoader - schema: "#/definitions/campaigns_schema" - $parameters: - path: "campaigns" - cursor_field: "updated_at" - transformations: - - type: AddFields - fields: - - type: AddedFieldDefinition - path: ["updated_at"] - value: "{{ record.get('attributes', {}).get('updated_at') }}" - - type: AddFields - fields: - - type: AddedFieldDefinition - path: ["attributes", "channel"] - value: "{{ stream_partition['campaign_type'] }}" - - campaigns_detailed_stream: - # Docs: https://developers.klaviyo.com/en/v2024-10-15/reference/get_messages_for_campaign - name: "campaigns_detailed" - $ref: "#/definitions/campaigns_stream" - schema_loader: - type: InlineSchemaLoader - schema: "#/definitions/campaigns_detailed_schema" - transformations: - - type: AddFields - fields: - - type: AddedFieldDefinition - path: ["updated_at"] - value: "{{ record.get('attributes', {}).get('updated_at') }}" - - type: CustomTransformation - class_name: source_klaviyo.components.campaign_details_transformations.CampaignsDetailedTransformation - - flows_stream: - # Docs: https://developers.klaviyo.com/en/reference/get_flows - name: "flows" - $ref: "#/definitions/base_stream" - incremental_sync: - type: DatetimeBasedCursor - cursor_field: "updated" - start_datetime: "{{ config.get('start_date', '2012-01-01T00:00:00Z') }}" - datetime_format: "%Y-%m-%dT%H:%M:%S%z" - cursor_datetime_formats: - - "%Y-%m-%dT%H:%M:%S.%f%z" - - "%Y-%m-%dT%H:%M:%S%z" - - "%Y-%m-%d %H:%M:%S%z" - step: P30D - cursor_granularity: PT1S - retriever: - $ref: "#/definitions/base_retriever" - requester: - $ref: "#/definitions/requester" - request_parameters: - "filter": "and(greater-or-equal({{ parameters.cursor_field }},{{ stream_interval.start_time }}),less-or-equal({{ parameters.cursor_field }},{{ stream_interval.end_time }}),equals(archived,{{ stream_partition['archived'] }}))" - "sort": "{{ parameters.cursor_field }}" - partition_router: - - type: ListPartitionRouter - values: - - "true" - - "false" - cursor_field: archived - state_migrations: - - type: CustomStateMigration - class_name: source_klaviyo.components.archived_to_per_partition_state_migration.ArchivedToPerPartitionStateMigration - schema_loader: - type: InlineSchemaLoader - schema: "#/definitions/flows_schema" - $parameters: - path: "flows" - cursor_field: "updated" - transformations: - - type: AddFields - fields: - - type: AddedFieldDefinition - path: ["updated"] - value: "{{ record.get('attributes', {}).get('updated') }}" - - # Semi-Incremental streams - metrics_stream: - # Docs: https://developers.klaviyo.com/en/reference/get_metrics - name: "metrics" - $ref: "#/definitions/base_semi_incremental_stream" - schema_loader: - type: InlineSchemaLoader - schema: "#/definitions/metrics_schema" - $parameters: - path: "metrics" - - lists_stream: - # Docs: https://developers.klaviyo.com/en/reference/get_lists - name: "lists" - $ref: "#/definitions/base_semi_incremental_stream" - schema_loader: - type: InlineSchemaLoader - schema: "#/definitions/lists_schema" - $parameters: - path: "lists" - - lists_detailed_stream: - # Docs: https://developers.klaviyo.com/en/reference/get_list - name: "lists_detailed" - $ref: "#/definitions/base_semi_incremental_stream" - schema_loader: - type: InlineSchemaLoader - schema: "#/definitions/lists_detailed_schema" - retriever: - $ref: "#/definitions/base_retriever" - requester: - $ref: "#/definitions/requester" - request_parameters: - "additional-fields[list]": "profile_count" - partition_router: - type: SubstreamPartitionRouter - parent_stream_configs: - - type: ParentStreamConfig - parent_key: "id" - stream: "#/definitions/lists_stream" - partition_field: "id" - $parameters: - path: "lists/{{ stream_slice.id }}" - - events_detailed_stream: - # Docs: https://developers.klaviyo.com/en/reference/get_event - name: "events_detailed" - $ref: "#/definitions/base_stream" - incremental_sync: - type: DatetimeBasedCursor - cursor_field: "{{ parameters.get('cursor_field') }}" - start_datetime: "{{ config.get('start_date', '2012-01-01T00:00:00Z') }}" - datetime_format: "%Y-%m-%dT%H:%M:%S%z" - cursor_datetime_formats: - - "%Y-%m-%dT%H:%M:%S.%f%z" - - "%Y-%m-%dT%H:%M:%S%z" - - "%Y-%m-%d %H:%M:%S%z" - schema_loader: - type: InlineSchemaLoader - schema: "#/definitions/events_detailed_schema" - retriever: - $ref: "#/definitions/base_retriever" - record_selector: - type: RecordSelector - extractor: - type: CustomRecordExtractor - class_name: source_klaviyo.components.included_fields_extractor.KlaviyoIncludedFieldExtractor - field_path: ["data"] - requester: - $ref: "#/definitions/requester" - request_parameters: - "include": "metric,attributions" - "fields[metric]": "name" - "filter": "greater-than({{ parameters.cursor_field }},{{ stream_interval.start_time }})" - "sort": "{{ parameters.cursor_field }}" - state_migrations: - - type: CustomStateMigration - class_name: source_klaviyo.components.per_partition_state_migration.PerPartitionToSingleStateMigration - transformations: - - type: AddFields - fields: - - type: AddedFieldDefinition - path: ["datetime"] - value: "{{ record.get('attributes', {}).get('datetime') }}" - $parameters: - path: "events" - cursor_field: "datetime" - - # Schemas - shared: - list_properties: - type: - type: string - id: - type: string - updated: - type: ["null", string] - format: date-time - attributes: - type: ["null", object] - properties: - name: - type: string - created: - type: ["null", string] - format: date-time - updated: - type: ["null", string] - format: date-time - opt_in_process: - type: ["null", string] - links: - type: ["null", object] - additionalProperties: true - properties: - self: - type: string - relationships: - type: ["null", object] - additionalProperties: true - properties: - profiles: - type: ["null", object] - properties: - links: - type: ["null", object] - properties: - self: - type: string - related: - type: string - tags: - type: ["null", object] - properties: - data: - type: array - items: - type: ["null", object] - properties: - type: - type: string - id: - type: string - links: - type: ["null", object] - properties: - self: - type: string - related: - type: string - flow-triggers: - type: ["null", object] - properties: - data: - type: ["null", object] - properties: - type: - type: ["null", string] - id: - type: ["null", string] - links: - type: ["null", object] - properties: - self: - type: ["null", string] - related: - type: ["null", string] - - subscriptions: - type: ["null", object] - properties: - email: - type: ["null", object] - properties: - marketing: - type: ["null", object] - properties: - can_receive_email_marketing: - type: boolean - consent: - type: string - timestamp: - type: ["null", string] - format: date-time - consent_timestamp: - type: ["null", string] - format: date-time - last_updated: - type: ["null", string] - format: date-time - method: - type: ["null", string] - method_detail: - type: ["null", string] - custom_method_detail: - type: ["null", string] - double_optin: - type: ["null", boolean] - suppressions: - type: ["null", array] - items: - type: ["null", object] - properties: - reason: - type: string - timestamp: - type: string - format: date-time - list_suppressions: - type: ["null", array] - items: - type: ["null", object] - properties: - list_id: - type: string - reason: - type: string - timestamp: - type: string - format: date-time - sms: - type: ["null", object] - properties: - marketing: - type: ["null", object] - properties: - can_receive_sms_marketing: - type: ["null", boolean] - consent: - type: ["null", string] - consent_timestamp: - type: ["null", string] - format: date-time - method: - type: ["null", string] - method_detail: - type: ["null", string] - last_updated: - type: ["null", string] - format: date-time - timestamp: - type: ["null", string] - format: date-time - transactional: - type: ["null", object] - properties: - can_receive_sms_marketing: - type: ["null", boolean] - consent: - type: ["null", string] - consent_timestamp: - type: ["null", string] - format: date-time - method: - type: ["null", string] - method_detail: - type: ["null", string] - last_updated: - type: ["null", string] - format: date-time - timestamp: - type: ["null", string] - format: date-time - mobile_push: - type: ["null", object] - properties: - marketing: - type: ["null", object] - properties: - can_receive_sms_marketing: - type: ["null", boolean] - consent: - type: ["null", string] - consent_timestamp: - type: ["null", string] - format: date-time - - profiles_schema: - $schema: "http://json-schema.org/draft-07/schema#" - type: object - additionalProperties: true - properties: - type: - type: ["null", string] - id: - type: string - updated: - type: ["null", string] - format: date-time - attributes: - type: ["null", object] - additionalProperties: true - properties: - email: - type: ["null", string] - phone_number: - type: ["null", string] - anonymous_id: - type: ["null", string] - external_id: - type: ["null", string] - first_name: - type: ["null", string] - last_name: - type: ["null", string] - organization: - type: ["null", string] - locale: - type: ["null", string] - title: - type: ["null", string] - image: - type: ["null", string] - created: - type: ["null", string] - format: date-time - updated: - type: ["null", string] - format: date-time - last_event_date: - type: ["null", string] - format: date-time - location: - type: ["null", object] - properties: - address1: - type: ["null", string] - address2: - type: ["null", string] - city: - type: ["null", string] - country: - type: ["null", string] - latitude: - oneOf: - - type: "null" - - type: number - - type: string - longitude: - oneOf: - - type: "null" - - type: number - - type: string - region: - type: ["null", string] - zip: - type: ["null", string] - timezone: - type: ["null", string] - ip: - type: ["null", string] - properties: - type: ["null", object] - additionalProperties: true - subscriptions: "#/definitions/shared/subscriptions" - predictive_analytics: - type: ["null", object] - properties: - historic_clv: - type: ["null", number] - predicted_clv: - type: ["null", number] - total_clv: - type: ["null", number] - historic_number_of_orders: - type: ["null", integer] - predicted_number_of_orders: - type: ["null", number] - average_days_between_orders: - type: ["null", number] - average_order_value: - type: ["null", number] - churn_probability: - type: ["null", number] - expected_date_of_next_order: - type: ["null", string] - links: - type: ["null", object] - properties: - self: - type: ["null", string] - relationships: - type: ["null", object] - properties: - lists: - type: ["null", object] - properties: - links: - type: ["null", object] - properties: - self: - type: ["null", string] - related: - type: ["null", string] - segments: - type: ["null", object] - properties: - links: - type: ["null", object] - properties: - self: - type: ["null", string] - related: - type: ["null", string] - segments: - type: ["null", object] - - global_exclusions_schema: - $schema: "http://json-schema.org/draft-07/schema#" - type: object - additionalProperties: true - properties: - type: - type: ["null", string] - id: - type: string - updated: - type: ["null", string] - format: date-time - attributes: - type: ["null", object] - additionalProperties: true - properties: - email: - type: ["null", string] - phone_number: - type: ["null", string] - anonymous_id: - type: ["null", string] - external_id: - type: ["null", string] - first_name: - type: ["null", string] - last_name: - type: ["null", string] - organization: - type: ["null", string] - locale: - type: ["null", string] - title: - type: ["null", string] - image: - type: ["null", string] - created: - type: ["null", string] - format: date-time - updated: - type: ["null", string] - format: date-time - last_event_date: - type: ["null", string] - format: date-time - location: - type: ["null", object] - properties: - address1: - type: ["null", string] - address2: - type: ["null", string] - city: - type: ["null", string] - country: - type: ["null", string] - latitude: - oneOf: - - type: "null" - - type: number - - type: string - longitude: - oneOf: - - type: "null" - - type: number - - type: string - region: - type: ["null", string] - zip: - type: ["null", string] - timezone: - type: ["null", string] - ip: - type: ["null", string] - properties: - type: ["null", object] - additionalProperties: true - subscriptions: "#/definitions/shared/subscriptions" - predictive_analytics: - type: ["null", object] - properties: - historic_clv: - type: ["null", number] - predicted_clv: - type: ["null", number] - total_clv: - type: ["null", number] - historic_number_of_orders: - type: ["null", integer] - predicted_number_of_orders: - type: ["null", number] - average_days_between_orders: - type: ["null", number] - average_order_value: - type: ["null", number] - churn_probability: - type: ["null", number] - expected_date_of_next_order: - type: ["null", string] - links: - type: ["null", object] - properties: - self: - type: ["null", string] - relationships: - type: ["null", object] - properties: - lists: - type: ["null", object] - properties: - links: - type: ["null", object] - properties: - self: - type: ["null", string] - related: - type: ["null", string] - segments: - type: ["null", object] - properties: - links: - type: ["null", object] - properties: - self: - type: ["null", string] - related: - type: ["null", string] - segments: - type: ["null", object] - - events_schema: - $schema: "http://json-schema.org/draft-07/schema#" - type: object - additionalProperties: true - properties: - type: - type: string - id: - type: string - datetime: - type: string - format: date-time - attributes: - type: ["null", object] - properties: - timestamp: - type: integer - event_properties: - type: ["null", object] - additionalProperties: true - datetime: - type: string - format: date-time - uuid: - type: string - links: - type: ["null", object] - properties: - self: - type: string - relationships: - type: ["null", object] - properties: - profile: - type: ["null", object] - properties: - data: - type: ["null", object] - properties: - type: - type: string - id: - type: string - links: - type: ["null", object] - additionalProperties: true - properties: - self: - type: string - related: - type: string - metric: - type: ["null", object] - properties: - data: - type: ["null", object] - properties: - type: - type: string - id: - type: string - links: - type: ["null", object] - additionalProperties: true - properties: - self: - type: string - related: - type: string - attributions: - type: ["null", object] - properties: - data: - type: ["null", array] - items: - type: ["null", object] - properties: - type: - type: string - id: - type: string - links: - type: ["null", object] - additionalProperties: true - properties: - self: - type: string - related: - type: string - - events_detailed_schema: - $schema: "http://json-schema.org/draft-07/schema#" - type: object - additionalProperties: true - properties: - type: - type: string - id: - type: string - datetime: - type: string - format: date-time - attributes: - type: ["null", object] - properties: - timestamp: - type: integer - event_properties: - type: ["null", object] - additionalProperties: true - datetime: - type: string - format: date-time - uuid: - type: string - links: - type: ["null", object] - properties: - self: - type: string - relationships: - type: ["null", object] - properties: - profile: - type: ["null", object] - properties: - data: - type: ["null", object] - properties: - type: - type: string - id: - type: string - links: - type: ["null", object] - additionalProperties: true - properties: - self: - type: string - related: - type: string - metric: - type: ["null", object] - properties: - data: - type: ["null", object] - properties: - type: - type: string - id: - type: string - name: - type: string - links: - type: ["null", object] - additionalProperties: true - properties: - self: - type: string - related: - type: string - - email_templates_schema: - $schema: "http://json-schema.org/draft-07/schema#" - type: object - additionalProperties: true - properties: - type: - type: string - id: - type: string - updated: - type: ["null", string] - format: date-time - attributes: - type: ["null", object] - additionalProperties: true - properties: - name: - type: string - editor_type: - type: ["null", string] - html: - type: string - text: - type: ["null", string] - created: - type: ["null", string] - format: date-time - updated: - type: ["null", string] - format: date-time - company_id: - type: ["null", string] - links: - type: ["null", object] - additionalProperties: true - properties: - self: - type: string - - metrics_schema: - $schema: "http://json-schema.org/draft-07/schema#" - type: object - additionalProperties: true - properties: - type: - type: string - id: - type: string - updated: - type: string - format: date-time - attributes: - type: ["null", object] - properties: - name: - type: string - created: - type: string - format: date-time - updated: - type: string - format: date-time - integration: - type: ["null", object] - additionalProperties: true - links: - type: ["null", object] - additionalProperties: true - properties: - self: - type: string - relationships: - type: ["null", object] - properties: - flow-triggers: - type: ["null", object] - properties: - data: - type: ["null", object] - properties: - type: - type: ["null", string] - id: - type: ["null", string] - links: - type: ["null", object] - properties: - self: - type: ["null", string] - related: - type: ["null", string] - - lists_schema: - $schema: "http://json-schema.org/draft-07/schema#" - type: object - additionalProperties: true - properties: "#/definitions/shared/list_properties" - - lists_detailed_schema: - $schema: "http://json-schema.org/draft-07/schema#" - type: object - additionalProperties: true - properties: - $ref: "#/definitions/shared/list_properties" - attributes: - type: ["null", object] - properties: - name: - type: string - created: - type: ["null", string] - format: date-time - updated: - type: ["null", string] - format: date-time - opt_in_process: - type: ["null", string] - profile_count: - type: ["null", integer] - - campaigns_schema: - $schema: "http://json-schema.org/draft-07/schema#" - type: object - additionalProperties: true - properties: - type: - type: string - id: - type: string - updated_at: - type: - - "null" - - string - format: date-time - attributes: - type: - - "null" - - object - additionalProperties: true - properties: - name: - type: string - status: - type: string - archived: - type: boolean - channel: - type: string - audiences: - type: - - "null" - - object - additionalProperties: true - properties: - included: - type: - - "null" - - array - items: - type: - - "null" - - string - excluded: - type: - - "null" - - array - items: - type: - - "null" - - string - send_options: - type: - - "null" - - object - properties: - ignore_unsubscribes: - type: - - "null" - - boolean - use_smart_sending: - type: - - "null" - - boolean - message: - type: string - tracking_options: - type: - - "null" - - object - additionalProperties: true - properties: - add_tracking_params: - type: - - "null" - - boolean - is_tracking_opens: - type: - - "null" - - boolean - is_tracking_clicks: - type: - - "null" - - boolean - is_add_utm: - type: - - "null" - - boolean - utm_params: - type: - - "null" - - array - items: - type: - - "null" - - object - properties: - name: - type: string - value: - type: string - send_strategy: - type: - - "null" - - object - additionalProperties: true - properties: - method: - type: string - options_static: - type: - - "null" - - object - properties: - datetime: - type: string - format: date-time - airbyte_type: timestamp_without_timezone - is_local: - type: - - "null" - - boolean - send_past_recipients_immediately: - type: - - "null" - - boolean - options_throttled: - type: - - "null" - - object - properties: - datetime: - type: string - format: date-time - airbyte_type: timestamp_without_timezone - throttle_percentage: - type: integer - options_sto: - type: - - "null" - - object - properties: - date: - type: string - format: date - created_at: - type: - - "null" - - string - format: date-time - scheduled_at: - type: - - "null" - - string - format: date-time - updated_at: - type: - - "null" - - string - format: date-time - send_time: - type: - - "null" - - string - format: date-time - links: - type: - - "null" - - object - additionalProperties: true - properties: - self: - type: string - relationships: - type: - - "null" - - object - additionalProperties: true - properties: - tags: - type: - - "null" - - object - properties: - data: - type: array - items: - type: - - "null" - - object - properties: - type: - type: string - id: - type: string - links: - type: - - "null" - - object - properties: - self: - type: string - related: - type: string - campaign-messages: - type: - - "null" - - object - properties: - data: - type: array - items: - type: - - "null" - - object - properties: - type: - type: string - id: - type: string - links: - type: - - "null" - - object - properties: - self: - type: string - related: - type: string - - campaigns_detailed_schema: - $schema: "http://json-schema.org/draft-07/schema#" - type: object - additionalProperties: true - properties: - type: - type: string - id: - type: string - updated_at: - type: - - "null" - - string - format: date-time - attributes: - type: - - "null" - - object - additionalProperties: true - properties: - name: - type: string - status: - type: string - archived: - type: boolean - channel: - type: string - audiences: - type: - - "null" - - object - additionalProperties: true - properties: - included: - type: - - "null" - - array - items: - type: - - "null" - - string - excluded: - type: - - "null" - - array - items: - type: - - "null" - - string - send_options: - type: - - "null" - - object - properties: - ignore_unsubscribes: - type: - - "null" - - boolean - use_smart_sending: - type: - - "null" - - boolean - message: - type: string - tracking_options: - type: - - "null" - - object - additionalProperties: true - properties: - add_tracking_params: - type: - - "null" - - boolean - is_tracking_opens: - type: - - "null" - - boolean - is_tracking_clicks: - type: - - "null" - - boolean - is_add_utm: - type: - - "null" - - boolean - utm_params: - type: - - "null" - - array - items: - type: - - "null" - - object - properties: - name: - type: string - value: - type: string - send_strategy: - type: - - "null" - - object - additionalProperties: true - properties: - method: - type: string - options_static: - type: - - "null" - - object - properties: - datetime: - type: string - format: date-time - airbyte_type: timestamp_without_timezone - is_local: - type: - - "null" - - boolean - send_past_recipients_immediately: - type: - - "null" - - boolean - options_throttled: - type: - - "null" - - object - properties: - datetime: - type: string - format: date-time - airbyte_type: timestamp_without_timezone - throttle_percentage: - type: integer - options_sto: - type: - - "null" - - object - properties: - date: - type: string - format: date - created_at: - type: - - "null" - - string - format: date-time - scheduled_at: - type: - - "null" - - string - format: date-time - updated_at: - type: - - "null" - - string - format: date-time - send_time: - type: - - "null" - - string - format: date-time - links: - type: - - "null" - - object - additionalProperties: true - properties: - self: - type: string - relationships: - type: - - "null" - - object - additionalProperties: true - properties: - tags: - type: - - "null" - - object - properties: - data: - type: array - items: - type: - - "null" - - object - properties: - type: - type: string - id: - type: string - links: - type: - - "null" - - object - properties: - self: - type: string - related: - type: string - campaign-messages: - type: - - "null" - - object - properties: - data: - type: array - items: - type: - - "null" - - object - properties: - type: - type: string - id: - type: string - links: - type: - - "null" - - object - properties: - self: - type: string - related: - type: string - estimated_recipient_count: - type: - - "null" - - integer - campaign_messages: - type: - - "null" - - array - items: - type: - - "null" - - object - properties: - type: - type: - - "null" - - string - id: - type: - - "null" - - string - attributes: - type: - - "null" - - object - properties: - label: - type: - - "null" - - string - channel: - type: - - "null" - - string - content: - type: - - "null" - - object - properties: - subject: - type: - - "null" - - string - preview_text: - type: - - "null" - - string - from_email: - type: - - "null" - - string - from_label: - type: - - "null" - - string - reply_to_email: - type: - - "null" - - string - cc_email: - type: - - "null" - - string - bcc_email: - type: - - "null" - - string - send_times: - type: - - "null" - - array - items: - type: - - "null" - - object - properties: - datetime: - type: - - "null" - - string - format: date-time - is_local: - type: - - "null" - - boolean - render_options: - type: - - "null" - - object - properties: - shorten_links: - type: - - "null" - - boolean - add_org_prefix: - type: - - "null" - - boolean - add_info_link: - type: - - "null" - - boolean - add_opt_out_language: - type: - - "null" - - boolean - created_at: - type: - - "null" - - string - format: date-time - airbyte_type: timestamp_without_timezone - updated_at: - type: - - "null" - - string - format: date-time - airbyte_type: timestamp_without_timezone - links: - type: - - "null" - - object - properties: - self: - type: - - "null" - - string - relationships: - type: - - "null" - - object - properties: - campaign: - type: - - "null" - - object - properties: - data: - type: - - "null" - - object - properties: - type: - type: - - "null" - - string - id: - type: - - "null" - - string - links: - type: - - "null" - - object - properties: - self: - type: - - "null" - - string - related: - type: - - "null" - - string - template: - type: - - "null" - - object - properties: - data: - type: - - "null" - - object - properties: - type: - type: - - "null" - - string - id: - type: - - "null" - - string - links: - type: - - "null" - - object - properties: - self: - type: - - "null" - - string - related: - type: - - "null" - - string - - flows_schema: - $schema: "http://json-schema.org/draft-07/schema#" - type: object - additionalProperties: true - properties: - type: - type: string - id: - type: string - updated: - type: string - format: date-time - attributes: - type: - - "null" - - object - additionalProperties: true - properties: - name: - type: string - status: - type: string - archived: - type: boolean - created: - type: string - format: date-time - updated: - type: string - format: date-time - trigger_type: - type: string - links: - type: - - "null" - - object - additionalProperties: true - properties: - self: - type: string - relationships: - type: - - "null" - - object - additionalProperties: true - properties: - flow-actions: - type: - - "null" - - object - properties: - data: - type: array - items: - type: - - "null" - - object - additionalProperties: true - properties: - type: - type: string - id: - type: string - links: - type: - - "null" - - object - properties: - self: - type: string - related: - type: string - tags: - type: - - "null" - - object - properties: - data: - type: array - items: - type: - - "null" - - object - properties: - type: - type: string - id: - type: string - links: - type: - - "null" - - object - properties: - self: - type: string - related: - type: string - -streams: - # Incremental streams - - "#/definitions/profiles_stream" - - "#/definitions/global_exclusions_stream" - - "#/definitions/events_stream" - - "#/definitions/events_detailed_stream" - - "#/definitions/email_templates_stream" - - "#/definitions/campaigns_stream" - - "#/definitions/campaigns_detailed_stream" - - "#/definitions/flows_stream" - - # Semi-Incremental streams - - "#/definitions/metrics_stream" - - "#/definitions/lists_stream" - - "#/definitions/lists_detailed_stream" - -check: - type: CheckStream - stream_names: - - metrics - -spec: - type: Spec - documentation_url: "https://docs.airbyte.com/integrations/sources/klaviyo" - connection_specification: - $schema: "http://json-schema.org/draft-07/schema#" - title: "Klaviyo Spec" - type: object - properties: - api_key: - type: string - title: "Api Key" - description: 'Klaviyo API Key. See our docs if you need help finding this key.' - airbyte_secret: true - order: 0 - start_date: - type: string - title: "Start Date" - description: "UTC date and time in the format 2017-01-25T00:00:00Z. Any data before this date will not be replicated. This field is optional - if not provided, all data will be replicated." - pattern: "^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$" - examples: ["2017-01-25T00:00:00Z"] - format: date-time - order: 1 - disable_fetching_predictive_analytics: - type: boolean - title: "Disable Fetching Predictive Analytics" - description: >- - Certain streams like the profiles stream can retrieve predictive analytics data from Klaviyo's - API. However, at high volume, this can lead to service availability issues on the API which can - be improved by not fetching this field. WARNING: Enabling this setting will stop the - "predictive_analytics" column from being populated in your downstream destination. - order: 2 - num_workers: - type: integer - title: Number of concurrent workers - minimum: 1 - maximum: 50 - default: 10 - examples: [1, 2, 3] - description: >- - The number of worker threads to use for the sync. - The performance upper boundary is based on the limit of your Klaviyo plan. - More info about the rate limit plan tiers can be found on Klaviyo's API docs. - order: 3 - required: ["api_key"] - -metadata: - testedStreams: - profiles: - streamHash: 7d27c2aee801ec7d0038722136c6b7e06b14a9ed - hasResponse: true - responsesAreSuccessful: true - hasRecords: true - primaryKeysArePresent: true - primaryKeysAreUnique: true - global_exclusions: - streamHash: 7e7633526c2855390903d6e60973bb13b23272d7 - hasResponse: true - responsesAreSuccessful: true - hasRecords: true - primaryKeysArePresent: true - primaryKeysAreUnique: true - events: - streamHash: af0180236001cbacc0788046bdc916026e1f82f6 - hasResponse: true - responsesAreSuccessful: true - hasRecords: false - primaryKeysArePresent: true - primaryKeysAreUnique: true - lists: - streamHash: 9edcccbf069463bf70bdc40db756e0f81eba032b - hasResponse: true - responsesAreSuccessful: true - hasRecords: true - primaryKeysArePresent: true - primaryKeysAreUnique: true - email_templates: - streamHash: 4c12cf304ffe3cd0fcaba1479498ad19c18c6f32 - hasResponse: true - responsesAreSuccessful: true - hasRecords: true - primaryKeysArePresent: true - primaryKeysAreUnique: true - metrics: - streamHash: 96e06644c47a223a29c85dc4318ec5f7da1cc414 - hasResponse: true - responsesAreSuccessful: true - hasRecords: true - primaryKeysArePresent: true - primaryKeysAreUnique: true - lists_detailed: - streamHash: 34e4a9f1fb0c879b915d8558d67feb887d76f8e5 - hasResponse: true - responsesAreSuccessful: false - hasRecords: true - primaryKeysArePresent: true - primaryKeysAreUnique: true - -api_budget: - type: HTTPAPIBudget - # Each policy here uses a MovingWindowCallRatePolicy with two rates: - # one for burst (per-second) and one for steady (per-minute). - policies: - # Campaigns and Campaigns Detailed - - type: MovingWindowCallRatePolicy - rates: - - limit: 10 # burst: 10 calls per second - interval: PT1S - - limit: 150 # steady: 150 calls per minute - interval: PT1M - matchers: - - method: GET - url_path_pattern: "^/api/campaigns($|/)" # matches '/campaigns' - # Flows - - type: MovingWindowCallRatePolicy - rates: - - limit: 3 # burst: 3 calls per second - interval: PT1S - - limit: 60 # steady: 60 calls per minute - interval: PT1M - matchers: - - method: GET - url_path_pattern: "^/api/flows($|/)" # matches '/flows' - # Profiles (and global_exclusions share the same endpoint) - - type: MovingWindowCallRatePolicy - rates: - - limit: 10 # burst: 10 calls per second - interval: PT1S - - limit: 150 # steady: 150 calls per minute - interval: PT1M - matchers: - - method: GET - url_path_pattern: "^/api/profiles($|/)" # matches '/profiles' (exact or with trailing slash/extra) - # Events (and events_detailed share the same endpoint) - - type: MovingWindowCallRatePolicy - rates: - - limit: 350 # burst: 350 calls per second - interval: PT1S - - limit: 3500 # steady: 3500 calls per minute - interval: PT1M - matchers: - - method: GET - url_path_pattern: "^/api/events($|/)" # matches '/events' and '/events_detailed' if using same endpoint - # Email Templates - - type: MovingWindowCallRatePolicy - rates: - - limit: 10 # burst: 10 calls per second - interval: PT1S - - limit: 150 # steady: 150 calls per minute - interval: PT1M - matchers: - - method: GET - url_path_pattern: "^/api/templates($|/)" # matches '/templates' - # Metrics - - type: MovingWindowCallRatePolicy - rates: - - limit: 10 # burst: 10 calls per second - interval: PT1S - - limit: 150 # steady: 150 calls per minute - interval: PT1M - matchers: - - method: GET - url_path_pattern: "^/api/metrics($|/)" - # Lists (the parent endpoint for lists streams) - - type: MovingWindowCallRatePolicy - rates: - - limit: 75 # burst: 75 calls per second - interval: PT1S - - limit: 700 # steady: 700 calls per minute - interval: PT1M - matchers: - - method: GET - url_path_pattern: "^/api/lists$" # exactly '/lists' - # Lists Detailed (uses a different URL path – note the extra segment) - - type: MovingWindowCallRatePolicy - rates: - - limit: 1 # burst: 1 call per second - interval: PT1S - - limit: 15 # steady: 15 calls per minute - interval: PT1M - matchers: - - method: GET - url_path_pattern: "^/api/lists/" # matches any path beginning with '/lists/' (e.g. '/lists/123') - params: - "additional-fields[list]": "profile_count" # Other API budget settings: - status_codes_for_ratelimit_hit: [429] - -# Klaviyo's rate limiting is different by endpoints: -# - XS: 1/s burst; 15/m steady -# - S: 3/s burst; 60/m steady -# - M: 10/s burst; 150/m steady -# - L: 75/s burst; 700/m steady -# - XL: 350/s burst; 3500/m steady -# - `lists_detailed`: 1/s burst, 15/m steady (Rate limits when using the additional-fields[list]=profile_count parameter in your API request) - -# As of 2024-11-11, we have the following streams: -# | Stream | Endpoint | Klaviyo Rate Limit Size | Source Concurrency Between Streams | Source Concurrency Within Stream | Source Max Number of Threads Sharing Rate Limits | | -#|-------------------|----------------------------------------------------------------------|-------------------------|------------------------------------|---------------------------------------------------|------------------------------------------------------------------|---------------------------------------------------------------------------------| -#| profiles | https://developers.klaviyo.com/en/v2023-02-22/reference/get_profiles | M | Yes, shared with global_exclusions | No as `step` is not defined in `incremental_sync` | 2 | With other streams (global_exclusions), not within stream as `step` not defined | -#| global_exclusions | https://developers.klaviyo.com/en/v2023-02-22/reference/get_profiles | M | Yes, shared with profiles | No as `step` is not defined in `incremental_sync` | 2 | With other streams (profiles), not within stream as `step` not defined | -#| events | https://developers.klaviyo.com/en/reference/get_events | XL | Yes, shared with events_detailed | Yes | number of steps for events + number of steps for events_detailed | With other streams (events_detailed) and within stream as sliced on `datetime` | -#| events_detailed | https://developers.klaviyo.com/en/reference/get_events | XL | Yes, shared with events | Yes | number of steps for events + number of steps for events_detailed | With other streams (events) and within stream as sliced on `datetime` | -#| email_templates | https://developers.klaviyo.com/en/reference/get_templates | M | None | No as `step` is not defined in `incremental_sync` | 1 | None | -#| metrics | https://developers.klaviyo.com/en/reference/get_metrics | M | None | No as `step` is not defined in `incremental_sync` | 1 | None | -#| lists | https://developers.klaviyo.com/en/reference/get_lists | L | Yes, shared with lists_detailed | No as `step` is not defined in `incremental_sync` | 2 | With other streams (lists_detailed), not within stream as `step` not defined | -#| lists_detailed | https://developers.klaviyo.com/en/reference/get_lists | 1/s | Yes, shared with lists | No as `step` is not defined in `incremental_sync` | 1 | With other streams (lists), not within stream as `step` not defined | -# Note: As of 2024-11-11, `metrics`, `lists` and `lists_detailed` are not supported by the Concurrent CDK as they do client side-filtering. - -# Based on the above, the only threads that allow for slicing and hence might perform more concurrent HTTP requests are `events` and `events_detailed`. There are no slicing for the others and hence the concurrency is limited by the number of streams querying the same endpoint. Given that the event endpoint is XL, we will set a default concurrency to 10. -concurrency_level: - type: ConcurrencyLevel - default_concurrency: "{{ config.get('num_workers', 25) }}" - max_concurrency: 50 diff --git a/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/run.py b/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/run.py deleted file mode 100644 index 54eda7f3b7cd6..0000000000000 --- a/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/run.py +++ /dev/null @@ -1,52 +0,0 @@ -# -# Copyright (c) 2023 Airbyte, Inc., all rights reserved. -# - - -import sys -import time -import traceback -from typing import List - -from orjson import orjson - -from airbyte_cdk.entrypoint import AirbyteEntrypoint, launch -from airbyte_cdk.models import AirbyteErrorTraceMessage, AirbyteMessage, AirbyteMessageSerializer, AirbyteTraceMessage, TraceType, Type -from source_klaviyo import SourceKlaviyo - - -def _get_source(args: List[str]): - catalog_path = AirbyteEntrypoint.extract_catalog(args) - config_path = AirbyteEntrypoint.extract_config(args) - state_path = AirbyteEntrypoint.extract_state(args) - try: - return SourceKlaviyo( - SourceKlaviyo.read_catalog(catalog_path) if catalog_path else None, - SourceKlaviyo.read_config(config_path) if config_path else None, - SourceKlaviyo.read_state(state_path) if state_path else None, - ) - except Exception as error: - print( - orjson.dumps( - AirbyteMessageSerializer.dump( - AirbyteMessage( - type=Type.TRACE, - trace=AirbyteTraceMessage( - type=TraceType.ERROR, - emitted_at=time.time_ns() // 1_000_000, - error=AirbyteErrorTraceMessage( - message=f"Error starting the sync. This could be due to an invalid configuration or catalog. Please contact Support for assistance. Error: {error}", - stack_trace=traceback.format_exc(), - ), - ), - ) - ) - ).decode() - ) - raise - - -def run() -> None: - args = sys.argv[1:] - source = _get_source(args) - launch(source, args) diff --git a/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/schemas/campaigns.json b/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/schemas/campaigns.json deleted file mode 100644 index 9156e2ee10d4f..0000000000000 --- a/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/schemas/campaigns.json +++ /dev/null @@ -1,8 +0,0 @@ -{ - "$schema": "http://json-schema.org/draft-07/schema#", - "type": "object", - "additionalProperties": true, - "properties": { - "$ref": "campaign_properties.json" - } -} diff --git a/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/schemas/campaigns_detailed.json b/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/schemas/campaigns_detailed.json deleted file mode 100644 index 6099be206ed72..0000000000000 --- a/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/schemas/campaigns_detailed.json +++ /dev/null @@ -1,56 +0,0 @@ -{ - "$schema": "http://json-schema.org/draft-07/schema#", - "type": "object", - "additionalProperties": true, - "properties": { - "$ref": "campaign_properties.json", - "estimated_recipient_count": { "type": ["null", "integer"] }, - "campaign_message": { - "type": ["null", "object"], - "properties": { - "type": { "type": "string" }, - "id": { "type": "string" }, - "attributes": { - "type": ["null", "object"], - "properties": { - "label": { "type": ["null", "string"] }, - "channel": { "type": ["null", "string"] }, - "content": { - "type": ["null", "object"], - "properties": { - "subject": { "type": ["null", "string"] }, - "preview_text": { "type": ["null", "string"] }, - "from_email": { "type": ["null", "string"] }, - "from_label": { "type": ["null", "string"] }, - "template_id": { "type": ["null", "string"] }, - "template_name": { "type": ["null", "string"] } - } - }, - "send_times": { - "type": ["null", "array"], - "items": { - "type": ["null", "object"], - "properties": { - "datetime": { - "type": ["null", "string"], - "format": "date-time" - }, - "is_local": { "type": ["null", "boolean"] } - } - } - }, - "created_at": { "type": ["null", "string"], "format": "date-time" }, - "updated_at": { "type": ["null", "string"], "format": "date-time" }, - "campaign_id": { "type": ["null", "string"] } - } - }, - "links": { - "type": ["null", "object"], - "properties": { - "self": { "type": "string" } - } - } - } - } - } -} diff --git a/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/schemas/flows.json b/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/schemas/flows.json deleted file mode 100644 index c5623ab0712c3..0000000000000 --- a/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/schemas/flows.json +++ /dev/null @@ -1,80 +0,0 @@ -{ - "$schema": "http://json-schema.org/draft-07/schema#", - "type": "object", - "additionalProperties": true, - "properties": { - "type": { "type": "string" }, - "id": { "type": "string" }, - "updated": { "type": "string", "format": "date-time" }, - "attributes": { - "type": ["null", "object"], - "additionalProperties": true, - "properties": { - "name": { "type": "string" }, - "status": { "type": "string" }, - "archived": { "type": "boolean" }, - "created": { "type": "string", "format": "date-time" }, - "updated": { "type": "string", "format": "date-time" }, - "trigger_type": { "type": "string" } - } - }, - "links": { - "type": ["null", "object"], - "additionalProperties": true, - "properties": { - "self": { "type": "string" } - } - }, - "relationships": { - "type": ["null", "object"], - "additionalProperties": true, - "properties": { - "flow-actions": { - "type": ["null", "object"], - "properties": { - "data": { - "type": "array", - "items": { - "type": ["null", "object"], - "additionalProperties": true, - "properties": { - "type": { "type": "string" }, - "id": { "type": "string" } - } - } - }, - "links": { - "type": ["null", "object"], - "properties": { - "self": { "type": "string" }, - "related": { "type": "string" } - } - } - } - }, - "tags": { - "type": ["null", "object"], - "properties": { - "data": { - "type": "array", - "items": { - "type": ["null", "object"], - "properties": { - "type": { "type": "string" }, - "id": { "type": "string" } - } - } - }, - "links": { - "type": ["null", "object"], - "properties": { - "self": { "type": "string" }, - "related": { "type": "string" } - } - } - } - } - } - } - } -} diff --git a/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/schemas/shared/campaign_properties.json b/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/schemas/shared/campaign_properties.json deleted file mode 100644 index 70d45311538e3..0000000000000 --- a/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/schemas/shared/campaign_properties.json +++ /dev/null @@ -1,137 +0,0 @@ -{ - "type": { "type": "string" }, - "id": { "type": "string" }, - "updated_at": { "type": ["null", "string"], "format": "date-time" }, - "attributes": { - "type": ["null", "object"], - "additionalProperties": true, - "properties": { - "name": { "type": "string" }, - "status": { "type": "string" }, - "archived": { "type": "boolean" }, - "channel": { "type": "string" }, - "audiences": { - "type": ["null", "object"], - "additionalProperties": true, - "properties": { - "included": { - "type": ["null", "array"], - "items": { - "type": ["null", "string"] - } - }, - "excluded": { - "type": ["null", "array"], - "items": { - "type": ["null", "string"] - } - } - } - }, - "send_options": { - "type": ["null", "object"], - "properties": { - "ignore_unsubscribes": { "type": ["null", "boolean"] }, - "use_smart_sending": { "type": ["null", "boolean"] } - } - }, - "message": { "type": "string" }, - "tracking_options": { - "type": ["null", "object"], - "additionalProperties": true, - "properties": { - "is_tracking_opens": { "type": ["null", "boolean"] }, - "is_tracking_clicks": { "type": ["null", "boolean"] }, - "is_add_utm": { "type": ["null", "boolean"] }, - "utm_params": { - "type": ["null", "array"], - "items": { - "type": ["null", "object"], - "properties": { - "name": { "type": "string" }, - "value": { "type": "string" } - } - } - } - } - }, - "send_strategy": { - "type": ["null", "object"], - "additionalProperties": true, - "properties": { - "method": { "type": "string" }, - "options_static": { - "type": ["null", "object"], - "properties": { - "datetime": { - "type": "string", - "format": "date-time", - "airbyte_type": "timestamp_without_timezone" - }, - "is_local": { "type": ["null", "boolean"] }, - "send_past_recipients_immediately": { - "type": ["null", "boolean"] - } - } - }, - "options_throttled": { - "type": ["null", "object"], - "properties": { - "datetime": { - "type": "string", - "format": "date-time", - "airbyte_type": "timestamp_without_timezone" - }, - "throttle_percentage": { "type": "integer" } - } - }, - "options_sto": { - "type": ["null", "object"], - "properties": { - "date": { "type": "string", "format": "date" } - } - } - } - }, - "created_at": { "type": ["null", "string"], "format": "date-time" }, - "scheduled_at": { "type": ["null", "string"], "format": "date-time" }, - "updated_at": { "type": ["null", "string"], "format": "date-time" }, - "send_time": { "type": ["null", "string"], "format": "date-time" } - } - }, - "links": { - "type": ["null", "object"], - "additionalProperties": true, - "properties": { - "self": { "type": "string" } - } - }, - "relationships": { - "type": ["null", "object"], - "additionalProperties": true, - "properties": { - "tags": { - "type": ["null", "object"], - "properties": { - "data": { - "type": "array", - "items": { - "type": ["null", "object"], - "properties": { - "type": { "type": "string" }, - "id": { "type": "string" } - } - } - }, - "links": { - "type": ["null", "object"], - "properties": { - "self": { "type": "string" }, - "related": { "type": "string" } - } - } - } - } - } - } -} diff --git a/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/source.py b/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/source.py deleted file mode 100644 index a37ef5802b3b9..0000000000000 --- a/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/source.py +++ /dev/null @@ -1,15 +0,0 @@ -# -# Copyright (c) 2023 Airbyte, Inc., all rights reserved. -# - - -from typing import Any, List, Mapping, Optional - -from airbyte_cdk import TState -from airbyte_cdk.models import ConfiguredAirbyteCatalog -from airbyte_cdk.sources.declarative.yaml_declarative_source import YamlDeclarativeSource - - -class SourceKlaviyo(YamlDeclarativeSource): - def __init__(self, catalog: Optional[ConfiguredAirbyteCatalog], config: Optional[Mapping[str, Any]], state: TState, **kwargs): - super().__init__(catalog=catalog, config=config, state=state, **{"path_to_yaml": "manifest.yaml"}) diff --git a/airbyte-integrations/connectors/source-klaviyo/unit_tests/conftest.py b/airbyte-integrations/connectors/source-klaviyo/unit_tests/conftest.py new file mode 100644 index 0000000000000..d3826f66680cb --- /dev/null +++ b/airbyte-integrations/connectors/source-klaviyo/unit_tests/conftest.py @@ -0,0 +1,3 @@ +# Copyright (c) 2024 Airbyte, Inc., all rights reserved. + +pytest_plugins = ["airbyte_cdk.test.utils.manifest_only_fixtures"] diff --git a/airbyte-integrations/connectors/source-klaviyo/unit_tests/integration/__init__.py b/airbyte-integrations/connectors/source-klaviyo/unit_tests/integration/__init__.py deleted file mode 100644 index e69de29bb2d1d..0000000000000 diff --git a/airbyte-integrations/connectors/source-klaviyo/unit_tests/integration/config.py b/airbyte-integrations/connectors/source-klaviyo/unit_tests/integration/config.py deleted file mode 100644 index a217637afe7d8..0000000000000 --- a/airbyte-integrations/connectors/source-klaviyo/unit_tests/integration/config.py +++ /dev/null @@ -1,15 +0,0 @@ -# Copyright (c) 2024 Airbyte, Inc., all rights reserved. -from datetime import datetime -from typing import Any, Dict - - -class KlaviyoConfigBuilder: - def __init__(self) -> None: - self._config = {"api_key": "an_api_key", "start_date": "2021-01-01T00:00:00Z"} - - def with_start_date(self, start_date: datetime) -> "KlaviyoConfigBuilder": - self._config["start_date"] = start_date.strftime("%Y-%m-%dT%H:%M:%SZ") - return self - - def build(self) -> Dict[str, Any]: - return self._config diff --git a/airbyte-integrations/connectors/source-klaviyo/unit_tests/integration/test_profiles.py b/airbyte-integrations/connectors/source-klaviyo/unit_tests/integration/test_profiles.py deleted file mode 100644 index 2395a79040fa4..0000000000000 --- a/airbyte-integrations/connectors/source-klaviyo/unit_tests/integration/test_profiles.py +++ /dev/null @@ -1,96 +0,0 @@ -# Copyright (c) 2023 Airbyte, Inc., all rights reserved. -import datetime -from typing import Any, Dict, Optional -from unittest import TestCase - -from source_klaviyo import SourceKlaviyo - -from airbyte_cdk.models import ConfiguredAirbyteCatalog, SyncMode -from airbyte_cdk.test.catalog_builder import CatalogBuilder -from airbyte_cdk.test.entrypoint_wrapper import EntrypointOutput, read -from airbyte_cdk.test.mock_http import HttpMocker, HttpRequest -from airbyte_cdk.test.mock_http.response_builder import ( - FieldPath, - HttpResponseBuilder, - NestedPath, - RecordBuilder, - create_record_builder, - create_response_builder, - find_template, -) -from integration.config import KlaviyoConfigBuilder - - -_ENDPOINT_TEMPLATE_NAME = "profiles" -_START_DATE = datetime.datetime(2021, 1, 1, tzinfo=datetime.timezone.utc) -_STREAM_NAME = "profiles" -_RECORDS_PATH = FieldPath("data") - - -def _config() -> KlaviyoConfigBuilder: - return KlaviyoConfigBuilder().with_start_date(_START_DATE) - - -def _catalog(sync_mode: SyncMode) -> ConfiguredAirbyteCatalog: - return CatalogBuilder().with_stream(_STREAM_NAME, sync_mode).build() - - -def _a_profile_request(start_date: datetime) -> HttpRequest: - return HttpRequest( - url=f"https://a.klaviyo.com/api/profiles", - query_params={ - "additional-fields[profile]": "predictive_analytics", - "page[size]": "100", - "filter": f"greater-than(updated,{start_date.strftime('%Y-%m-%dT%H:%M:%S%z')})", - "sort": "updated", - }, - ) - - -def _a_profile() -> RecordBuilder: - return create_record_builder( - find_template(_ENDPOINT_TEMPLATE_NAME, __file__), - _RECORDS_PATH, - record_id_path=FieldPath("id"), - record_cursor_path=NestedPath(["attributes", "updated"]), - ) - - -def _profiles_response() -> HttpResponseBuilder: - return create_response_builder( - find_template(_ENDPOINT_TEMPLATE_NAME, __file__), - _RECORDS_PATH, - ) - - -def _read( - config_builder: KlaviyoConfigBuilder, sync_mode: SyncMode, state: Optional[Dict[str, Any]] = None, expecting_exception: bool = False -) -> EntrypointOutput: - catalog = _catalog(sync_mode) - config = config_builder.build() - return read(SourceKlaviyo(catalog, config, state), config, catalog, state, expecting_exception) - - -class FullRefreshTest(TestCase): - @HttpMocker() - def test_when_read_then_extract_records(self, http_mocker: HttpMocker) -> None: - http_mocker.get( - _a_profile_request(_START_DATE), - _profiles_response().with_record(_a_profile()).build(), - ) - - output = _read(_config(), SyncMode.full_refresh) - - assert len(output.records) == 1 - - @HttpMocker() - def test_given_region_is_number_when_read_then_cast_as_string(self, http_mocker: HttpMocker) -> None: - http_mocker.get( - _a_profile_request(_START_DATE), - _profiles_response().with_record(_a_profile().with_field(NestedPath(["attributes", "location", "region"]), 10)).build(), - ) - - output = _read(_config(), SyncMode.full_refresh) - - assert len(output.records) == 1 - assert isinstance(output.records[0].record.data["attributes"]["location"]["region"], str) diff --git a/airbyte-integrations/connectors/source-klaviyo/poetry.lock b/airbyte-integrations/connectors/source-klaviyo/unit_tests/poetry.lock similarity index 96% rename from airbyte-integrations/connectors/source-klaviyo/poetry.lock rename to airbyte-integrations/connectors/source-klaviyo/unit_tests/poetry.lock index 331d589d97b68..26b893949a6f1 100644 --- a/airbyte-integrations/connectors/source-klaviyo/poetry.lock +++ b/airbyte-integrations/connectors/source-klaviyo/unit_tests/poetry.lock @@ -1,14 +1,14 @@ -# This file is automatically @generated by Poetry 1.8.4 and should not be changed by hand. +# This file is automatically @generated by Poetry 1.8.3 and should not be changed by hand. [[package]] name = "airbyte-cdk" -version = "6.33.7" +version = "6.34.0" description = "A framework for writing Airbyte Connectors." optional = false python-versions = "<3.13,>=3.10" files = [ - {file = "airbyte_cdk-6.33.7-py3-none-any.whl", hash = "sha256:28ff2cd13c37c56ae62308a86d915570514e01528170cabfd32c7a66ac76002a"}, - {file = "airbyte_cdk-6.33.7.tar.gz", hash = "sha256:e38d721ebff63f6817dcb98799104ae3b22b0f72148de49401429679cdd70ac6"}, + {file = "airbyte_cdk-6.34.0-py3-none-any.whl", hash = "sha256:4740021a8c10cd808cbee096a4c11c69da811bc2a89dd9388702fd81b775b004"}, + {file = "airbyte_cdk-6.34.0.tar.gz", hash = "sha256:3208c23b41288e22f649d546b1f1346d37069e0194c69b83d07bbcf841cac715"}, ] [package.dependencies] @@ -94,16 +94,6 @@ doc = ["Sphinx (>=7.4,<8.0)", "packaging", "sphinx-autodoc-typehints (>=1.2.0)", test = ["anyio[trio]", "coverage[toml] (>=7)", "exceptiongroup (>=1.2.0)", "hypothesis (>=4.0)", "psutil (>=5.9)", "pytest (>=7.0)", "trustme", "truststore (>=0.9.1)", "uvloop (>=0.21)"] trio = ["trio (>=0.26.1)"] -[[package]] -name = "atomicwrites" -version = "1.4.1" -description = "Atomic file writes." -optional = false -python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*" -files = [ - {file = "atomicwrites-1.4.1.tar.gz", hash = "sha256:81b2c9071a49367a7f770170e5eec8cb66567cfbbc8c73d20ce5ca4a8d71cf11"}, -] - [[package]] name = "attributes-doc" version = "0.4.0" @@ -497,20 +487,6 @@ files = [ [package.extras] test = ["pytest (>=6)"] -[[package]] -name = "freezegun" -version = "1.5.1" -description = "Let your Python tests travel through time" -optional = false -python-versions = ">=3.7" -files = [ - {file = "freezegun-1.5.1-py3-none-any.whl", hash = "sha256:bf111d7138a8abe55ab48a71755673dbaa4ab87f4cff5634a4442dfec34c15f1"}, - {file = "freezegun-1.5.1.tar.gz", hash = "sha256:b29dedfcda6d5e8e083ce71b2b542753ad48cfec44037b3fc79702e2980a89e9"}, -] - -[package.dependencies] -python-dateutil = ">=2.7" - [[package]] name = "genson" version = "1.3.0" @@ -736,7 +712,10 @@ files = [ [package.dependencies] httpx = ">=0.23.0,<1" orjson = {version = ">=3.9.14,<4.0.0", markers = "platform_python_implementation != \"PyPy\""} -pydantic = {version = ">=1,<3", markers = "python_full_version < \"3.12.4\""} +pydantic = [ + {version = ">=1,<3", markers = "python_full_version < \"3.12.4\""}, + {version = ">=2.7.4,<3.0.0", markers = "python_full_version >= \"3.12.4\""}, +] requests = ">=2,<3" requests-toolbelt = ">=1.0.0,<2.0.0" @@ -1024,6 +1003,7 @@ files = [ numpy = [ {version = ">=1.22.4", markers = "python_version < \"3.11\""}, {version = ">=1.23.2", markers = "python_version == \"3.11\""}, + {version = ">=1.26.0", markers = "python_version >= \"3.12\""}, ] python-dateutil = ">=2.8.2" pytz = ">=2020.1" @@ -1054,40 +1034,6 @@ sql-other = ["SQLAlchemy (>=2.0.0)", "adbc-driver-postgresql (>=0.8.0)", "adbc-d test = ["hypothesis (>=6.46.1)", "pytest (>=7.3.2)", "pytest-xdist (>=2.2.0)"] xml = ["lxml (>=4.9.2)"] -[[package]] -name = "pendulum" -version = "2.1.2" -description = "Python datetimes made easy" -optional = false -python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*" -files = [ - {file = "pendulum-2.1.2-cp27-cp27m-macosx_10_15_x86_64.whl", hash = "sha256:b6c352f4bd32dff1ea7066bd31ad0f71f8d8100b9ff709fb343f3b86cee43efe"}, - {file = "pendulum-2.1.2-cp27-cp27m-win_amd64.whl", hash = "sha256:318f72f62e8e23cd6660dbafe1e346950281a9aed144b5c596b2ddabc1d19739"}, - {file = "pendulum-2.1.2-cp35-cp35m-macosx_10_15_x86_64.whl", hash = "sha256:0731f0c661a3cb779d398803655494893c9f581f6488048b3fb629c2342b5394"}, - {file = "pendulum-2.1.2-cp35-cp35m-manylinux1_i686.whl", hash = "sha256:3481fad1dc3f6f6738bd575a951d3c15d4b4ce7c82dce37cf8ac1483fde6e8b0"}, - {file = "pendulum-2.1.2-cp35-cp35m-manylinux1_x86_64.whl", hash = "sha256:9702069c694306297ed362ce7e3c1ef8404ac8ede39f9b28b7c1a7ad8c3959e3"}, - {file = "pendulum-2.1.2-cp35-cp35m-win_amd64.whl", hash = "sha256:fb53ffa0085002ddd43b6ca61a7b34f2d4d7c3ed66f931fe599e1a531b42af9b"}, - {file = "pendulum-2.1.2-cp36-cp36m-macosx_10_15_x86_64.whl", hash = "sha256:c501749fdd3d6f9e726086bf0cd4437281ed47e7bca132ddb522f86a1645d360"}, - {file = "pendulum-2.1.2-cp36-cp36m-manylinux1_i686.whl", hash = "sha256:c807a578a532eeb226150d5006f156632df2cc8c5693d778324b43ff8c515dd0"}, - {file = "pendulum-2.1.2-cp36-cp36m-manylinux1_x86_64.whl", hash = "sha256:2d1619a721df661e506eff8db8614016f0720ac171fe80dda1333ee44e684087"}, - {file = "pendulum-2.1.2-cp36-cp36m-win_amd64.whl", hash = "sha256:f888f2d2909a414680a29ae74d0592758f2b9fcdee3549887779cd4055e975db"}, - {file = "pendulum-2.1.2-cp37-cp37m-macosx_10_15_x86_64.whl", hash = "sha256:e95d329384717c7bf627bf27e204bc3b15c8238fa8d9d9781d93712776c14002"}, - {file = "pendulum-2.1.2-cp37-cp37m-manylinux1_i686.whl", hash = "sha256:4c9c689747f39d0d02a9f94fcee737b34a5773803a64a5fdb046ee9cac7442c5"}, - {file = "pendulum-2.1.2-cp37-cp37m-manylinux1_x86_64.whl", hash = "sha256:1245cd0075a3c6d889f581f6325dd8404aca5884dea7223a5566c38aab94642b"}, - {file = "pendulum-2.1.2-cp37-cp37m-win_amd64.whl", hash = "sha256:db0a40d8bcd27b4fb46676e8eb3c732c67a5a5e6bfab8927028224fbced0b40b"}, - {file = "pendulum-2.1.2-cp38-cp38-macosx_10_15_x86_64.whl", hash = "sha256:f5e236e7730cab1644e1b87aca3d2ff3e375a608542e90fe25685dae46310116"}, - {file = "pendulum-2.1.2-cp38-cp38-manylinux1_i686.whl", hash = "sha256:de42ea3e2943171a9e95141f2eecf972480636e8e484ccffaf1e833929e9e052"}, - {file = "pendulum-2.1.2-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:7c5ec650cb4bec4c63a89a0242cc8c3cebcec92fcfe937c417ba18277d8560be"}, - {file = "pendulum-2.1.2-cp38-cp38-win_amd64.whl", hash = "sha256:33fb61601083f3eb1d15edeb45274f73c63b3c44a8524703dc143f4212bf3269"}, - {file = "pendulum-2.1.2-cp39-cp39-manylinux1_i686.whl", hash = "sha256:29c40a6f2942376185728c9a0347d7c0f07905638c83007e1d262781f1e6953a"}, - {file = "pendulum-2.1.2-cp39-cp39-manylinux1_x86_64.whl", hash = "sha256:94b1fc947bfe38579b28e1cccb36f7e28a15e841f30384b5ad6c5e31055c85d7"}, - {file = "pendulum-2.1.2.tar.gz", hash = "sha256:b06a0ca1bfe41c990bbf0c029f0b6501a7f2ec4e38bfec730712015e8860f207"}, -] - -[package.dependencies] -python-dateutil = ">=2.6,<3.0" -pytzdata = ">=2020.1" - [[package]] name = "platformdirs" version = "4.3.6" @@ -1149,17 +1095,6 @@ files = [ dev = ["black", "check-manifest", "coverage", "packaging", "pylint", "pyperf", "pypinfo", "pytest-cov", "requests", "rstcheck", "ruff", "sphinx", "sphinx_rtd_theme", "toml-sort", "twine", "virtualenv", "wheel"] test = ["pytest", "pytest-xdist", "setuptools"] -[[package]] -name = "py" -version = "1.11.0" -description = "library with cross-python path, ini-parsing, io, code, log facilities" -optional = false -python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*" -files = [ - {file = "py-1.11.0-py2.py3-none-any.whl", hash = "sha256:607c53218732647dff4acdfcd50cb62615cedf612e72d1724fb1a0cc6405b378"}, - {file = "py-1.11.0.tar.gz", hash = "sha256:51c75c4126074b472f746a24399ad32f6053d1b34b68d2fa41e558e6f4a98719"}, -] - [[package]] name = "pycparser" version = "2.22" @@ -1378,44 +1313,25 @@ files = [ [[package]] name = "pytest" -version = "6.2.5" +version = "8.3.4" description = "pytest: simple powerful testing with Python" optional = false -python-versions = ">=3.6" +python-versions = ">=3.8" files = [ - {file = "pytest-6.2.5-py3-none-any.whl", hash = "sha256:7310f8d27bc79ced999e760ca304d69f6ba6c6649c0b60fb0e04a4a77cacc134"}, - {file = "pytest-6.2.5.tar.gz", hash = "sha256:131b36680866a76e6781d13f101efb86cf674ebb9762eb70d3082b6f29889e89"}, + {file = "pytest-8.3.4-py3-none-any.whl", hash = "sha256:50e16d954148559c9a74109af1eaf0c945ba2d8f30f0a3d3335edde19788b6f6"}, + {file = "pytest-8.3.4.tar.gz", hash = "sha256:965370d062bce11e73868e0335abac31b4d3de0e82f4007408d242b4f8610761"}, ] [package.dependencies] -atomicwrites = {version = ">=1.0", markers = "sys_platform == \"win32\""} -attrs = ">=19.2.0" colorama = {version = "*", markers = "sys_platform == \"win32\""} +exceptiongroup = {version = ">=1.0.0rc8", markers = "python_version < \"3.11\""} iniconfig = "*" packaging = "*" -pluggy = ">=0.12,<2.0" -py = ">=1.8.2" -toml = "*" - -[package.extras] -testing = ["argcomplete", "hypothesis (>=3.56)", "mock", "nose", "requests", "xmlschema"] - -[[package]] -name = "pytest-mock" -version = "3.14.0" -description = "Thin-wrapper around the mock package for easier use with pytest" -optional = false -python-versions = ">=3.8" -files = [ - {file = "pytest-mock-3.14.0.tar.gz", hash = "sha256:2719255a1efeceadbc056d6bf3df3d1c5015530fb40cf347c0f9afac88410bd0"}, - {file = "pytest_mock-3.14.0-py3-none-any.whl", hash = "sha256:0b72c38033392a5f4621342fe11e9219ac11ec9d375f8e2a0c164539e0d70f6f"}, -] - -[package.dependencies] -pytest = ">=6.2.5" +pluggy = ">=1.5,<2" +tomli = {version = ">=1", markers = "python_version < \"3.11\""} [package.extras] -dev = ["pre-commit", "pytest-asyncio", "tox"] +dev = ["argcomplete", "attrs (>=19.2)", "hypothesis (>=3.56)", "mock", "pygments (>=2.7.2)", "requests", "setuptools", "xmlschema"] [[package]] name = "python-dateutil" @@ -1456,17 +1372,6 @@ files = [ {file = "pytz-2024.2.tar.gz", hash = "sha256:2aa355083c50a0f93fa581709deac0c9ad65cca8a9e9beac660adcbd493c798a"}, ] -[[package]] -name = "pytzdata" -version = "2020.1" -description = "The Olson timezone database for Python." -optional = false -python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*" -files = [ - {file = "pytzdata-2020.1-py2.py3-none-any.whl", hash = "sha256:e1e14750bcf95016381e4d472bad004eef710f2d6417240904070b3d6654485f"}, - {file = "pytzdata-2020.1.tar.gz", hash = "sha256:3efa13b335a00a8de1d345ae41ec78dd11c9f8807f522d39850f2dd828681540"}, -] - [[package]] name = "pyyaml" version = "6.0.2" @@ -1906,14 +1811,44 @@ doc = ["reno", "sphinx"] test = ["pytest", "tornado (>=4.5)", "typeguard"] [[package]] -name = "toml" -version = "0.10.2" -description = "Python Library for Tom's Obvious, Minimal Language" +name = "tomli" +version = "2.2.1" +description = "A lil' TOML parser" optional = false -python-versions = ">=2.6, !=3.0.*, !=3.1.*, !=3.2.*" +python-versions = ">=3.8" files = [ - {file = "toml-0.10.2-py2.py3-none-any.whl", hash = "sha256:806143ae5bfb6a3c6e736a764057db0e6a0e05e338b5630894a5f779cabb4f9b"}, - {file = "toml-0.10.2.tar.gz", hash = "sha256:b3bda1d108d5dd99f4a20d24d9c348e91c4db7ab1b749200bded2f839ccbe68f"}, + {file = "tomli-2.2.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:678e4fa69e4575eb77d103de3df8a895e1591b48e740211bd1067378c69e8249"}, + {file = "tomli-2.2.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:023aa114dd824ade0100497eb2318602af309e5a55595f76b626d6d9f3b7b0a6"}, + {file = "tomli-2.2.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ece47d672db52ac607a3d9599a9d48dcb2f2f735c6c2d1f34130085bb12b112a"}, + {file = "tomli-2.2.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:6972ca9c9cc9f0acaa56a8ca1ff51e7af152a9f87fb64623e31d5c83700080ee"}, + {file = "tomli-2.2.1-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:c954d2250168d28797dd4e3ac5cf812a406cd5a92674ee4c8f123c889786aa8e"}, + {file = "tomli-2.2.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:8dd28b3e155b80f4d54beb40a441d366adcfe740969820caf156c019fb5c7ec4"}, + {file = "tomli-2.2.1-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:e59e304978767a54663af13c07b3d1af22ddee3bb2fb0618ca1593e4f593a106"}, + {file = "tomli-2.2.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:33580bccab0338d00994d7f16f4c4ec25b776af3ffaac1ed74e0b3fc95e885a8"}, + {file = "tomli-2.2.1-cp311-cp311-win32.whl", hash = "sha256:465af0e0875402f1d226519c9904f37254b3045fc5084697cefb9bdde1ff99ff"}, + {file = "tomli-2.2.1-cp311-cp311-win_amd64.whl", hash = "sha256:2d0f2fdd22b02c6d81637a3c95f8cd77f995846af7414c5c4b8d0545afa1bc4b"}, + {file = "tomli-2.2.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:4a8f6e44de52d5e6c657c9fe83b562f5f4256d8ebbfe4ff922c495620a7f6cea"}, + {file = "tomli-2.2.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:8d57ca8095a641b8237d5b079147646153d22552f1c637fd3ba7f4b0b29167a8"}, + {file = "tomli-2.2.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4e340144ad7ae1533cb897d406382b4b6fede8890a03738ff1683af800d54192"}, + {file = "tomli-2.2.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:db2b95f9de79181805df90bedc5a5ab4c165e6ec3fe99f970d0e302f384ad222"}, + {file = "tomli-2.2.1-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:40741994320b232529c802f8bc86da4e1aa9f413db394617b9a256ae0f9a7f77"}, + {file = "tomli-2.2.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:400e720fe168c0f8521520190686ef8ef033fb19fc493da09779e592861b78c6"}, + {file = "tomli-2.2.1-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:02abe224de6ae62c19f090f68da4e27b10af2b93213d36cf44e6e1c5abd19fdd"}, + {file = "tomli-2.2.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:b82ebccc8c8a36f2094e969560a1b836758481f3dc360ce9a3277c65f374285e"}, + {file = "tomli-2.2.1-cp312-cp312-win32.whl", hash = "sha256:889f80ef92701b9dbb224e49ec87c645ce5df3fa2cc548664eb8a25e03127a98"}, + {file = "tomli-2.2.1-cp312-cp312-win_amd64.whl", hash = "sha256:7fc04e92e1d624a4a63c76474610238576942d6b8950a2d7f908a340494e67e4"}, + {file = "tomli-2.2.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:f4039b9cbc3048b2416cc57ab3bda989a6fcf9b36cf8937f01a6e731b64f80d7"}, + {file = "tomli-2.2.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:286f0ca2ffeeb5b9bd4fcc8d6c330534323ec51b2f52da063b11c502da16f30c"}, + {file = "tomli-2.2.1-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a92ef1a44547e894e2a17d24e7557a5e85a9e1d0048b0b5e7541f76c5032cb13"}, + {file = "tomli-2.2.1-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9316dc65bed1684c9a98ee68759ceaed29d229e985297003e494aa825ebb0281"}, + {file = "tomli-2.2.1-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:e85e99945e688e32d5a35c1ff38ed0b3f41f43fad8df0bdf79f72b2ba7bc5272"}, + {file = "tomli-2.2.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:ac065718db92ca818f8d6141b5f66369833d4a80a9d74435a268c52bdfa73140"}, + {file = "tomli-2.2.1-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:d920f33822747519673ee656a4b6ac33e382eca9d331c87770faa3eef562aeb2"}, + {file = "tomli-2.2.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:a198f10c4d1b1375d7687bc25294306e551bf1abfa4eace6650070a5c1ae2744"}, + {file = "tomli-2.2.1-cp313-cp313-win32.whl", hash = "sha256:d3f5614314d758649ab2ab3a62d4f2004c825922f9e370b29416484086b264ec"}, + {file = "tomli-2.2.1-cp313-cp313-win_amd64.whl", hash = "sha256:a38aa0308e754b0e3c67e344754dff64999ff9b513e691d0e786265c93583c69"}, + {file = "tomli-2.2.1-py3-none-any.whl", hash = "sha256:cb55c73c5f4408779d0cf3eef9f762b9c9f147a77de7b258bef0a5628adc85cc"}, + {file = "tomli-2.2.1.tar.gz", hash = "sha256:cd45e1dc79c835ce60f7404ec8119f2eb06d38b1deba146f07ced3bbc44505ff"}, ] [[package]] @@ -2111,5 +2046,5 @@ files = [ [metadata] lock-version = "2.0" -python-versions = "^3.10,<3.12" -content-hash = "2982e20fc923aa1516f081137731b723f75086fc9991d93b369e214abda2c4f3" +python-versions = "^3.10,<3.13" +content-hash = "1a2584c3bdbd78cb1ec046246cab9d361315441c98b94cb70ec39fe91d4b9a70" diff --git a/airbyte-integrations/connectors/source-klaviyo/unit_tests/pyproject.toml b/airbyte-integrations/connectors/source-klaviyo/unit_tests/pyproject.toml new file mode 100644 index 0000000000000..cd46a2f334a32 --- /dev/null +++ b/airbyte-integrations/connectors/source-klaviyo/unit_tests/pyproject.toml @@ -0,0 +1,21 @@ +[build-system] +requires = [ "poetry-core>=1.0.0",] +build-backend = "poetry.core.masonry.api" + +[tool.poetry] +name = "source-klaviyo-tests" +version = "2.14.0" +description = "Unit tests for source-klaviyo" +authors = ["Airbyte "] + +[tool.poetry.dependencies] +python = "^3.10,<3.13" +airbyte-cdk = "^6" +pytest = "^8" +requests-mock = "*" + +[tool.pytest.ini_options] +filterwarnings = [ + "ignore:This class is experimental*" +] + diff --git a/airbyte-integrations/connectors/source-klaviyo/unit_tests/resource/http/response/profiles.json b/airbyte-integrations/connectors/source-klaviyo/unit_tests/resource/http/response/profiles.json deleted file mode 100644 index 42a3553ec9b9c..0000000000000 --- a/airbyte-integrations/connectors/source-klaviyo/unit_tests/resource/http/response/profiles.json +++ /dev/null @@ -1,94 +0,0 @@ -{ - "data": [ - { - "type": "profile", - "id": "01G4CDTEP140TDXZ0692XSA61K", - "attributes": { - "email": "tracey.witting@developer-tools.shopifyapps.com", - "phone_number": "+13523772689", - "external_id": null, - "anonymous_id": null, - "first_name": "Tracey", - "last_name": "Witting", - "organization": null, - "locale": null, - "title": null, - "image": null, - "created": "2022-05-31T06:46:01+00:00", - "updated": "2022-05-31T06:46:01+00:00", - "last_event_date": null, - "location": { - "address1": "2088 Rempel Road", - "region": null, - "zip": null, - "country": "United States", - "address2": null, - "longitude": null, - "city": null, - "latitude": null, - "timezone": null, - "ip": null - }, - "properties": { - "Accepts Marketing": false, - "Shopify Tags": ["developer-tools-generator"] - }, - "subscriptions": { - "email": { - "marketing": { - "consent": "NEVER_SUBSCRIBED", - "timestamp": null, - "method": null, - "method_detail": null, - "custom_method_detail": null, - "double_optin": null, - "suppressions": [], - "list_suppressions": [] - } - }, - "sms": { - "marketing": { - "consent": "NEVER_SUBSCRIBED", - "timestamp": null, - "method": null, - "method_detail": null - } - } - }, - "predictive_analytics": { - "historic_clv": null, - "predicted_clv": null, - "total_clv": null, - "historic_number_of_orders": null, - "predicted_number_of_orders": null, - "average_days_between_orders": null, - "average_order_value": null, - "churn_probability": null, - "expected_date_of_next_order": null - } - }, - "relationships": { - "lists": { - "links": { - "self": "https://a.klaviyo.com/api/profiles/01G4CDTEP140TDXZ0692XSA61K/relationships/lists/", - "related": "https://a.klaviyo.com/api/profiles/01G4CDTEP140TDXZ0692XSA61K/lists/" - } - }, - "segments": { - "links": { - "self": "https://a.klaviyo.com/api/profiles/01G4CDTEP140TDXZ0692XSA61K/relationships/segments/", - "related": "https://a.klaviyo.com/api/profiles/01G4CDTEP140TDXZ0692XSA61K/segments/" - } - } - }, - "links": { - "self": "https://a.klaviyo.com/api/profiles/01G4CDTEP140TDXZ0692XSA61K/" - } - } - ], - "links": { - "self": "https://a.klaviyo.com/api/profiles?additional-fields%5Bprofile%5D=predictive_analytics&page%5Bsize%5D=100&filter=greater-than%28updated%2C2021-01-01T00%3A00%3A00%2B0000%29&sort=updated&page%5Bcursor%5D=bmV4dDo6dXBkYXRlZDo6MjAyMi0wNS0zMSAwNjo0NjowMSswMDowMDo6aWQ6OjAxRzRDRFRFTTZKMjBHRzVRNU41Q0Q4V0pW", - "next": null, - "prev": null - } -} diff --git a/airbyte-integrations/connectors/source-klaviyo/unit_tests/test_campaign_detailed_transformation.py b/airbyte-integrations/connectors/source-klaviyo/unit_tests/test_campaign_detailed_transformation.py deleted file mode 100644 index adb74ab3e80ad..0000000000000 --- a/airbyte-integrations/connectors/source-klaviyo/unit_tests/test_campaign_detailed_transformation.py +++ /dev/null @@ -1,90 +0,0 @@ -# Copyright (c) 2024 Airbyte, Inc., all rights reserved. - -from source_klaviyo.components.campaign_details_transformations import CampaignsDetailedTransformation - - -def test_transform(requests_mock): - config = {"api_key": "api_key"} - transformator = CampaignsDetailedTransformation(config=config) - input_record = { - "id": "campaign_id", - "relationships": {"campaign-messages": {"links": {"related": "https://a.klaviyo.com/api/related_link"}}}, - } - - requests_mock.register_uri( - "GET", - f"https://a.klaviyo.com/api/campaign-recipient-estimations/{input_record['id']}", - status_code=200, - json={"data": {"attributes": {"estimated_recipient_count": 10}}}, - complete_qs=True, - ) - requests_mock.register_uri( - "GET", - input_record["relationships"]["campaign-messages"]["links"]["related"], - status_code=200, - json={"data": [{"attributes": {"field": "field"}}]}, - complete_qs=True, - ) - - transformator.transform(input_record) - - assert "campaign_messages" in input_record - assert "estimated_recipient_count" in input_record - - -def test_transform_not_campaign_messages(requests_mock): - config = {"api_key": "api_key"} - transformator = CampaignsDetailedTransformation(config=config) - input_record = { - "id": "campaign_id", - "relationships": {"campaign-messages": {"links": {"related": "https://a.klaviyo.com/api/related_link"}}}, - } - - requests_mock.register_uri( - "GET", - f"https://a.klaviyo.com/api/campaign-recipient-estimations/{input_record['id']}", - status_code=200, - json={"data": {"attributes": {"estimated_recipient_count": 10}}}, - complete_qs=True, - ) - requests_mock.register_uri( - "GET", - input_record["relationships"]["campaign-messages"]["links"]["related"], - status_code=200, - json={}, - complete_qs=True, - ) - - transformator.transform(input_record) - - assert "campaign_messages" in input_record - assert "estimated_recipient_count" in input_record - - -def test_transform_not_estimated_recipient_count(requests_mock): - config = {"api_key": "api_key"} - transformator = CampaignsDetailedTransformation(config=config) - input_record = { - "id": "campaign_id", - "relationships": {"campaign-messages": {"links": {"related": "https://a.klaviyo.com/api/related_link"}}}, - } - - requests_mock.register_uri( - "GET", - f"https://a.klaviyo.com/api/campaign-recipient-estimations/{input_record['id']}", - status_code=200, - json={"data": {"attributes": {}}}, - complete_qs=True, - ) - requests_mock.register_uri( - "GET", - input_record["relationships"]["campaign-messages"]["links"]["related"], - status_code=200, - json={"data": [{"attributes": {"field": "field"}}]}, - complete_qs=True, - ) - - transformator.transform(input_record) - - assert "campaign_messages" in input_record - assert "estimated_recipient_count" in input_record diff --git a/airbyte-integrations/connectors/source-klaviyo/unit_tests/test_campaigns_state_migration.py b/airbyte-integrations/connectors/source-klaviyo/unit_tests/test_campaigns_state_migration.py deleted file mode 100644 index 086648c90ad51..0000000000000 --- a/airbyte-integrations/connectors/source-klaviyo/unit_tests/test_campaigns_state_migration.py +++ /dev/null @@ -1,117 +0,0 @@ -# Copyright (c) 2024 Airbyte, Inc., all rights reserved. - -from unittest.mock import MagicMock - -import pytest -from source_klaviyo.components.archived_to_per_partition_state_migration import ( - ArchivedToPerPartitionStateMigration, - CampaignsStateMigration, -) - - -@pytest.mark.parametrize( - ("state", "should_migrate"), - ( - ({"updated_at": "2120-10-10T00:00:00+00:00", "archived": {"updated_at": "2020-10-10T00:00:00+00:00"}}, True), - ({"updated_at": "2120-10-10T00:00:00+00:00"}, True), - ({}, False), - ( - { - "states": [ - {"partition": {"archived": "true", "campaign_type": "sms"}, "cursor": {"updated_at": "2023-10-10T00:00:00+0000"}}, - {"partition": {"archived": "false", "campaign_type": "sms"}, "cursor": {"updated_at": "2023-10-10T00:00:00+0000"}}, - {"partition": {"archived": "true", "campaign_type": "email"}, "cursor": {"updated_at": "2023-10-10T00:00:00+0000"}}, - {"partition": {"archived": "false", "campaign_type": "email"}, "cursor": {"updated_at": "2023-10-10T00:00:00+0000"}}, - ] - }, - False, - ), - ), -) -def test_should_migrate(state, should_migrate): - config = {} - declarative_stream = MagicMock() - state_migrator = ArchivedToPerPartitionStateMigration(config=config, declarative_stream=declarative_stream) - assert state_migrator.should_migrate(state) == should_migrate - - -@pytest.mark.parametrize( - ("state", "expected_state"), - ( - ( - {"updated_at": "2120-10-10T00:00:00+00:00", "archived": {"updated_at": "2020-10-10T00:00:00+00:00"}}, - { - "states": [ - {"cursor": {"updated_at": "2020-10-10T00:00:00+00:00"}, "partition": {"archived": "true"}}, - {"cursor": {"updated_at": "2120-10-10T00:00:00+00:00"}, "partition": {"archived": "false"}}, - ] - }, - ), - ( - {"archived": {"updated_at": "2020-10-10T00:00:00+00:00"}}, - { - "states": [ - {"cursor": {"updated_at": "2020-10-10T00:00:00+00:00"}, "partition": {"archived": "true"}}, - {"cursor": {"updated_at": "2012-01-01T00:00:00Z"}, "partition": {"archived": "false"}}, - ] - }, - ), - ( - {"updated_at": "2120-10-10T00:00:00+00:00"}, - { - "states": [ - {"cursor": {"updated_at": "2012-01-01T00:00:00Z"}, "partition": {"archived": "true"}}, - {"cursor": {"updated_at": "2120-10-10T00:00:00+00:00"}, "partition": {"archived": "false"}}, - ] - }, - ), - ), -) -def test_migrate(state, expected_state): - config = {} - declarative_stream = MagicMock() - declarative_stream.incremental_sync.cursor_field = "updated_at" - state_migrator = ArchivedToPerPartitionStateMigration(config=config, declarative_stream=declarative_stream) - assert state_migrator.migrate(state) == expected_state - - -@pytest.mark.parametrize( - ("state", "expected_state"), - ( - ( - {"updated_at": "2120-10-10T00:00:00+00:00", "archived": {"updated_at": "2020-10-10T00:00:00+00:00"}}, - { - "states": [ - {"cursor": {"updated_at": "2020-10-10T00:00:00+00:00"}, "partition": {"archived": "true", "campaign_type": "email"}}, - {"cursor": {"updated_at": "2120-10-10T00:00:00+00:00"}, "partition": {"archived": "false", "campaign_type": "email"}}, - ] - }, - ), - ( - {"archived": {"updated_at": "2020-10-10T00:00:00+00:00"}}, - { - "states": [ - {"cursor": {"updated_at": "2020-10-10T00:00:00+00:00"}, "partition": {"archived": "true", "campaign_type": "email"}}, - {"cursor": {"updated_at": "2012-01-01T00:00:00Z"}, "partition": {"archived": "false", "campaign_type": "email"}}, - ] - }, - ), - ( - { - "updated_at": "2120-10-10T00:00:00+00:00", - }, - { - "states": [ - {"cursor": {"updated_at": "2012-01-01T00:00:00Z"}, "partition": {"archived": "true", "campaign_type": "email"}}, - {"cursor": {"updated_at": "2120-10-10T00:00:00+00:00"}, "partition": {"archived": "false", "campaign_type": "email"}}, - ] - }, - ), - ), -) -def test_migrate_campaigns(state, expected_state): - config = {} - declarative_stream = MagicMock() - declarative_stream.incremental_sync.cursor_field = "updated_at" - state_migrator = CampaignsStateMigration(config=config, declarative_stream=declarative_stream) - assert state_migrator.migrate(state) == expected_state diff --git a/airbyte-integrations/connectors/source-klaviyo/unit_tests/test_components.py b/airbyte-integrations/connectors/source-klaviyo/unit_tests/test_components.py new file mode 100644 index 0000000000000..40210e1410583 --- /dev/null +++ b/airbyte-integrations/connectors/source-klaviyo/unit_tests/test_components.py @@ -0,0 +1,321 @@ +# Copyright (c) 2024 Airbyte, Inc., all rights reserved. + +from unittest.mock import MagicMock, Mock, patch + +import pytest +from requests.models import Response + +from airbyte_cdk.sources.declarative.models import ( + CustomRetriever, + DatetimeBasedCursor, + DeclarativeStream, + ParentStreamConfig, + SubstreamPartitionRouter, +) +from airbyte_cdk.sources.declarative.parsers.manifest_component_transformer import ManifestComponentTransformer +from airbyte_cdk.sources.declarative.parsers.manifest_reference_resolver import ManifestReferenceResolver +from airbyte_cdk.sources.declarative.parsers.model_to_component_factory import ModelToComponentFactory + + +factory = ModelToComponentFactory() +resolver = ManifestReferenceResolver() +transformer = ManifestComponentTransformer() + + +def test_transform(components_module, requests_mock): + config = {"api_key": "api_key"} + transformator = components_module.CampaignsDetailedTransformation(config=config) + input_record = { + "id": "campaign_id", + "relationships": {"campaign-messages": {"links": {"related": "https://a.klaviyo.com/api/related_link"}}}, + } + + requests_mock.register_uri( + "GET", + f"https://a.klaviyo.com/api/campaign-recipient-estimations/{input_record['id']}", + status_code=200, + json={"data": {"attributes": {"estimated_recipient_count": 10}}}, + complete_qs=True, + ) + requests_mock.register_uri( + "GET", + input_record["relationships"]["campaign-messages"]["links"]["related"], + status_code=200, + json={"data": [{"attributes": {"field": "field"}}]}, + complete_qs=True, + ) + + transformator.transform(input_record) + + assert "campaign_messages" in input_record + assert "estimated_recipient_count" in input_record + + +def test_transform_not_campaign_messages(components_module, requests_mock): + config = {"api_key": "api_key"} + transformator = components_module.CampaignsDetailedTransformation(config=config) + input_record = { + "id": "campaign_id", + "relationships": {"campaign-messages": {"links": {"related": "https://a.klaviyo.com/api/related_link"}}}, + } + + requests_mock.register_uri( + "GET", + f"https://a.klaviyo.com/api/campaign-recipient-estimations/{input_record['id']}", + status_code=200, + json={"data": {"attributes": {"estimated_recipient_count": 10}}}, + complete_qs=True, + ) + requests_mock.register_uri( + "GET", + input_record["relationships"]["campaign-messages"]["links"]["related"], + status_code=200, + json={}, + complete_qs=True, + ) + + transformator.transform(input_record) + + assert "campaign_messages" in input_record + assert "estimated_recipient_count" in input_record + + +def test_transform_not_estimated_recipient_count(components_module, requests_mock): + config = {"api_key": "api_key"} + transformator = components_module.CampaignsDetailedTransformation(config=config) + input_record = { + "id": "campaign_id", + "relationships": {"campaign-messages": {"links": {"related": "https://a.klaviyo.com/api/related_link"}}}, + } + + requests_mock.register_uri( + "GET", + f"https://a.klaviyo.com/api/campaign-recipient-estimations/{input_record['id']}", + status_code=200, + json={"data": {"attributes": {}}}, + complete_qs=True, + ) + requests_mock.register_uri( + "GET", + input_record["relationships"]["campaign-messages"]["links"]["related"], + status_code=200, + json={"data": [{"attributes": {"field": "field"}}]}, + complete_qs=True, + ) + + transformator.transform(input_record) + + assert "campaign_messages" in input_record + assert "estimated_recipient_count" in input_record + + +@pytest.mark.parametrize( + ("state", "should_migrate"), + ( + ({"updated_at": "2120-10-10T00:00:00+00:00", "archived": {"updated_at": "2020-10-10T00:00:00+00:00"}}, True), + ({"updated_at": "2120-10-10T00:00:00+00:00"}, True), + ({}, False), + ( + { + "states": [ + {"partition": {"archived": "true", "campaign_type": "sms"}, "cursor": {"updated_at": "2023-10-10T00:00:00+0000"}}, + {"partition": {"archived": "false", "campaign_type": "sms"}, "cursor": {"updated_at": "2023-10-10T00:00:00+0000"}}, + {"partition": {"archived": "true", "campaign_type": "email"}, "cursor": {"updated_at": "2023-10-10T00:00:00+0000"}}, + {"partition": {"archived": "false", "campaign_type": "email"}, "cursor": {"updated_at": "2023-10-10T00:00:00+0000"}}, + ] + }, + False, + ), + ), +) +def test_should_migrate(components_module, state, should_migrate): + config = {} + declarative_stream = MagicMock() + state_migrator = components_module.ArchivedToPerPartitionStateMigration(config=config, declarative_stream=declarative_stream) + assert state_migrator.should_migrate(state) == should_migrate + + +@pytest.mark.parametrize( + ("state", "expected_state"), + ( + ( + {"updated_at": "2120-10-10T00:00:00+00:00", "archived": {"updated_at": "2020-10-10T00:00:00+00:00"}}, + { + "states": [ + {"cursor": {"updated_at": "2020-10-10T00:00:00+00:00"}, "partition": {"archived": "true"}}, + {"cursor": {"updated_at": "2120-10-10T00:00:00+00:00"}, "partition": {"archived": "false"}}, + ] + }, + ), + ( + {"archived": {"updated_at": "2020-10-10T00:00:00+00:00"}}, + { + "states": [ + {"cursor": {"updated_at": "2020-10-10T00:00:00+00:00"}, "partition": {"archived": "true"}}, + {"cursor": {"updated_at": "2012-01-01T00:00:00Z"}, "partition": {"archived": "false"}}, + ] + }, + ), + ( + {"updated_at": "2120-10-10T00:00:00+00:00"}, + { + "states": [ + {"cursor": {"updated_at": "2012-01-01T00:00:00Z"}, "partition": {"archived": "true"}}, + {"cursor": {"updated_at": "2120-10-10T00:00:00+00:00"}, "partition": {"archived": "false"}}, + ] + }, + ), + ), +) +def test_migrate(components_module, state, expected_state): + config = {} + declarative_stream = MagicMock() + declarative_stream.incremental_sync.cursor_field = "updated_at" + state_migrator = components_module.ArchivedToPerPartitionStateMigration(config=config, declarative_stream=declarative_stream) + assert state_migrator.migrate(state) == expected_state + + +@pytest.mark.parametrize( + ("state", "expected_state"), + ( + ( + {"updated_at": "2120-10-10T00:00:00+00:00", "archived": {"updated_at": "2020-10-10T00:00:00+00:00"}}, + { + "states": [ + {"cursor": {"updated_at": "2020-10-10T00:00:00+00:00"}, "partition": {"archived": "true", "campaign_type": "email"}}, + {"cursor": {"updated_at": "2120-10-10T00:00:00+00:00"}, "partition": {"archived": "false", "campaign_type": "email"}}, + ] + }, + ), + ( + {"archived": {"updated_at": "2020-10-10T00:00:00+00:00"}}, + { + "states": [ + {"cursor": {"updated_at": "2020-10-10T00:00:00+00:00"}, "partition": {"archived": "true", "campaign_type": "email"}}, + {"cursor": {"updated_at": "2012-01-01T00:00:00Z"}, "partition": {"archived": "false", "campaign_type": "email"}}, + ] + }, + ), + ( + { + "updated_at": "2120-10-10T00:00:00+00:00", + }, + { + "states": [ + {"cursor": {"updated_at": "2012-01-01T00:00:00Z"}, "partition": {"archived": "true", "campaign_type": "email"}}, + {"cursor": {"updated_at": "2120-10-10T00:00:00+00:00"}, "partition": {"archived": "false", "campaign_type": "email"}}, + ] + }, + ), + ), +) +def test_migrate_campaigns(components_module, state, expected_state): + config = {} + declarative_stream = MagicMock() + declarative_stream.incremental_sync.cursor_field = "updated_at" + state_migrator = components_module.CampaignsStateMigration(config=config, declarative_stream=declarative_stream) + assert state_migrator.migrate(state) == expected_state + + +@pytest.fixture +def mock_response(): + return Mock(spec=Response) + + +@pytest.fixture +def mock_decoder(): + return Mock() + + +@pytest.fixture +def mock_config(): + return Mock() + + +@pytest.fixture +def mock_field_path(): + return [Mock() for _ in range(2)] + + +@pytest.fixture +def extractor(components_module, mock_config, mock_field_path, mock_decoder): + return components_module.KlaviyoIncludedFieldExtractor(mock_field_path, mock_config, mock_decoder) + + +@patch("dpath.get") +@patch("dpath.values") +def test_extract_records_by_path(mock_values, mock_get, extractor, mock_response, mock_decoder): + mock_values.return_value = [{"key": "value"}] + mock_get.return_value = {"key": "value"} + mock_decoder.decode.return_value = {"data": "value"} + + field_paths = ["data"] + records = list(extractor.extract_records_by_path(mock_response, field_paths)) + assert records == [{"key": "value"}] + + mock_values.return_value = [] + mock_get.return_value = None + records = list(extractor.extract_records_by_path(mock_response, ["included"])) + assert records == [] + + +def test_update_target_records_with_included(extractor): + target_records = [{"relationships": {"type1": {"data": {"id": 1}}}}] + included_records = [{"id": 1, "type": "type1", "attributes": {"key": "value"}}] + + updated_records = list(extractor.update_target_records_with_included(target_records, included_records)) + assert updated_records[0]["relationships"]["type1"]["data"] == {"id": 1, "key": "value"} + + +def test_migrate_a_valid_legacy_state_to_per_partition(components_module): + input_state = { + "states": [ + {"partition": {"parent_id": "13506132"}, "cursor": {"last_changed": "2023-12-27T08:34:39+00:00"}}, + {"partition": {"parent_id": "14351124"}, "cursor": {"last_changed": "2022-12-27T08:35:39+00:00"}}, + ] + } + + migrator = _migrator(components_module) + + assert migrator.should_migrate(input_state) + + expected_state = {"last_changed": "2022-12-27T08:35:39+00:00"} + + assert migrator.migrate(input_state) == expected_state + + +def test_should_not_migrate(components_module): + input_state = {"last_changed": "2022-12-27T08:35:39+00:00"} + migrator = _migrator(components_module) + assert not migrator.should_migrate(input_state) + + +def _migrator(components_module): + partition_router = SubstreamPartitionRouter( + type="SubstreamPartitionRouter", + parent_stream_configs=[ + ParentStreamConfig( + type="ParentStreamConfig", + parent_key="{{ parameters['parent_key_id'] }}", + partition_field="parent_id", + stream=DeclarativeStream( + type="DeclarativeStream", retriever=CustomRetriever(type="CustomRetriever", class_name="a_class_name") + ), + ) + ], + ) + cursor = DatetimeBasedCursor( + type="DatetimeBasedCursor", + cursor_field="{{ parameters['cursor_field'] }}", + datetime_format="%Y-%m-%dT%H:%M:%S.%fZ", + start_datetime="1970-01-01T00:00:00.0Z", + ) + config = {} + parameters = {"cursor_field": "last_changed", "parent_key_id": "id"} + + declarative_stream = MagicMock() + declarative_stream.retriever.partition_router = partition_router + declarative_stream.incremental_sync = cursor + declarative_stream.parameters = parameters + + return components_module.PerPartitionToSingleStateMigration(config=config, declarative_stream=declarative_stream) diff --git a/airbyte-integrations/connectors/source-klaviyo/unit_tests/test_included_extractor.py b/airbyte-integrations/connectors/source-klaviyo/unit_tests/test_included_extractor.py deleted file mode 100644 index df84e66f80295..0000000000000 --- a/airbyte-integrations/connectors/source-klaviyo/unit_tests/test_included_extractor.py +++ /dev/null @@ -1,59 +0,0 @@ -# -# Copyright (c) 2023 Airbyte, Inc., all rights reserved. -# - -from unittest.mock import Mock, patch - -import pytest -from requests.models import Response -from source_klaviyo.components.included_fields_extractor import KlaviyoIncludedFieldExtractor - - -@pytest.fixture -def mock_response(): - return Mock(spec=Response) - - -@pytest.fixture -def mock_decoder(): - return Mock() - - -@pytest.fixture -def mock_config(): - return Mock() - - -@pytest.fixture -def mock_field_path(): - return [Mock() for _ in range(2)] - - -@pytest.fixture -def extractor(mock_config, mock_field_path, mock_decoder): - return KlaviyoIncludedFieldExtractor(mock_field_path, mock_config, mock_decoder) - - -@patch("dpath.get") -@patch("dpath.values") -def test_extract_records_by_path(mock_values, mock_get, extractor, mock_response, mock_decoder): - mock_values.return_value = [{"key": "value"}] - mock_get.return_value = {"key": "value"} - mock_decoder.decode.return_value = {"data": "value"} - - field_paths = ["data"] - records = list(extractor.extract_records_by_path(mock_response, field_paths)) - assert records == [{"key": "value"}] - - mock_values.return_value = [] - mock_get.return_value = None - records = list(extractor.extract_records_by_path(mock_response, ["included"])) - assert records == [] - - -def test_update_target_records_with_included(extractor): - target_records = [{"relationships": {"type1": {"data": {"id": 1}}}}] - included_records = [{"id": 1, "type": "type1", "attributes": {"key": "value"}}] - - updated_records = list(extractor.update_target_records_with_included(target_records, included_records)) - assert updated_records[0]["relationships"]["type1"]["data"] == {"id": 1, "key": "value"} diff --git a/airbyte-integrations/connectors/source-klaviyo/unit_tests/test_per_partition_state_migration.py b/airbyte-integrations/connectors/source-klaviyo/unit_tests/test_per_partition_state_migration.py deleted file mode 100644 index 7cf3138d22f90..0000000000000 --- a/airbyte-integrations/connectors/source-klaviyo/unit_tests/test_per_partition_state_migration.py +++ /dev/null @@ -1,79 +0,0 @@ -# -# Copyright (c) 2024 Airbyte, Inc., all rights reserved. -# - -from unittest.mock import MagicMock - -from source_klaviyo.components.per_partition_state_migration import PerPartitionToSingleStateMigration - -from airbyte_cdk.sources.declarative.models import ( - CustomRetriever, - DatetimeBasedCursor, - DeclarativeStream, - ParentStreamConfig, - SubstreamPartitionRouter, -) -from airbyte_cdk.sources.declarative.parsers.manifest_component_transformer import ManifestComponentTransformer -from airbyte_cdk.sources.declarative.parsers.manifest_reference_resolver import ManifestReferenceResolver -from airbyte_cdk.sources.declarative.parsers.model_to_component_factory import ModelToComponentFactory - - -factory = ModelToComponentFactory() - -resolver = ManifestReferenceResolver() - -transformer = ManifestComponentTransformer() - - -def test_migrate_a_valid_legacy_state_to_per_partition(): - input_state = { - "states": [ - {"partition": {"parent_id": "13506132"}, "cursor": {"last_changed": "2023-12-27T08:34:39+00:00"}}, - {"partition": {"parent_id": "14351124"}, "cursor": {"last_changed": "2022-12-27T08:35:39+00:00"}}, - ] - } - - migrator = _migrator() - - assert migrator.should_migrate(input_state) - - expected_state = {"last_changed": "2022-12-27T08:35:39+00:00"} - - assert migrator.migrate(input_state) == expected_state - - -def test_should_not_migrate(): - input_state = {"last_changed": "2022-12-27T08:35:39+00:00"} - migrator = _migrator() - assert not migrator.should_migrate(input_state) - - -def _migrator(): - partition_router = SubstreamPartitionRouter( - type="SubstreamPartitionRouter", - parent_stream_configs=[ - ParentStreamConfig( - type="ParentStreamConfig", - parent_key="{{ parameters['parent_key_id'] }}", - partition_field="parent_id", - stream=DeclarativeStream( - type="DeclarativeStream", retriever=CustomRetriever(type="CustomRetriever", class_name="a_class_name") - ), - ) - ], - ) - cursor = DatetimeBasedCursor( - type="DatetimeBasedCursor", - cursor_field="{{ parameters['cursor_field'] }}", - datetime_format="%Y-%m-%dT%H:%M:%S.%fZ", - start_datetime="1970-01-01T00:00:00.0Z", - ) - config = {} - parameters = {"cursor_field": "last_changed", "parent_key_id": "id"} - - declarative_stream = MagicMock() - declarative_stream.retriever.partition_router = partition_router - declarative_stream.incremental_sync = cursor - declarative_stream.parameters = parameters - - return PerPartitionToSingleStateMigration(config=config, declarative_stream=declarative_stream) diff --git a/airbyte-integrations/connectors/source-klaviyo/unit_tests/test_source.py b/airbyte-integrations/connectors/source-klaviyo/unit_tests/test_source.py deleted file mode 100644 index 17cc35eb63524..0000000000000 --- a/airbyte-integrations/connectors/source-klaviyo/unit_tests/test_source.py +++ /dev/null @@ -1,73 +0,0 @@ -# -# Copyright (c) 2023 Airbyte, Inc., all rights reserved. -# - - -import logging - -import pendulum -import pytest -from integration.config import KlaviyoConfigBuilder -from source_klaviyo.source import SourceKlaviyo - -from airbyte_cdk.test.catalog_builder import CatalogBuilder -from airbyte_cdk.test.state_builder import StateBuilder - - -logger = logging.getLogger("airbyte") - - -def _source() -> SourceKlaviyo: - catalog = CatalogBuilder().build() - config = KlaviyoConfigBuilder().build() - state = StateBuilder().build() - return SourceKlaviyo(catalog, config, state) - - -@pytest.mark.parametrize( - ("status_code", "is_connection_successful", "error_msg"), - ( - (200, True, None), - ( - 400, - False, - ("Bad request. Please check your request parameters."), - ), - ( - 403, - False, - ("Please provide a valid API key and make sure it has permissions to read specified streams."), - ), - ), -) -def test_check_connection(requests_mock, status_code, is_connection_successful, error_msg): - requests_mock.register_uri( - "GET", - "https://a.klaviyo.com/api/metrics", - status_code=status_code, - json={"end": 1, "total": 1} if 200 >= status_code < 300 else {}, - ) - source = _source() - success, error = source.check_connection(logger=logger, config={"api_key": "api_key"}) - assert success is is_connection_successful - assert error == error_msg - - -def test_check_connection_unexpected_error(requests_mock): - exception_info = "Something went wrong" - requests_mock.register_uri("GET", "https://a.klaviyo.com/api/metrics", exc=Exception(exception_info)) - source = _source() - success, error = source.check_connection(logger=logger, config={"api_key": "api_key"}) - assert success is False - assert error == f"Unable to connect to stream metrics - {exception_info}" - - -def test_streams(): - source = _source() - config = {"api_key": "some_key", "start_date": pendulum.datetime(2020, 10, 10).isoformat()} - streams = source.streams(config) - expected_streams_number = 11 - assert len(streams) == expected_streams_number - - # ensure only unique stream names are returned - assert len({stream.name for stream in streams}) == expected_streams_number diff --git a/airbyte-integrations/connectors/source-klaviyo/unit_tests/test_streams.py b/airbyte-integrations/connectors/source-klaviyo/unit_tests/test_streams.py deleted file mode 100644 index 017adc32cca58..0000000000000 --- a/airbyte-integrations/connectors/source-klaviyo/unit_tests/test_streams.py +++ /dev/null @@ -1,256 +0,0 @@ -# -# Copyright (c) 2023 Airbyte, Inc., all rights reserved. -# - - -import urllib.parse -from datetime import datetime -from typing import Any, List, Mapping, Optional - -import pendulum -import pytest -import requests -from dateutil.relativedelta import relativedelta -from freezegun import freeze_time -from integration.config import KlaviyoConfigBuilder -from source_klaviyo.source import SourceKlaviyo - -from airbyte_cdk.models import SyncMode -from airbyte_cdk.sources.streams import Stream -from airbyte_cdk.test.catalog_builder import CatalogBuilder, ConfiguredAirbyteStreamBuilder -from airbyte_cdk.test.entrypoint_wrapper import EntrypointOutput, read -from airbyte_cdk.test.state_builder import StateBuilder - - -_ANY_ATTEMPT_COUNT = 123 -API_KEY = "some_key" -START_DATE = pendulum.datetime(2020, 10, 10) -CONFIG = {"api_key": API_KEY, "start_date": START_DATE} -CONFIG_NO_DATE = {"api_key": API_KEY} - -EVENTS_STREAM_DEFAULT_START_DATE = "2012-01-01T00:00:00+00:00" -EVENTS_STREAM_CONFIG_START_DATE = "2021-11-08T00:00:00+00:00" -EVENTS_STREAM_STATE_DATE = (datetime.fromisoformat(EVENTS_STREAM_CONFIG_START_DATE) + relativedelta(years=1)).isoformat() -EVENTS_STREAM_TESTING_FREEZE_TIME = "2023-12-12 12:00:00" - - -def get_step_diff(provided_date: str) -> int: - """ - This function returns the difference in weeks between provided date and freeze time. - """ - provided_date = datetime.fromisoformat(provided_date).replace(tzinfo=None) - freeze_date = datetime.strptime(EVENTS_STREAM_TESTING_FREEZE_TIME, "%Y-%m-%d %H:%M:%S") - return (freeze_date - provided_date).days // 7 - - -def read_records(stream_name: str, config: Mapping[str, Any], states: Mapping[str, Any] = dict()) -> List[Mapping[str, Any]]: - state = StateBuilder() - for stream_name_key in states: - state.with_stream_state(stream_name_key, states[stream_name_key]) - source = SourceKlaviyo(CatalogBuilder().build(), config, state.build()) - output = read( - source, - config, - CatalogBuilder().with_stream(ConfiguredAirbyteStreamBuilder().with_name(stream_name)).build(), - ) - return [r.record.data for r in output.records] - - -def get_stream_by_name(stream_name: str, config: Mapping[str, Any], states: Mapping[str, Any] = dict()) -> Stream: - state = StateBuilder() - for stream_name_key in states: - state.with_stream_state(stream_name_key, states[stream_name_key]) - source = SourceKlaviyo(CatalogBuilder().build(), KlaviyoConfigBuilder().build(), state.build()) - matches_by_name = [stream_config for stream_config in source.streams(config) if stream_config.name == stream_name] - if not matches_by_name: - raise ValueError("Please provide a valid stream name.") - return matches_by_name[0] - - -def get_records(stream: Stream, sync_mode: Optional[SyncMode] = SyncMode.full_refresh) -> List[Mapping[str, Any]]: - records = [] - for stream_slice in stream.stream_slices(sync_mode=sync_mode): - for record in stream.read_records(sync_mode=sync_mode, stream_slice=stream_slice): - records.append(dict(record)) - return records - - -@pytest.fixture(name="response") -def response_fixture(mocker): - return mocker.Mock(spec=requests.Response) - - -class TestSemiIncrementalKlaviyoStream: - @pytest.mark.parametrize( - ("start_date", "stream_state", "input_records", "expected_records"), - ( - ( - "2021-11-08T00:00:00Z", - "2022-11-07T00:00:00+00:00", - [ - {"attributes": {"updated": "2022-11-08T00:00:00+00:00"}}, - {"attributes": {"updated": "2023-11-08T00:00:00+00:00"}}, - {"attributes": {"updated": "2021-11-08T00:00:00+00:00"}}, - ], - [ - {"attributes": {"updated": "2022-11-08T00:00:00+00:00"}, "updated": "2022-11-08T00:00:00+00:00"}, - {"attributes": {"updated": "2023-11-08T00:00:00+00:00"}, "updated": "2023-11-08T00:00:00+00:00"}, - ], - ), - ( - "2021-11-09T00:00:00Z", - None, - [ - {"attributes": {"updated": "2022-11-08T00:00:00+00:00"}}, - {"attributes": {"updated": "2023-11-08T00:00:00+00:00"}}, - {"attributes": {"updated": "2021-11-08T00:00:00+00:00"}}, - ], - [ - {"attributes": {"updated": "2022-11-08T00:00:00+00:00"}, "updated": "2022-11-08T00:00:00+00:00"}, - {"attributes": {"updated": "2023-11-08T00:00:00+00:00"}, "updated": "2023-11-08T00:00:00+00:00"}, - ], - ), - ("2021-11-08T00:00:00Z", "2022-11-07T00:00:00+00:00", [], []), - ), - ) - def test_read_records(self, start_date, stream_state, input_records, expected_records, requests_mock): - state = {"metrics": {"updated": stream_state}} if stream_state else {} - requests_mock.register_uri("GET", f"https://a.klaviyo.com/api/metrics", status_code=200, json={"data": input_records}) - records = read_records("metrics", CONFIG_NO_DATE | {"start_date": start_date}, state) - assert records == expected_records - - -class TestProfilesStream: - def test_read_records(self, requests_mock): - stream = get_stream_by_name("profiles", CONFIG) - json = { - "data": [ - { - "type": "profile", - "id": "00AA0A0AA0AA000AAAAAAA0AA0", - "attributes": {"email": "name@airbyte.io", "updated": "2023-03-10T20:36:36+00:00"}, - "properties": {"Status": "onboarding_complete"}, - }, - { - "type": "profile", - "id": "AAAA1A1AA1AA111AAAAAAA1AA1", - "attributes": {"email": "name2@airbyte.io", "updated": "2023-02-10T20:36:36+00:00"}, - "properties": {"Status": "onboarding_started"}, - }, - ], - } - requests_mock.register_uri("GET", f"https://a.klaviyo.com/api/profiles", status_code=200, json=json) - - records = get_records(stream=stream) - assert records == [ - { - "type": "profile", - "id": "00AA0A0AA0AA000AAAAAAA0AA0", - "updated": "2023-03-10T20:36:36+00:00", - "attributes": {"email": "name@airbyte.io", "updated": "2023-03-10T20:36:36+00:00"}, - "properties": {"Status": "onboarding_complete"}, - }, - { - "type": "profile", - "id": "AAAA1A1AA1AA111AAAAAAA1AA1", - "updated": "2023-02-10T20:36:36+00:00", - "attributes": {"email": "name2@airbyte.io", "updated": "2023-02-10T20:36:36+00:00"}, - "properties": {"Status": "onboarding_started"}, - }, - ] - - -class TestGlobalExclusionsStream: - def test_read_records(self, requests_mock): - stream = get_stream_by_name("global_exclusions", CONFIG) - json = { - "data": [ - { - "type": "profile", - "id": "00AA0A0AA0AA000AAAAAAA0AA0", - "attributes": { - "updated": "2023-03-10T20:36:36+00:00", - "subscriptions": {"email": {"marketing": {"suppression": [{"reason": "SUPPRESSED"}]}}}, - }, - }, - { - "type": "profile", - "id": "AAAA1A1AA1AA111AAAAAAA1AA1", - "attributes": {"updated": "2023-02-10T20:36:36+00:00"}, - }, - ], - } - requests_mock.register_uri("GET", f"https://a.klaviyo.com/api/profiles", status_code=200, json=json) - - records = get_records(stream=stream) - assert records == [ - { - "type": "profile", - "id": "00AA0A0AA0AA000AAAAAAA0AA0", - "attributes": { - "updated": "2023-03-10T20:36:36+00:00", - "subscriptions": {"email": {"marketing": {"suppressions": [{"reason": "SUPPRESSED"}]}}}, - }, - "updated": "2023-03-10T20:36:36+00:00", - } - ] - - -class TestCampaignsStream: - @freeze_time(pendulum.datetime(2020, 11, 10).isoformat()) - def test_read_records(self, requests_mock): - input_records = { - "sms": { - "true": {"attributes": {"name": "Some name 1", "archived": True, "updated_at": "2020-10-21T00:00:00+0000"}}, - "false": {"attributes": {"name": "Some name 1", "archived": False, "updated_at": "2020-10-20T00:00:00+0000"}}, - }, - "email": { - "true": {"attributes": {"name": "Some name 1", "archived": True, "updated_at": "2020-10-18T00:00:00+0000"}}, - "false": {"attributes": {"name": "Some name 1", "archived": False, "updated_at": "2020-10-23T00:00:00+0000"}}, - }, - } - - stream = get_stream_by_name("campaigns", CONFIG) - expected_records = [ - { - "attributes": {"archived": True, "name": "Some name 1", "updated_at": "2020-10-21T00:00:00+0000", "channel": "sms"}, - "updated_at": "2020-10-21T00:00:00+0000", - }, - { - "attributes": {"archived": False, "name": "Some name 1", "updated_at": "2020-10-20T00:00:00+0000", "channel": "sms"}, - "updated_at": "2020-10-20T00:00:00+0000", - }, - { - "attributes": {"archived": True, "name": "Some name 1", "updated_at": "2020-10-18T00:00:00+0000", "channel": "email"}, - "updated_at": "2020-10-18T00:00:00+0000", - }, - { - "attributes": {"archived": False, "name": "Some name 1", "updated_at": "2020-10-23T00:00:00+0000", "channel": "email"}, - "updated_at": "2020-10-23T00:00:00+0000", - }, - ] - - records = [] - base_url = "https://a.klaviyo.com/api/campaigns" - - for stream_slice in stream.stream_slices(sync_mode=SyncMode.full_refresh): - query_params = { - "filter": f"and(greater-or-equal(updated_at,{stream_slice['start_time']}),less-or-equal(updated_at,{stream_slice['end_time']}),equals(messages.channel,'{stream_slice['campaign_type']}'),equals(archived,{stream_slice['archived']}))", - "sort": "updated_at", - } - encoded_query = urllib.parse.urlencode(query_params) - encoded_url = f"{base_url}?{encoded_query}" - requests_mock.register_uri( - "GET", - encoded_url, - status_code=200, - json={"data": input_records[stream_slice["campaign_type"]][stream_slice["archived"]]}, - complete_qs=True, - ) - - for record in stream.read_records(sync_mode=SyncMode.full_refresh, stream_slice=stream_slice): - records.append(record) - - assert len(records) == len(expected_records) - for expected_record, record in zip(expected_records, records): - assert expected_record == dict(record) diff --git a/docs/integrations/sources/klaviyo.md b/docs/integrations/sources/klaviyo.md index 0060840b8b247..8595fd66924a3 100644 --- a/docs/integrations/sources/klaviyo.md +++ b/docs/integrations/sources/klaviyo.md @@ -95,6 +95,7 @@ contain the `predictive_analytics` field and workflows depending on this field w | Version | Date | Pull Request | Subject | |:--------|:-----------|:-----------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| 2.14.0 | 2025-02-19 | [x](https://github.com/airbytehq/airbyte/pull/x) | Migrate to Manifest-only | | 2.13.0 | 2025-02-18 | [51551](https://github.com/airbytehq/airbyte/pull/51551) | Upgrade to API v2024-10-15 | | 2.12.1 | 2025-02-15 | [52710](https://github.com/airbytehq/airbyte/pull/52710) | Update dependencies | | 2.12.0 | 2025-02-11 | [53223](https://github.com/airbytehq/airbyte/pull/53223) | Add API Budget |