✨Source Youtube Analytics - Migrate Python CDK to Low-code CDK to Manifest-only

airbyte-integrations/connectors/source-youtube-analytics/README.md

@@ -1,89 +1,63 @@
-# Youtube-Analytics source connector
+# Youtube analytics source connector
 
+This directory contains the manifest-only connector for `source-youtube-analytics`.
+This _manifest-only_ connector is not a Python package on its own, as it runs inside of the base `source-declarative-manifest` image.
 
-This is the repository for the Youtube-Analytics source connector, written in Python.
-For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.com/integrations/sources/youtube-analytics).
+For information about how to configure and use this connector within Airbyte, see [the connector's full documentation](https://docs.airbyte.com/integrations/sources/youtube-analytics).
 
 ## Local development
 
-### Prerequisites
-* Python (~=3.9)
-* Poetry (~=1.7) - installation instructions [here](https://python-poetry.org/docs/#installation)
+We recommend using the Connector Builder to edit this connector.
+Using either Airbyte Cloud or your local Airbyte OSS instance, navigate to the **Builder** tab and select **Import a YAML**.
+Then select the connector's `manifest.yaml` file to load the connector into the Builder. You're now ready to make changes to the connector!
 
+If you prefer to develop locally, you can follow the instructions below.
 
-### Installing the connector
-From this connector directory, run:
-```bash
-poetry install --with dev
-```
-
-
-### Create credentials
-**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.com/integrations/sources/youtube-analytics)
-to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_youtube_analytics/spec.yaml` file.
-Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
-See `sample_files/sample_config.json` for a sample config file.
-
-
-### Locally running the connector
-```
-poetry run source-youtube-analytics spec
-poetry run source-youtube-analytics check --config secrets/config.json
-poetry run source-youtube-analytics discover --config secrets/config.json
-poetry run source-youtube-analytics read --config secrets/config.json --catalog sample_files/configured_catalog.json
-```
+### Building the docker image
 
-### Running unit tests
-To run unit tests locally, from the connector directory run:
-```
-poetry run pytest unit_tests
-```
+You can build any manifest-only connector with `airbyte-ci`:
 
-### Building the docker image
 1. Install [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md)
 2. Run the following command to build the docker image:
+
 ```bash
 airbyte-ci connectors --name=source-youtube-analytics build
 ```
 
 An image will be available on your host with the tag `airbyte/source-youtube-analytics:dev`.
 
+### Creating credentials
+
+**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.com/integrations/sources/youtube-analytics)
+to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `spec` object in the connector's `manifest.yaml` file.
+Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
 
 ### Running as a docker container
-Then run any of the connector commands as follows:
-```
+
+Then run any of the standard source connector commands:
+
+```bash
 docker run --rm airbyte/source-youtube-analytics:dev spec
 docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-youtube-analytics:dev check --config /secrets/config.json
 docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-youtube-analytics:dev discover --config /secrets/config.json
 docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-youtube-analytics:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
 ```
 
-### Running our CI test suite
-You can run our full test suite locally using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md):
-```bash
-airbyte-ci connectors --name=source-youtube-analytics test
-```
+### Running the CI test suite
 
-### Customizing acceptance Tests
-Customize `acceptance-test-config.yml` file to configure acceptance tests. See [Connector Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/connector-acceptance-tests-reference) for more information.
-If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
+You can run our full test suite locally using [`airbyte-ci`](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md):
 
-### Dependency Management
-All of your dependencies should be managed via Poetry. 
-To add a new dependency, run:
 ```bash
-poetry add <package-name>
+airbyte-ci connectors --name=source-youtube-analytics test
 ```
 
-Please commit the changes to `pyproject.toml` and `poetry.lock` files.
-
 ## Publishing a new version of the connector
-You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
-1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=source-youtube-analytics test`
-2. Bump the connector version (please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors)): 
+
+If you want to contribute changes to `source-youtube-analytics`, here's how you can do that:
+1. Make your changes locally, or load the connector's manifest into Connector Builder and make changes there.
+2. Make sure your changes are passing our test suite with `airbyte-ci connectors --name=source-youtube-analytics test`
+3. Bump the connector version (please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors)):
     - bump the `dockerImageTag` value in in `metadata.yaml`
-    - bump the `version` value in `pyproject.toml`
-3. Make sure the `metadata.yaml` content is up to date.
 4. Make sure the connector documentation and its changelog is up to date (`docs/integrations/sources/youtube-analytics.md`).
 5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention).
 6. Pat yourself on the back for being an awesome contributor.

airbyte-integrations/connectors/source-youtube-analytics/acceptance-test-config.yml

@@ -3,7 +3,7 @@
 connector_image: airbyte/source-youtube-analytics:dev
 tests:
   spec:
-    - spec_path: "source_youtube_analytics/spec.json"
+    - spec_path: "manifest.yaml"
   connection:
     - config_path: "secrets/config.json"
       status: "succeed"

airbyte-integrations/connectors/source-youtube-analytics/components.py

@@ -0,0 +1,148 @@
+#
+# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
+#
+
+from dataclasses import dataclass
+from typing import Any, Mapping, Optional, Union
+
+import requests
+
+from airbyte_cdk.sources.declarative.requesters.error_handlers import DefaultErrorHandler
+from airbyte_cdk.sources.declarative.requesters.http_requester import HttpRequester
+from airbyte_cdk.sources.declarative.requesters.request_options.interpolated_request_options_provider import (
+    InterpolatedRequestOptionsProvider,
+    RequestInput,
+)
+from airbyte_cdk.sources.streams.http.error_handlers import ErrorResolution, ResponseAction
+
+
+@dataclass
+class CreationRequester(HttpRequester):
+    request_body_json: Optional[RequestInput] = None
+
+    def __post_init__(self, parameters: Mapping[str, Any]) -> None:
+        self.request_options_provider = InterpolatedRequestOptionsProvider(
+            request_body_json=self.request_body_json,
+            config=self.config,
+            parameters=parameters or {},
+        )
+        super().__post_init__(parameters)
+
+    def send_request(self, **kwargs):
+        request, jobs_response = self._http_client.send_request(
+            http_method="GET",
+            url=self._join_url(self.get_url_base(), "jobs"),
+            request_kwargs={"stream": self.stream_response},
+        )
+        jobs_list = jobs_response.json().get("jobs", [])
+        stream_name = self.name.split(" - ")[-1]
+        if stream_name in [job["reportTypeId"] for job in jobs_list]:
+            return jobs_response
+        else:
+            return super().send_request(**kwargs)
+
+
+@dataclass
+class PollingRequester(HttpRequester):
+    def send_request(self, **kwargs):
+        jobs_response = super().send_request(**kwargs)
+        jobs_list = jobs_response.json().get("jobs", [])
+        job_resource = [job for job in jobs_list if job["reportTypeId"] == self.name][0]
+        job_id = job_resource["id"]
+
+        request, reports_response = self._http_client.send_request(
+            http_method=self.get_method().value,
+            url=self._join_url(self.get_url_base(), f"jobs/{job_id}/reports"),
+            request_kwargs={"stream": self.stream_response},
+            params={"startTimeAtOrAfter": self.start_time} if self.start_time else {},
+        )
+        return reports_response
+
+
+@dataclass
+class YoutubeAnalyticsErrorHandler(DefaultErrorHandler):
+    def daily_quota_exceeded(self, response: requests.Response):
+        """Response example:
+            {
+              "error": {
+                "code": 429,
+                "message": "Quota exceeded for quota metric 'Free requests' and limit 'Free requests per minute' of service 'youtubereporting.googleapis.com' for consumer 'project_number:863188056127'.",
+                "status": "RESOURCE_EXHAUSTED",
+                "details": [
+                  {
+                    "reason": "RATE_LIMIT_EXCEEDED",
+                    "metadata": {
+                      "consumer": "projects/863188056127",
+                      "quota_limit": "FreeQuotaRequestsPerMinutePerProject",
+                      "quota_limit_value": "60",
+                      "quota_metric": "youtubereporting.googleapis.com/free_quota_requests",
+                      "service": "youtubereporting.googleapis.com",
+                    }
+                  },
+                ]
+              }
+            }
+
+        :param response:
+        :return:
+        """
+        details = response.json().get("error", {}).get("details", [])
+        for detail in details:
+            if detail.get("reason") == "RATE_LIMIT_EXCEEDED":
+                if detail.get("metadata", {}).get("quota_limit") == "FreeQuotaRequestsPerDayPerProject":
+                    return True, f"Exceeded daily quota: {detail.get('metadata', {}).get('quota_limit_value')} reqs/day"
+                break
+        return False, ""
+
+    def should_retry(self, response: requests.Response):
+        """
+        Override to set different conditions for backoff based on the response from the server.
+
+        By default, back off on the following HTTP response statuses:
+         - 500s to handle transient server errors
+         - 429 (Too Many Requests) indicating rate limiting:
+            Different behavior in case of 'RATE_LIMIT_EXCEEDED':
+
+            Requests Per Minute:
+            "message": "Quota exceeded for quota metric 'Free requests' and limit 'Free requests per minute' of service 'youtubereporting.googleapis.com' for consumer 'project_number:863188056127'."
+            "quota_limit": "FreeQuotaRequestsPerMinutePerProject",
+            "quota_limit_value": "60",
+
+            --> use increased retry_factor (30 seconds)
+
+            Requests Per Day:
+            "message": "Quota exceeded for quota metric 'Free requests' and limit 'Free requests per day' of service 'youtubereporting.googleapis.com' for consumer 'project_number:863188056127"
+            "quota_limit": "FreeQuotaRequestsPerDayPerProject
+            "quota_limit_value": "20000",
+
+            --> just throw an error, next scan is reasonable to start only in 1 day.
+        """
+        if 500 <= response.status_code < 600:
+            return True, ""
+
+        if response.status_code == 429 and not self.daily_quota_exceeded(response):
+            return True, ""
+
+        return False, ""
+
+    def interpret_response(self, response_or_exception: Optional[Union[requests.Response, Exception]]) -> ErrorResolution:
+        """
+        Interprets responses and exceptions, providing custom error resolutions for specific exceptions.
+        """
+        if_retry, error_message = self.should_retry(response_or_exception)
+        if if_retry:
+            return ErrorResolution(
+                response_action=ResponseAction.RETRY,
+                error_message=error_message,
+            )
+        return super().interpret_response(response_or_exception)
+
+    def backoff_time(
+        self,
+        response_or_exception: Optional[Union[requests.Response, requests.RequestException]],
+        attempt_count: int = 0,
+    ) -> Optional[float]:
+        """
+        Default FreeQuotaRequestsPerMinutePerProject is 60 reqs/min, so reasonable delay is 30 seconds
+        """
+        return 30