[SDCT-533] Differentiate log storage from CI jobs failure analysis

hugo/content/en/continuous_integration/pipelines/_index.md

@@ -53,7 +53,7 @@ Select your CI provider to set up CI Visibility in Datadog:
 
 | <div style="width:200px"><div> | AWS CodePipeline | Azure Pipelines | Buildkite | CircleCI | Codefresh | GitHub Actions | GitLab | Jenkins | TeamCity | Other CI Providers |
 |---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
-| {{< ci-details title="Logs collection" >}}Retrieval of pipeline or job logs from the CI provider. Logs are displayed on the <strong>Logs</strong> tab in the Pipeline Execution view.{{< /ci-details >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} |  | {{< X >}} | {{< X >}} | {{< X >}} |  |  |
+| {{< ci-details title="Log storage" >}}Storage of pipeline or job logs from the CI provider. Logs are displayed on the <strong>Logs</strong> tab in the Pipeline Execution view.{{< /ci-details >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} |  | {{< X >}} | {{< X >}} | {{< X >}} |  |  |
 | {{< ci-details title="Infrastructure correlation" >}}Correlation of host-level information for the Datadog Agent, CI pipelines, or job runners to CI pipeline execution data.{{< /ci-details >}} |  |  | {{< X >}} |  |  | {{< X >}} | {{< X >}} | {{< X >}} |  |  |
 | {{< ci-details title="Running pipelines" >}}Identification of pipelines executions that are running with associated tracing.{{< /ci-details >}} | {{< X >}} | | | | | {{< X >}} | {{< X >}} | {{< X >}} | | {{< X >}} |
 | {{< ci-details title="Running jobs" >}}Identification of job executions that are running with associated tracing.{{< /ci-details >}} |  |  |  |  |  | {{< X >}} | {{< X >}} |  |  | {{< X >}} |
@@ -65,7 +65,7 @@ Select your CI provider to set up CI Visibility in Datadog:
 
 | <div style="width:200px"><div> | AWS CodePipeline | Azure Pipelines | Buildkite | CircleCI | Codefresh | GitHub Actions | GitLab | Jenkins | TeamCity | Other CI Providers |
 |---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
-| {{< ci-details title="Job failure analysis" >}}Uses LLM models on relevant logs to analyze the root cause of failed CI jobs. <a href="https://docs.datadoghq.com/continuous_integration/guides/use_ci_jobs_failure_analysis/">More info</a>.{{< /ci-details >}} |  | {{< X >}} |  |  |  | {{< X >}} | {{< X >}} | |  |  |
+| {{< ci-details title="Job failure analysis" >}}Uses LLM models on relevant logs to analyze the root cause of failed CI jobs. <a href="https://docs.datadoghq.com/continuous_integration/guides/use_ci_jobs_failure_analysis/">More info</a>.{{< /ci-details >}} |  | {{< X >}} | {{< X >}} | {{< X >}} |  | {{< X >}} | {{< X >}} | |  |  |
 | {{< ci-details title="Critical path" >}}Identification of CI jobs that are on the critical path of the pipeline. <a href="https://docs.datadoghq.com/continuous_integration/guides/identify_highest_impact_jobs_with_critical_path/">More info</a>{{< /ci-details >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} | {{< X >}} |
 | {{< ci-details title="Automatic job retries" >}}Datadog retries failed jobs classified as transient by its AI error model. <a href="https://docs.datadoghq.com/continuous_integration/pipelines/automatic_retries/">More info</a>.{{< /ci-details >}} |  |  |  |  |  | {{< X >}} | {{< X >}} |  |  |  |
 

hugo/content/en/continuous_integration/pipelines/automatic_retries.md

@@ -97,8 +97,8 @@ A failed job is not retried when:
 [1]: /continuous_integration/pipelines/github/
 [2]: /continuous_integration/pipelines/gitlab/
 [3]: /integrations/guide/source-code-integration/
-[4]: /continuous_integration/pipelines/github/#collect-job-logs
-[5]: /continuous_integration/pipelines/gitlab/#collect-job-logs
+[4]: /continuous_integration/pipelines/github/#store-job-logs
+[5]: /continuous_integration/pipelines/gitlab/#store-job-logs
 [6]: /continuous_integration/guides/use_ci_jobs_failure_analysis/
 [7]: https://app.datadoghq.com/ci/settings/ci-cd/repositories
 [8]: /continuous_integration/pipelines/github/#configure-a-github-app

hugo/content/en/continuous_integration/pipelines/azure.md

@@ -29,7 +29,7 @@ Set up CI Visibility for Azure Pipelines to gain real time insights into your CI
 | Pipeline Visibility                             | Platform                            | Definition                                                |
 |-------------------------------------------------|-------------------------------------|-----------------------------------------------------------|
 | [CI jobs failure analysis][21] | CI jobs failure analysis | Use LLM models on relevant logs to analyze the root cause of failed CI jobs. |
-| Logs correlation                                | Logs correlation                    | Correlate pipeline and job spans to logs. Requires [job log collection](#collect-job-logs). |
+| Logs correlation                                | Logs correlation                    | Correlate pipeline and job spans to logs. Requires [job log storage](#store-job-logs). |
 | [Custom tags][10] [and measures at runtime][11] | Custom tags and measures at runtime | Configure [custom tags and measures][6] at runtime.       |
 | [Custom spans][15]                              | Custom spans                        | Configure custom spans for your pipelines.                |
 | [Filter CI Jobs on the critical path][19]       | Filter CI Jobs on the critical path | Filter by jobs on the critical path.                      |
@@ -158,25 +158,21 @@ Example for enabling the hooks in specified projects:
 
 You can set custom tags for all pipeline and job spans from your Azure projects to improve traceability. For more information, see [Custom Tags and Measures][6].
 
-### Collect job logs
+### Manage job logs
 
-To enable log collection for Azure DevOps pipelines:
+CI jobs failure analysis uses LLM models to compute the analysis for failed CI jobs based on relevant logs coming from Azure Pipelines.
 
+You can also add job failure analysis to a PR comment. See the guide on [using PR comments][22].
 
-1. Set up the Datadog Azure DevOps integration by following the steps in the [Azure integration tile][14].
-
-2. In Datadog, open [**CI/CD Optimization** > **Settings** > **Azure DevOps**][23].
-3. Enable log pulling for Azure DevOps.
-
-Logs are billed separately from CI Visibility. Log retention, exclusion, and indexes are configured in [Log Management][18]. Logs for Azure jobs can be identified by the `datadog.product:cipipeline` and `source:azurepipelines` tags.
+For a full explanation, see the guide on [using CI jobs failure analysis][21].
 
-### CI jobs failure analysis
+This feature is enabled by default. To manage Log Analysis settings, go to [CI/CD Repository settings][23], and configure Log Analysis at the Datadog organization level or for the desired repositories.
 
-If job logs collection is enabled, CI Visibility uses LLM models to compute the analysis for failed CI jobs based on relevant logs coming from Azure Pipelines.
+#### Store job logs
 
-You can also add job failure analysis to a PR comment. See the guide on [using PR comments][22].
+To enable Log Storage, go to [CI/CD Repository settings][23], and enable Log Storage at the Datadog organization level or for the desired repositories.
 
-For a full explanation, see the guide on [using CI jobs failure analysis][21].
+Log Storage is billed separately from CI Visibility. Log retention, exclusion, and indexes are configured in [Log Management][18]. Logs for Azure jobs can be identified by the `datadog.product:cipipeline` and `source:azurepipelines` tags.
 
 ## Visualize pipeline data in Datadog
 
@@ -195,12 +191,11 @@ The {{< ui >}}CI Pipeline List{{< /ui >}} page shows data for only the default b
 [9]: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/approvals?view=azure-devops&tabs=check-pass#approvals
 [10]: /glossary/#custom-tag
 [11]: /glossary/#custom-measure
-[14]: https://app.datadoghq.com/integrations/azure
 [15]: /glossary/#custom-span
 [17]: /continuous_integration/search/#search-for-pipelines
 [18]: /logs/guide/best-practices-for-log-management/
 [19]: /continuous_integration/guides/identify_highest_impact_jobs_with_critical_path/
 [20]: /glossary/#pipeline-execution-time
 [21]: /continuous_integration/guides/use_ci_jobs_failure_analysis/
 [22]: /continuous_integration/guides/use_ci_jobs_failure_analysis/#using-pr-comments
-[23]: https://app.datadoghq.com/ci/settings/ci-cd/azure-devops
+[23]: https://app.datadoghq.com/ci/settings/ci-cd/repositories?tab=repository

hugo/content/en/continuous_integration/pipelines/buildkite.md

@@ -25,14 +25,15 @@ Set up CI Visibility for Buildkite to optimize your resource usage, reduce overh
 | Pipeline Visibility | Platform | Definition |
 |---|---|---|
 | [Partial retries][9] | Partial pipelines | View partially retried pipeline executions. |
+| [CI jobs failure analysis][22] | CI jobs failure analysis | Uses LLM models on relevant logs to analyze the root cause of failed CI jobs. |
 | Infrastructure metric correlation | Infrastructure metric correlation | Correlate jobs to [infrastructure host metrics][6] for Buildkite agents. |
 | [Manual steps][12] | Manual steps | View manually triggered pipelines. |
 | [Queue time][13] | Queue time | View the amount of time pipeline jobs sit in the queue before processing. |
 | [Custom tags][10] [and measures at runtime][11] | Custom tags and measures at runtime | Configure [custom tags and measures][6] at runtime. |
 | [Custom spans][14] | Custom spans | Configure custom spans for your pipelines. |
 | [Filter CI Jobs on the critical path][17] | Filter CI Jobs on the critical path | Filter by jobs on the critical path. |
 | [Execution time][18] | Execution time  | View the amount of time pipelines have been running jobs. |
-| Logs correlation | Logs correlation | Correlate pipeline and job spans to logs and enable [job log collection][20]. |
+| Logs correlation | Logs correlation | Correlate pipeline and job spans to logs and enable [job log storage][20]. |
 
 
 ### Terminology
@@ -57,11 +58,21 @@ To set up the Datadog integration for [Buildkite][1]:
    * {{< ui >}}Branch filtering{{< /ui >}}: Leave empty to trace all branches or select the subset of branches you want to trace.
 3. Click {{< ui >}}Add Datadog Pipeline Visibility Notification{{< /ui >}} to save the integration.
 
-### Collect job logs
+### Manage job logs
+
+CI jobs failure analysis uses LLM models to compute the analysis for failed CI jobs based on relevant logs coming from Buildkite.
+
+For a full explanation, see the guide on [using CI jobs failure analysis][22].
+
+Log Analysis is enabled by default in Datadog. To receive Buildkite job logs for analysis, add your API token in the [Buildkite integration tile][19]. To manage Log Analysis settings, go to [CI/CD Repository settings][23], and configure Log Analysis at the Datadog organization level or for the desired repositories.
+
+#### Store job logs
 
 The Datadog Buildkite integration collects logs from your finished Buildkite jobs and forwards them to Datadog. To install and configure this integration, see the [Buildkite integration documentation][19].
 
-Datadog bills logs separately from CI Visibility. Configure log retention, exclusion filters, and indexes in [Log Management][21]. To scope these rules to Buildkite logs, filter on the `datadog.product:cipipeline` and `source:buildkite` tags.
+To enable Log Storage, go to [CI/CD Repository settings][23], and enable Log Storage at the Datadog organization level or for the desired repositories.
+
+Log Storage is billed separately from CI Visibility. Configure log retention, exclusion filters, and indexes in [Log Management][21]. To scope these rules to Buildkite logs, filter on the `datadog.product:cipipeline` and `source:buildkite` tags.
 
 ## Advanced configuration
 
@@ -163,5 +174,7 @@ The {{< ui >}}CI Pipeline List{{< /ui >}} page shows data for only the default b
 [17]: /continuous_integration/guides/identify_highest_impact_jobs_with_critical_path/
 [18]: /glossary/#pipeline-execution-time
 [19]: /integrations/buildkite/
-[20]: /continuous_integration/pipelines/buildkite/#collect-job-logs
+[20]: /continuous_integration/pipelines/buildkite/#store-job-logs
 [21]: /logs/
+[22]: /continuous_integration/guides/use_ci_jobs_failure_analysis/
+[23]: https://app.datadoghq.com/ci/settings/ci-cd/repositories?tab=repository

hugo/content/en/continuous_integration/pipelines/circleci.md

@@ -28,7 +28,8 @@ Set up CI Visibility for CircleCI to optimize the performance of your pipelines,
 | Pipeline Visibility | Platform | Definition |
 |---|---|---|
 | [Partial retries][12] | Partial pipelines | View partially retried pipeline executions. |
-| Logs correlation | Logs correlation | Correlate pipeline and job spans to logs and enable [job log collection][10]. |
+| [CI jobs failure analysis][20] | CI jobs failure analysis | Uses LLM models on relevant logs to analyze the root cause of failed CI jobs. |
+| Logs correlation | Logs correlation | Correlate pipeline and job spans to logs and enable [job log storage][10]. |
 | [Custom spans][13] | Custom spans | Configure custom spans for your pipelines. |
 | Custom pre-defined tags | Custom pre-defined tags | Set [custom tags][6] to all generated pipeline and job spans. |
 | [Custom tags][14] [and measures at runtime][15] | Custom tags and measures at runtime | Configure [custom tags and measures][7] at runtime. |
@@ -103,11 +104,21 @@ If a `key:value` pair contains any commas, surround it with quotes. For example,
 
 To display and filter the teams associated with your pipelines, add `team:<your-team>` as a custom tag. The custom tag name must match your [Datadog Teams][8] team handle exactly.
 
-### Collect job logs
+### Manage job logs
+
+CI jobs failure analysis uses LLM models to compute the analysis for failed CI jobs based on relevant logs coming from CircleCI.
+
+For a full explanation, see the guide on [using CI jobs failure analysis][20].
+
+Log Analysis is enabled by default in Datadog. To receive CircleCI job logs for analysis, add your API token in the [CircleCI integration tile][11]. To manage Log Analysis settings, go to [CI/CD Repository settings][22], and configure Log Analysis at the Datadog organization level or for the desired repositories.
+
+#### Store job logs
 
 The Datadog CircleCI integration collects logs from your finished CircleCI jobs and forwards them to Datadog. To install and configure this integration, see the [CircleCI integration documentation][11].
 
-Logs are billed separately from CI Visibility. Log retention, exclusion, and indexes are configured in [Log Management][16]. Logs for CircleCI jobs can be identified by the `datadog.product:cipipeline` and `source:circleci` tags.
+To enable Log Storage, go to [CI/CD Repository settings][22], and enable Log Storage at the Datadog organization level or for the desired repositories.
+
+Log Storage is billed separately from CI Visibility. Log retention, exclusion, and indexes are configured in [Log Management][16]. Logs for CircleCI jobs can be identified by the `datadog.product:cipipeline` and `source:circleci` tags.
 
 ## Visualize pipeline data in Datadog
 
@@ -128,7 +139,7 @@ The {{< ui >}}CI Pipeline List{{< /ui >}} page shows data for only the default b
 [7]: /continuous_integration/pipelines/custom_tags_and_measures/?tab=linux
 [8]: /account_management/teams/
 [9]: https://raw.githubusercontent.com/DataDog/ci-visibility-circle-ci/main/service_hooks.py
-[10]: /continuous_integration/pipelines/circleci/#enable-log-collection
+[10]: /continuous_integration/pipelines/circleci/#store-job-logs
 [11]: /integrations/circleci/#setup
 [12]: /glossary/#partial-retry
 [13]: /glossary/#custom-span
@@ -137,4 +148,6 @@ The {{< ui >}}CI Pipeline List{{< /ui >}} page shows data for only the default b
 [16]: /logs/guide/best-practices-for-log-management/
 [17]: /continuous_integration/search/#search-for-pipelines
 [18]: /continuous_integration/guides/identify_highest_impact_jobs_with_critical_path/
-[19]: /glossary/#pipeline-execution-time
+[19]: /glossary/#pipeline-execution-time
+[20]: /continuous_integration/guides/use_ci_jobs_failure_analysis/
+[22]: https://app.datadoghq.com/ci/settings/ci-cd/repositories?tab=repository