Document soda-managed agent (#702)

* Document soda-managed agent

* adjusted toc

* Remaining updates and changes to include soda-hosted agent

* Update image

* adjusted typo

* correct link

* Refined security language

* Added details for role parameter in Snowflake connections

* Adjusted details, added command to see logs

* Add snowflake troubleshooting

* Added agent migration instructions

* added release notes

* Clarified migration guide

Author: janet-can

_data/nav.yml

@@ -13,6 +13,8 @@
     page: soda/setup-guide.md
   - subtitle: Install Soda Library
     page: soda-library/install.md
+  - subtitle: Set up a Soda-hosted agent
+    page: soda-agent/managed-agent.md
   - subtitle: Deploy a Soda Agent
     page: soda-agent/deploy.md
   - subtitle: Soda Agent extras

_includes/about-soda.md

@@ -14,7 +14,7 @@ Soda works by taking the data quality checks that you prepare and using them to
 
 To test your data quality, you choose a flavor of Soda (choose a deployment model) which enables you to configure connections with your data sources and define data quality checks, then run scans that execute your data quality checks.
 
-* **Connect to your data source.** <br />Connect Soda to a data source such as Snowflake, Amazon Athena, or Big Query by providing access details for your data source such as host, port, and data source login credentials. 
+* **Connect to your data source.** <br />Connect Soda to a data source such as Snowflake, Amazon Athena, or BigQuery by providing access details for your data source such as host, port, and data source login credentials. 
 * **Define checks to surface bad-quality data.** <br />Define data quality checks using Soda Checks Language (SodaCL), a domain-specific language for data quality testing. A Soda Check is a test that Soda performs when it scans a dataset in your data source. 
 * **Run a scan to execute your data quality checks.** <br />During a scan, Soda does not ingest your data, it only scans it for quality metrics, then uses the metadata to prepare scan results<sup>1</sup>. After a scan, each check results in one of three default states:
     * pass: the values in the dataset match or fall within the thresholds you specified

_includes/access-managed-agent.md

@@ -0,0 +1,5 @@
+1. If you have not already done so, create a Soda Cloud account at <a href="https://cloud.soda.io/signup?utm_source=docs" target="_blank"> cloud.soda.io</a>. If you already have a Soda account, log in.
+2. By default, Soda prepares a Soda-hosted agent for all newly-created accounts. However, if you are an Admin in an existing Soda Cloud account and wish to use a Soda-hosted agent, navigate to **your avatar** > **Organization Settings**. In the **Organization** tab, click the checkbox to **Enable Soda-hosted Agent**. 
+3. Navigate to **your avatar** > **Data Sources**, then access the **Agents** tab. Notice your out-of-the-box Soda-hosted agent that is up and running.
+
+![soda-hosted-agent](/assets/images/soda-hosted-agent.png){:height="700px" width="700px"}

_includes/banner-agreements.md

@@ -0,0 +1,4 @@
+<div class="info">
+  <span class="closebtn" onclick="this.parentElement.style.display='none';">&times;</span>
+  The <strong>agreement</strong> feature is being deprecated and is only available upon request. Contact <a href="mailto:[email protected]">Soda Support</a> to request access.
+</div>

_includes/compatible-cloud-datasources.md

@@ -1,6 +1,6 @@
 <table>
   <tr>
-    <td>Amazon Athena<br /> Amazon Redshift<br />  Azure Synapse<br /> ClickHouse <br /> Databricks SQL<br />Denodo <br /> Dremio <br /> DuckDB <br /> GCP Big Query<br /> Google CloudSQL</td>
+    <td>Amazon Athena<br /> Amazon Redshift<br />  Azure Synapse<br /> ClickHouse <br /> Databricks SQL<br />Denodo <br /> Dremio <br /> DuckDB <br /> GCP BigQuery<br /> Google CloudSQL</td>
     <td>IBM DB2<br /> MotherDuck <br /> MS SQL Server<sup>1</sup><br /> MySQL<br > OracleDB<br />PostgreSQL<br /> Presto <br /> Snowflake<br /> Trino<br /> Vertica </td>
   </tr>
 </table>

_release-notes/soda-hosted-agent.md

@@ -0,0 +1,10 @@
+---
+name: "Soda-hosted Agent"
+date: 2024-02-01
+products:
+  - soda-cloud
+---
+
+Introducing a secure, out-of-the-box Soda-hosted Agent to manage access to data sources from within your Soda Cloud account. Quickly configure connections to your data sources in the Soda Cloud user interface, then empower all your colleagues to explore datasets, access check results, customize collections, and create their own no-code checks for data quality.
+
+Learn how to [Set up a Soda-hosted agent]({% link soda-agent/managed-agent.md %}).

assets/images/soda-hosted-agent.png

@@ -36,10 +36,10 @@ <h2>Get started</h2>
                 <div>
                     <img src="/assets/images/icons/[email protected]" width="54" height="40">
                     <h2>What's new?</h2>
-                    <a href="/soda-cl/soda-cl-overview.html#define-sodacl-checks">✨ Create no-code checks ✨</a>  
+                    <a href="/soda-agent/managed-agent.html">Set up Soda-hosted Agent</a> 
+                    <a href="/soda-cl/soda-cl-overview.html#define-sodacl-checks">Create no-code checks</a>  
                     <a href="/soda-cloud/scan-mgmt.html">Manage scheduled scans</a> 
-                    <a href="/soda/data-contracts.html">Data contracts</a> 
-                    <a href="/soda/new-documentation.html#october-11-2023">Newly-revised documentation</a>                                    
+                    <a href="/soda/data-contracts.html">Create data contracts</a>                                    
                 </div>
                 <div>
                     <img src="/assets/images/icons/[email protected]" width="54" height="40">

assets/images/soda-hosted-agent1.png

@@ -10,11 +10,11 @@ redirect_from: /soda-agent/
 <!--Linked to UI, access Shlink-->
 *Last modified on {% last_modified_at %}*
 
-The **Soda Agent** is a tool that empowers Soda Cloud users to securely access data sources to scan for data quality. Create a Kubernetes cluster in a cloud services provider environment, then use Helm to deploy a Soda Agent in the cluster. 
+The **Soda Agent** is a tool that empowers Soda Cloud users to securely access data sources to scan for data quality. For a self-hosted agent, create a Kubernetes cluster in a cloud services provider environment, then use Helm to deploy a Soda Agent in the cluster. 
 
-This setup enables Soda Cloud users to securely connect to data sources (Snowflake, Amazon Athena, etc.) from within the Soda Cloud web application. Any user in your Soda Cloud account can add a new data source via the agent, then write their own no-code checks and agreements to check for data quality in the new data source. 
+This setup enables Soda Cloud users to securely connect to data sources (Snowflake, Amazon Athena, etc.) from within the Soda Cloud web application. Any user in your Soda Cloud account can add a new data source via the agent, then write their own no-code checks to check for data quality in the new data source. 
 
-What follows is an extremely abridged introduction to a few basic elements involved in the deployment and setup of a Soda Agent.
+What follows is an extremely abridged introduction to a few basic elements involved in the deployment and setup of a self-hosted Soda Agent.
 
 ![agent-diagram](/assets/images/agent-diagram.png){:height="700px" width="700px"}
 

assets/images/with-library.png

@@ -19,9 +19,9 @@ redirect_from:
 <!--Linked to UI, access Shlink-->
 *Last modified on {% last_modified_at %}*
 
-The **Soda Agent** is a tool that empowers Soda Cloud users to securely access data sources to scan for data quality. Create a Kubernetes cluster, then use Helm to deploy a Soda Agent in the cluster. 
+The **Soda Agent** is a tool that empowers Soda Cloud users to securely access data sources to scan for data quality. Create a Kubernetes cluster, then use Helm to deploy a self-hosted Soda Agent in the cluster.
 
-This setup enables Soda Cloud users to securely connect to data sources (BigQuery, Snowflake, etc.) from within the Soda Cloud web application. Any user in your Soda Cloud account can add a new data source via the agent, then write their own no-code checks and agreements to check for data quality in the new data source.
+This setup enables Soda Cloud users to securely connect to data sources (BigQuery, Snowflake, etc.) from within the Soda Cloud web application. Any user in your Soda Cloud account can add a new data source via the agent, then write their own no-code checks and agreements to check for data quality in the new data source. Alternatively, if you use a BigQuery, MySQL, PostgreSQL, or Snowflake data source, you can use a secure, out-of-the-box [Soda-hosted agent]({% link soda-agent/managed-agent.md %}) made available for every Soda Cloud organization.
 
 As a step in the **Get started roadmap**, this guide offers instructions to set up, install, and configure Soda in a [self-hosted agent deployment model]({% link soda/setup-guide.md %}#self-hosted-agent).
 
@@ -173,7 +173,7 @@ REVISION: 1
 ```shell
 minikube kubectl -- describe pods
 ```
-4. In your Soda Cloud account, navigate to **your avatar** > **Data Sources** > **Agents** tab. Refresh the page to verify that you see the agent you just created in the list of Agents. <br/><br/>Be aware that this may take several minutes to appear in your list of Soda Agents. Use the `describe pods` command in step 3 to check the status of the deployment. When `State: Running` and `Ready: True`, then you can refresh and see the agent in Soda Cloud.
+4. In your Soda Cloud account, navigate to **your avatar** > **Data Sources** > **Agents** tab. Refresh the page to verify that you see the agent you just created in the list of Agents. <br/><br/>Be aware that this may take several minutes to appear in your list of Soda Agents. Use the `describe pods` command in step 3 to check the status of the deployment. When `State: Running` and `Ready: True`, then you can refresh and see the agent in Soda Cloud. 
 ```shell
 ...
 Containers:
@@ -190,7 +190,11 @@ Containers:
 ```
 ![agent-deployed](/assets/images/agent-deployed.png){:height="600px" width="600px"}
 
-<br />
+If you do no see the agent listed in Soda Cloud, use the following command to review status and investigate the logs.
+```shell
+kubectl logs -l agent.soda.io/component=orchestrator -n soda-agent -f
+```
+<br/>
 
 #### Deploy using a values YAML file
 
@@ -246,6 +250,11 @@ Containers:
 ```
 ![agent-deployed](/assets/images/agent-deployed.png){:height="700px" width="700px"}
 
+If you do no see the agent listed in Soda Cloud, use the following command to review status and investigate the logs.
+```shell
+kubectl logs -l agent.soda.io/component=orchestrator -n soda-agent -f
+```
+
 If you use private key authentication with a Soda Agent, refer to [Soda Agent extras]({% link soda-agent/secrets.md %}#use-a-values-file-to-store-private-key-authentication-values).
 
 <br />
@@ -258,7 +267,7 @@ If you use private key authentication with a Soda Agent, refer to [Soda Agent ex
 
 1. Uninstall the Soda Agent in the cluster.
 ```shell
-helm delete soda-agent -n soda-agent
+helm uninstall soda-agent -n soda-agent
 ```
 2. Delete the cluster.
 ```shell
@@ -421,6 +430,11 @@ Containers:
 ```
 ![agent-deployed](/assets/images/agent-deployed.png){:height="700px" width="700px"}
 
+If you do no see the agent listed in Soda Cloud, use the following command to review status and investigate the logs.
+```shell
+kubectl logs -l agent.soda.io/component=orchestrator -n soda-agent -f
+```
+
 <br />
 
 #### Deploy using a values YAML file
@@ -478,6 +492,11 @@ Containers:
 ```
 ![agent-deployed](/assets/images/agent-deployed.png){:height="700px" width="700px"}
 
+If you do no see the agent listed in Soda Cloud, use the following command to review status and investigate the logs.
+```shell
+kubectl logs -l agent.soda.io/component=orchestrator -n soda-agent -f
+```
+
 <br />
 
 ### (Optional) Connect via AWS PrivateLink
@@ -494,6 +513,11 @@ kubectl -n soda-agent rollout restart deploy
 5. After you have started the agent and validated that it is running, log into your Soda Cloud account, then navigate to **your avatar** > **Data Sources** > **Agents** tab. Refresh the page to verify that you see the agent you just created in the list of Agents. 
 ![agent-deployed](/assets/images/agent-deployed.png){:height="700px" width="700px"}
 
+If you do no see the agent listed in Soda Cloud, use the following command to review status and investigate the logs.
+```shell
+kubectl logs -l agent.soda.io/component=orchestrator -n soda-agent -f
+```
+
 
 ### About the `helm install` command
 
@@ -504,7 +528,7 @@ kubectl -n soda-agent rollout restart deploy
 
 1. Uninstall the Soda Agent in the cluster.
 ```shell
-helm delete soda-agent -n soda-agent
+helm uninstall soda-agent -n soda-agent
 ```
 2. Remove the Fargate profile.
 ```shell
@@ -823,6 +847,11 @@ soda-agent-orchestrator-ffd74c76-5g7tl   1/1     Running   0          32s
 Be aware that this may take several minutes to appear in your list of Soda Agents. 
 ![agent-deployed](/assets/images/agent-deployed.png){:height="700px" width="700px"}
 
+If you do no see the agent listed in Soda Cloud, use the following command to review status and investigate the logs.
+```shell
+kubectl logs -l agent.soda.io/component=orchestrator -n soda-agent -f
+```
+
 <br /> 
 
 #### Deploy using CLI only - virtual cluster
@@ -877,6 +906,11 @@ soda-agent-orchestrator-ffd74c76-5g7tl   1/1     Running   0          32s
 Be aware that this may take several minutes to appear in your list of Soda Agents. 
 ![agent-deployed](/assets/images/agent-deployed.png){:height="700px" width="700px"}
 
+If you do no see the agent listed in Soda Cloud, use the following command to review status and investigate the logs.
+```shell
+kubectl logs -l agent.soda.io/component=orchestrator -n soda-agent -f
+```
+
 <br /> 
 
 #### Deploy using a values YAML file
@@ -928,6 +962,11 @@ kubectl describe pods -n soda-agent
 8. In your Soda Cloud account, navigate to **your avatar** > **Data Sources** > **Agents** tab. Refresh the page to verify that you see the agent you just created in the list of Agents. 
 ![agent-deployed](/assets/images/agent-deployed.png){:height="700px" width="700px"}
 
+If you do no see the agent listed in Soda Cloud, use the following command to review status and investigate the logs.
+```shell
+kubectl logs -l agent.soda.io/component=orchestrator -n soda-agent -f
+```
+
 <br /> 
 
 ## About the `helm install` command
@@ -1178,6 +1217,11 @@ Status:           Running
 ```
 ![agent-deployed](/assets/images/agent-deployed.png){:height="600px" width="600px"}
 
+If you do no see the agent listed in Soda Cloud, use the following command to review status and investigate the logs.
+```shell
+kubectl logs -l agent.soda.io/component=orchestrator -n soda-agent -f
+```
+
 <br />
 
 #### Deploy using a values YAML file
@@ -1234,6 +1278,11 @@ Status:           Running
 ```
 ![agent-deployed](/assets/images/agent-deployed.png){:height="700px" width="700px"}
 
+If you do no see the agent listed in Soda Cloud, use the following command to review status and investigate the logs.
+```shell
+kubectl logs -l agent.soda.io/component=orchestrator -n soda-agent -f
+```
+
 <br /> 
 
 ## About the `helm install` command
@@ -1244,7 +1293,7 @@ Status:           Running
 
 1. Uninstall the Soda Agent in the cluster.
 ```shell
-helm delete soda-agent -n soda-agent
+helm uninstall soda-agent -n soda-agent
 ```
 2. Delete the cluster.
 ```shell
@@ -1273,7 +1322,7 @@ In your Soda Cloud account, navigate to **your avatar** > **Data Sources**. Clic
 | -----------------------   | ---------- |
 | Data Source Label | Provide a unique identifier for the data source. Soda Cloud uses the label you provide to define the immutable name of the data source against which it runs the Default Scan.|
 | Default Scan Schedule Label | Provide a name for the default scan schedule for this data sources. The scan schedule indicates which Soda Agent to use to execute the scan, and when.  |
-| Default Scan Schedule Agent | Select the name of a Soda Agent that you have previously set up in your secure environment and connected to a specific data source. This identifies the Soda Agent to which Soda Cloud must connect in order to run its scan. |
+| Default Scan Schedule Agent | Select the name of a Soda Agent that you have previously set up in your secure environment. This identifies the Soda Agent to which Soda Cloud must connect in order to run its scan. |
 | Schedule Definition | Provide the scan frequency details Soda Cloud uses to execute scans according to your needs. If you wish, you can define the schedule as a cron expression. |
 | Starting At | Select the time of day to run the scan. The default value is midnight. |
 | Time Zone | Select a timezone. The default value is UTC. |
@@ -1289,7 +1338,7 @@ To more securely provide sensitive values such as usernames and passwords, use e
 
 Access the data source-specific connection configurations listed below to copy+paste the connection syntax into the editing panel, then adjust the values to correspond with your data source's details. Access connection configuration details in [Data source reference]({% link soda/connect-athena.md %}) section of Soda documentation.
 
-See also: [Use a file reference for a Big Query data source connection](#use-a-file-reference-for-a-big-query-data-source-connection)
+See also: [Use a file reference for a BigQuery data source connection](#use-a-file-reference-for-a-bigquery-data-source-connection)
 
 <br />
 
@@ -1352,7 +1401,7 @@ automated monitoring:
 
 <br />
 
-### Use a file reference for a Big Query data source connection
+### Use a file reference for a BigQuery data source connection
 
 If you already store information about your data source in a JSON file in a secure location, you can configure your BigQuery data source connection details in Soda Cloud to refer to the JSON file for service account information. To do so, you must add two elements:
 * `volumes` and `volumeMounts` parameters in the `values.yml` file that your Soda Agent helm chart uses