Changes for SSL cert errors, SF proxy params, data contracts, SSO SP-initiated, reference check error (#804)

janet-can · web-flow · commit a19fb3d324d9 · 2024-06-17T16:55:19.000-07:00
* Troubleshoot SSL certificate error.

* Reference and Snowflake connection troubleshooting

* Clarified scan def names in programmatic scans

* Data contract clarifications and adjustments
diff --git a/_data/nav.yml b/_data/nav.yml
@@ -226,6 +226,8 @@
     page: soda/connect-trino.md
   - subtitle: Connect to Vertica
     page: soda/connect-vertica.md
+  - subtitle: Troubleshoot connections
+    page: soda/connect-troubleshoot.md
 
 - title: Soda Cloud API
   page: api-docs/public-cloud-api-v1.md
diff --git a/_includes/custom-sampler.md b/_includes/custom-sampler.md
@@ -1,3 +1,4 @@
 * You can save Soda Library scan results anywhere in your system; the `scan_result` object contains all the scan result information. To import Soda Library in Python so you can utilize the `Scan()` object, [install a Soda Library package]({% link soda-library/programmatic.md %}), then use `from soda.scan import Scan`.
+* If you provide a name for the scan definition to identify inline checks in a programmatic scan as independent of other inline checks in a different programmatic scan or pipeline, be sure to set a unique scan definition name for each programmatic scan. Using the same scan definition name in multiple programmatic scans results in confused check results in Soda Cloud.
 * If you wish to collect samples of failed rows when a check fails, you can employ a custom sampler; see [Configure a failed row sampler]({% link soda-cl/failed-rows-checks.md %}#configure-a-failed-row-sampler).
 * Be sure to include any variables in your programmatic scan *before* the check YAML files. Soda requires the variable input for any variables defined in the check YAML files.
diff --git a/_includes/snowflake-proxy.md b/_includes/snowflake-proxy.md
@@ -0,0 +1,18 @@
+**Problem:** While attempting to connect Soda to a Snowflake data source using proxy parameters, you encounter an error that reads something similar to `Could not connect to data source "name_db": 250001 (08001): Failed to connect to DB: mydb.eu-west-1.snowflakecomputing.com:443. Incoming request with IP/Token xx.xxx.xx.xxx is not allowed to access Snowflake.`
+
+```yaml
+data_source: my_data_source
+  type: snowflake
+  ...
+  session_param:
+    QUERY_TAG: soda-test
+    QUOTED_IDENTIFIERS_IGNORE_CASE: false
+  proxy_http: http://a-proxy-o-dd-dddd-net:8000
+  proxy_https: https://a-proxy-o-dd-dddd-net:8000
+```
+
+**Solution:** When connecting to a Snowflake data source by proxyy, be sure to set the new proxy environment variables from the command-line using export statements, as in the following example.
+```shell
+export HTTP_PROXY=http://a-proxy-o-dd-dddd-net:8000
+export HTTPS_PROXY=https://a-proxy-o-dd-dddd-net:8000
+```
diff --git a/assets/images/experimental.png b/assets/images/experimental.png
diff --git a/soda-cl/reference.md b/soda-cl/reference.md
@@ -105,7 +105,7 @@ To review the failed rows in Soda Cloud, navigate to the **Checks** dashboard, t
 | ✓ | Use quotes when identifying dataset or column names; see [example](#example-with-quotes). <br />Note that the type of quotes you use must match that which your data source uses. For example, BigQuery uses a backtick ({% raw %}`{% endraw %}) as a quotation mark. | [Use quotes in a check]({% link soda-cl/optional-config.md %}#use-quotes-in-a-check) |
 |   | Use wildcard characters ({% raw %} % {% endraw %} or {% raw %} * {% endraw %}) in values in the check. | - |
 |   | Use for each to apply reference checks to multiple datasets in one scan. | - |
-| ✓ | Apply a dataset filter to partition data during a scan; see [example](#example-with-dataset-filter). | [Scan a portion of your dataset]({% link soda-cl/optional-config.md %}#scan-a-portion-of-your-dataset) |
+| ✓ | Apply a dataset filter to partition data during a scan; see [example](#example-with-dataset-filter). If you encounter difficulties, see [Filter not passed with reference check]({% link soda-cl/troubleshoot.md %}#filter-not-passed-with-reference-check). | [Scan a portion of your dataset]({% link soda-cl/optional-config.md %}#scan-a-portion-of-your-dataset) |
 
 #### Example with check name 
 {% include code-header.html %}
@@ -123,6 +123,9 @@ checks for dim_department_group:
 ```
 
 #### Example with dataset filter
+
+Refer to [Troubleshoot SodaCL]({% link soda-cl/troubleshoot.md %}#filter-not-passed-with-reference-check) to address challenges specific to reference checks with dataset filters.
+
 {% include code-header.html %}
 ```yaml
 filter customers_c8d90f60 [daily]:
@@ -132,7 +135,7 @@ checks for customers_c8d90f60 [daily]:
   - values in (cat) must exist in customers_europe (cat2)
 ```
 
-Refer to [Troubleshoot SodaCL]({% link soda-cl/troubleshoot.md %}#filter-not-passed-with-reference-check) to address challenges specific to reference checks with dataset filters.
+
 
 <br />
 
diff --git a/soda-cloud/sso.md b/soda-cloud/sso.md
@@ -34,6 +34,8 @@ When an organization's IT Admin revokes a user's access to Soda Cloud through th
 
 Once your organization enables SSO for all Soda Cloud users, Soda Cloud blocks all non-SSO login attempts and password changes via <a href="https://cloud.soda.io/login" target="_blank">cloud.soda.io/login<a/>. If an employee attempts a non-SSO login or attempts to change a password using "Forgot password?" on <a href="https://cloud.soda.io/login" target="_blank">cloud.soda.io/login<a/>, Soda Cloud presents a message that explains that they must log in or change their password using their SSO provider. 
 
+Soda Cloud supports both Identity Provider Initiated (IdP-initiated), and Service Provider Initiated (SP-initiated) single sign-on integrations. Be sure to incidate which type of SSO your organization uses when setting it up with the Soda Support team.
+
 
 ## Add Soda Cloud to Azure AD
 
@@ -63,7 +65,7 @@ Once your organization enables SSO for all Soda Cloud users, Soda Cloud blocks a
 * **Azure AD Identifier** (Section 4 in Azure). This is the IdP entity, ID, or Identity Provider Issuer that Soda needs
 * **Login URL** (Section 4 in Azure). This is the IdP SSO service URL, or Identity Provider Single Sign-On URL that Soda needs.
 * **X.509 Certificate**. Click the **Download** link next to **Certificate (Base64)**.
-12. Email the copied and downloaded values to <a href="mailto:support@soda.io">support@soda.io</a>. With those values, Soda completes the SSO configuration for your organization in cloud.soda.io and notifies you of completion.
+12. Email the copied and downloaded values to <a href="mailto:support@soda.io">support@soda.io</a>. With those values, Soda completes the SSO configuration for your organization in cloud.soda.io and notifies you of completion. Soda Cloud supports both Identity Provider Initiated (IdP-initiated), and Service Provider Initiated (SP-initiated) single sign-on integrations; be sure to incidate which type of SSO your organization uses.
 13. Test the integration by assigning the Soda application in Azure AD to a single user, then requesting that they log in.
 14. After a successful single-user test of the sign in, assign access to the Soda Azure AD app to users and/or user groups in your organization.
 
@@ -89,7 +91,7 @@ The values for these fields are unique to your organization and are provided to
 * **Identity Provider Single Sign-On URL**
 * **Identity Provider Issuer**
 * **X.509 Certificate**
-11. Email the copied and downloaded values to <a href="mailto:support@soda.io">support@soda.io</a>. With those values, Soda completes the SSO configuration for your organization in cloud.soda.io and notifies you of completion.
+11. Email the copied and downloaded values to <a href="mailto:support@soda.io">support@soda.io</a>. With those values, Soda completes the SSO configuration for your organization in cloud.soda.io and notifies you of completion. Soda Cloud supports both Identity Provider Initiated (IdP-initiated), and Service Provider Initiated (SP-initiated) single sign-on integrations; be sure to incidate which type of SSO your organization uses.
 12. Test the integration by assigning the Soda application in Okta to a single user, then requesting that they log in.
 13. After a successful single-user test of the sign in, assign access to the Soda Okta app to users and/or user groups in your organization.
 
@@ -107,7 +109,7 @@ The values for these fields are unique to your organization and are provided to
 5. On the **SAML Attribute mapping** page, add two Google directory attributes and map as follows:
 * Last Name → User.FamilyName
 * First Name → User.GivenName
-6. Email the copied and downloaded values to <a href="mailto:support@soda.io">support@soda.io</a>. With those values, Soda completes the SSO configuration for your organization in cloud.soda.io and notifies you of completion.
+6. Email the copied and downloaded values to <a href="mailto:support@soda.io">support@soda.io</a>. With those values, Soda completes the SSO configuration for your organization in cloud.soda.io and notifies you of completion. Soda Cloud supports both Identity Provider Initiated (IdP-initiated), and Service Provider Initiated (SP-initiated) single sign-on integrations; be sure to incidate which type of SSO your organization uses.
 7. In the Google Workspace admin portal, use Google's instructions to <a href="https://support.google.com/a/answer/6087519?hl=en&ref_topic=7559288" target="_blank">Turn on your SAML app</a> and verify that SSO works with the new custom app for Soda.
 
 
diff --git a/soda-library/programmatic.md b/soda-library/programmatic.md
@@ -140,6 +140,8 @@ scan.execute()
 scan.set_verbose(True)
 
 # Set scan definition name, equivalent to CLI -s option
+# The scan definition name MUST be unique to this scan, and
+# not duplicated in any other programmatic scan
 ##################
 scan.set_scan_definition_name("YOUR_SCHEDULE_NAME")
 
diff --git a/soda-library/run-a-scan.md b/soda-library/run-a-scan.md
@@ -282,9 +282,8 @@ Because Soda Library pushes scan results to Soda Cloud, you may not want to chan
 
 **Problem:** In a Windows environment, you see an error that reads `[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (ssl_c:997)`.
 
-**Short-term solution:** Use `pip install pip-system-certs` to temporarily resolve the issue. This install works to resolve the issue only on Windows machines where the Ops team installs all the certificates needed through Group Policy Objects, or similar. However, the fix is short-term because when you try to run this in a pipeline on another machine, the error will reappear.
+**Solution:** Use `pip install pip-system-certs` to potentially resolve the issue. This install works to resolve the issue only on Windows machines where the Ops team installs all the certificates needed through Group Policy Objects, or similar. 
 
-**Short-term solution:** Contact your Operations or System Admin team to obtain the proxy certificate.
 
   </div>
   <div class="panel" id="three-panel" markdown="1">
@@ -359,6 +358,8 @@ scan.execute()
 scan.set_verbose(True)
 
 # Set scan definition name, equivalent to CLI -s option
+# The scan definition name MUST be unique to this scan, and
+# not duplicated in any other programmatic scan
 ##################
 scan.set_scan_definition_name("YOUR_SCHEDULE_NAME")
 
diff --git a/soda/connect-snowflake.md b/soda/connect-snowflake.md
@@ -171,6 +171,10 @@ checks for VOLUME:
             name: Trader row count
 ```
 
+<br />
+
+{% include snowflake-proxy.md %}
+
 <br />
 <br />
 
diff --git a/soda/connect-troubleshoot.md b/soda/connect-troubleshoot.md
@@ -0,0 +1,39 @@
+---
+layout: default
+title: Troubleshoot data source connections
+description: 
+parent: 
+---
+
+# Troubleshoot data source connections
+Last modified on {% last_modified_at %}
+
+
+
+## SSL certificate error
+
+**Problem:** You encounter an SSL certificate error while attempting to connect Soda to a data source.
+
+**Solution:** Use `pip install pip-system-certs` to potentially resolve the issue. This install works to resolve the issue only on Windows machines where the Ops team installs all the certificates needed through Group Policy Objects, or similar. 
+
+## Snowflake proxy connection error
+
+{% include snowflake-proxy.md %}
+
+## Go further
+
+* Access [Troubleshoot SodaCL]({% link soda-cl/troubleshoot.md %}) for help resolving issues running scans with SodaCL.
+* Need help? Join the <a href="https://community.soda.io/slack" target="_blank"> Soda community on Slack</a>.
+
+<br />
+
+---
+
+Was this documentation helpful?
+
+<!-- LikeBtn.com BEGIN -->
+<span class="likebtn-wrapper" data-theme="tick" data-i18n_like="Yes" data-ef_voting="grow" data-show_dislike_label="true" data-counter_zero_show="true" data-i18n_dislike="No"></span>
+<script>(function(d,e,s){if(d.getElementById("likebtn_wjs"))return;a=d.createElement(e);m=d.getElementsByTagName(e)[0];a.async=1;a.id="likebtn_wjs";a.src=s;m.parentNode.insertBefore(a, m)})(document,"script","//w.likebtn.com/js/w/widget.js");</script>
+<!-- LikeBtn.com END -->
+
+{% include docs-footer.md %}
diff --git a/soda/data-contracts-checks.md b/soda/data-contracts-checks.md
@@ -5,14 +5,14 @@ description: Soda data contract checks enable you to verify data quality early i
 parent: Create a data contract
 ---
 
-# Data contract check reference
-<br />![experimental](/assets/images/experimental.png){:height="150px" width="150px"} <br />
+# Data contract check reference <br />
+![experimental](/assets/images/experimental.png){:height="300px" width="300px"} <br />
 *Last modified on {% last_modified_at %}*
 
 Soda data contracts is a Python library that verifies data quality standards as early and often as possible in a data pipeline so as to prevent negative downstream impact. Learn more [About Soda data contracts]({% link soda/data-contracts.md %}#about-data-contracts).
 
 <small>✖️ &nbsp;&nbsp; Requires Soda Core Scientific</small><br />
-<small>✔️ &nbsp;&nbsp; Supported in Soda Core 3.3.3 or greater</small><br />
+<small>✔️ &nbsp;&nbsp; Experimentally supported in Soda Core 3.3.3 or greater for PostgreSQL, Spark, and Snowflake</small><br />
 <small>✖️ &nbsp;&nbsp; Supported in Soda Library + Soda Cloud</small><br />
 <small>✖️ &nbsp;&nbsp; Supported in Soda Cloud Agreements + Soda Agent</small><br />
 <small>✖️ &nbsp;&nbsp; Supported by SodaGPT</small><br />
@@ -46,6 +46,8 @@ Note that data contracts checks do not follow SodaCL syntax.
 ```yaml
 dataset: dim_employee
 
+...
+
 columns:
 - name: id
   checks:
@@ -78,6 +80,8 @@ This check compares the maximum value in the column to the time the scan runs; t
 ```yaml
 dataset: dim_customer
 
+...
+
 columns:
 - name: date_first_purchase
   checks:
@@ -101,6 +105,8 @@ See also: [Combine missing and validity](#combine-missing-and-validity)
 ```yaml
 dataset: dim_customer
 
+...
+
 columns: 
 - name: title
   checks: 
@@ -136,6 +142,8 @@ columns:
 ```yaml
 dataset: dim_customer
 
+...
+
 columns: 
 - name: first_name
   checks: 
@@ -158,6 +166,8 @@ checks:
 ```yaml
 dataset: dim_customer
 
+...
+
 columns:
 - name: yearly_income
   checks:
@@ -190,6 +200,8 @@ Relative to a [SQL metric query](#sql-metric-query) check, a SQL metric expressi
 {% include code-header.html %}
 ```yaml
 dataset: CUSTOMERS
+...
+
 columns:
   - name: id
   # SQL metric expression check for a column
@@ -205,6 +217,8 @@ columns:
 {% include code-header.html %}
 ```yaml
 dataset: CUSTOMERS
+...
+
 columns:
   - name: id
   - name: country
@@ -231,6 +245,8 @@ You can apply a SQL metric check to one or more columns or to an entire dataset.
 {% include code-header.html %}
 ```yaml
 dataset: CUSTOMERS
+...
+
 columns:
   # SQL metric query check for a column
   - name: id
@@ -248,6 +264,8 @@ columns:
 {% include code-header.html %}
 ```yaml
 dataset: CUSTOMERS
+...
+
 columns:
   - name: id
 checks:
@@ -274,6 +292,8 @@ checks:
 ```yaml
 dataset: dim_customer
 
+...
+
 columns: 
 - name: first_name
   data_type: character varying
@@ -324,6 +344,8 @@ The referential dataset must exist in the same warehouse as the dataset identifi
 ```yaml
 dataset: dim_employee
 
+...
+
 columns:
 - name: country
   checks:
@@ -343,6 +365,8 @@ You can combine column configuration keys to include both missing and validity p
 ```yaml
 dataset: dim_product
 
+...
+
 columns:
 - name: size
   checks:
@@ -360,6 +384,8 @@ In the example below, Soda considers any row that failed the `no_missing_values`
 ```yaml
 dataset: dim_product
 
+...
+
 columns:
 - name: size
   checks:
@@ -386,6 +412,8 @@ The example below verifies that the only valid value for the column `currency` i
 ```yaml
 dataset: dim_product
 
+...
+
 columns:
 - name: country
 - name: currency
diff --git a/soda/data-contracts-verify.md b/soda/data-contracts-verify.md
@@ -5,16 +5,16 @@ description: Use a Python API to verify data contract checks programmatically wi
 parent: Create a data contract
 ---
 
-# Verify a data contract
-<br />![experimental](/assets/images/experimental.png){:height="150px" width="150px"} <br />
+# Verify a data contract <br />
+![experimental](/assets/images/experimental.png){:height="300px" width="300px"} <br />
 *Last modified on {% last_modified_at %}*
 
 To verify a **Soda data contract** is to scan the data in a warehouse to execute the data contract checks you defined in a contracts YAML file. Available as a Python library, you run the scan programmatically, invoking Soda data contracts in a CI/CD workflow when you create a new pull request, or in a data pipeline after importing or transforming new data. 
 
 When deciding when to verify a data contract, consider that contract verification works best on new data as soon as it is produced so as to limit its exposure to other systems or users who might access it. The earlier in a pipeline or workflow, the better!  Further, best practice suggests that you store batches of new data in a temporary table, verify a contract on the batches, then append the data to a larger table.
 
 <small>✖️ &nbsp;&nbsp; Requires Soda Core Scientific</small><br />
-<small>✔️ &nbsp;&nbsp; Supported in Soda Core 3.3.3 or greater</small><br />
+<small>✔️ &nbsp;&nbsp; Experimentally supported in Soda Core 3.3.3 or greater for PostgreSQL, Spark, and Snowflake</small><br />
 <small>✖️ &nbsp;&nbsp; Supported in Soda Library + Soda Cloud</small><br />
 <small>✖️ &nbsp;&nbsp; Supported in Soda Cloud Agreements + Soda Agent</small><br />
 <small>✖️ &nbsp;&nbsp; Supported by SodaGPT</small><br />
@@ -63,7 +63,7 @@ When deciding when to verify a data contract, consider that contract verificatio
         .execute()
     )
 
-    logging.debug(str(contract_verification_result))
+    print(str(contract_verification_result))
     ```
 4. At runtime, Soda connects with your warehouse and verifies the contract by executing the data contract checks in your file. Use `${SCHEMA}` syntax to provide any environment variable values in a contract YAML file. Soda returns results of the verification as pass or fail check results, or indicate errors if any exist; see below.
 
@@ -117,7 +117,7 @@ contract_verification: ContractVerification = (
 )
 
 if contract_verification.logs.has_errors():
-  logging.error(f"The contract has syntax or semantic errors: \n{contract_verification.logs}")
+  print(f"The contract has syntax or semantic errors: \n{contract_verification.logs}")
 ```
 
 ## Add a check identity
diff --git a/soda/data-contracts-write.md b/soda/data-contracts-write.md
diff --git a/soda/data-contracts.md b/soda/data-contracts.md
diff --git a/soda/new-documentation.md b/soda/new-documentation.md
diff --git a/soda/quick-start-prod.md b/soda/quick-start-prod.md
