Revision and expansion of SodaCL documentation. (#263)

* First draft of numeric metrics

* Tiny notation.

* Further revision and reorganization.

* New and revised docs for several SodaCL checks and metrics.

Co-authored-by: Janet Revell <[email protected]>

Author: janet-can

_data/nav.yml

@@ -113,30 +113,32 @@
   subcategories:
   - subtitle: SodaCL overview
     page: soda-cl/soda-cl-overview.md
-  - subtitle: Soda Core overview
-    page: soda-cl/soda-core-overview.md
+  - subtitle: Metrics and checks
+    page: soda-cl/metrics-and-checks.md
+  - subtitle: Anomaly score check
+    page: soda-cl/anomaly-score.md
+  - subtitle: Dataset filters
+    page: soda-cl/dataset-filters.md
   - subtitle: Distribution checks
     page: soda-cl/distribution.md
   - subtitle: Duplicate checks
     page: soda-cl/duplicates.md
-  - subtitle: For each checks
+  - subtitle: For each
     page: soda-cl/for-each.md
   - subtitle: Freshness checks
     page: soda-cl/freshness.md
-  - subtitle: Metrics and thresholds
-    page: soda-cl/metrics-thresholds.md
   - subtitle: Missing and validity checks
     page: soda-cl/missing-validity.md
-  - subtitle: Quotes
-    page: soda-cl/quotes.md
+  - subtitle: Numeric metrics
+    page: soda-cl/numeric-metrics.md
+  - subtitle: Optional check configurations
+    page: soda-cl/optional-config.md
   - subtitle: Reference checks
     page: soda-cl/reference.md
   - subtitle: Row count checks
     page: soda-cl/row-count.md
   - subtitle: Schema checks
     page: soda-cl/schema.md
-  - subtitle: Table filters
-    page: soda-cl/table-filters.md
   - subtitle: User-defined checks
     page: soda-cl/user-defined.md
 

_data/navcl.yml

@@ -0,0 +1,21 @@
+It can be time-consuming to check exceptionally large datasets for data quality in their entirety. Instead of checking whole datasets, you can use a **dataset filter** to specify a portion of data in a dataset against which Soda Core executes a check.
+
+1. In your checks YAML file, add a section header called `filter`, then append a dataset name and, in square brackets, the name of the filter. Refer to the example below.
+2. Nested under the `filter` header, use a SQL expression to specify the portion of data in a dataset that Soda Core must check. The SQL expression in the example references two variables: `ts_start` and `ts_end`. When you run the `soda scan` command, you must include these two variables as options in the command.
+```yaml
+filter CUSTOMERS [daily]:
+  where: TIMESTAMP '${ts_start}' <= "ts" AND "ts" < TIMESTAMP '${ts_end}'
+```
+3. Add a separate section for `checks for your_dataset_name [filter name]`. Any checks you nest under this header execute *only* against the portion of data that the expression in the filter section defines. Refer to the example below.
+4. Write any checks you wish for the dataset and the columns in it.
+```yaml
+checks for CUSTOMERS [daily]:
+  - row_count = 6
+  - missing(cat) = 2
+```
+5. When you wish to execute the checks, use Soda Core to run a scan of your data source and use the `-v` option to include the values for the variables you included in your filter expression, as in the example below. 
+```shell
+soda scan -d snowflake_customer_data -v ts_start=2022-03-11 ts_end=2022-03-15 checks.yml
+```
+
+If you wish to run checks on the same dataset *without* using a filter, add a separate section for `checks for your_dataset_name` without the appended filter name. Any checks you nest under this header execute against all the data in the dataset. 

_includes/dataset-filters.md

@@ -0,0 +1,27 @@
+Add a **for each** section to your checks YAML file to specify a list of checks you wish to execute on multiple datasets. 
+
+1. Add a `for each table T` section header anywhere in your YAML file. The purpose of the `T` is only to ensure that every `for each` configuration has a unique name. 
+2. Nested under the section header, add two nested keys, one for `tables` and one for `checks`. 
+3. Nested under `tables`, add a list of datasets against which to run the checks. Refer to the example below that illustrates how to use `include` and `exclude` configurations and wildcard characters {% raw %} (%) {% endraw %}.
+4. Nested under `checks`, write the checks you wish to execute against all the datasets listed under `tables`. 
+
+```yaml
+for each table T:
+  tables:
+    # include the dataset 
+    - dim_customers
+    # include all datasets matching the wildcard expression
+    - dim_products%
+    # (optional) explicitly add the word include to make the list more readable
+    - include dim_employee
+    # exclude a specific dataset
+    - exclude fact_survey_response
+    # exclude any datasets matching the wildcard expression
+    - exclude prospective_%
+  checks:
+    - row_count > 0
+```
+
+* Soda Core dataset names matching is case insensitive.
+* If any of your checks specify column names as arguments, make sure the column exists in all datasets listed under the `tables` heading.
+* To add multiple for each configurations in your checks YAML file, configure another `for each` section header with a different letter identifier, such as `for each table R`.

_includes/foreach-config.md

@@ -0,0 +1,11 @@
+```yaml
+ = 
+ < 
+ >
+ <=
+ >=
+ !=
+ <> 
+ between 
+ not between 
+```

_includes/list-symbols.md

@@ -0,0 +1,52 @@
+---
+layout: default
+title: Anomaly score check
+description: 
+parent: Soda CL 
+---
+
+# Anomaly score check ![beta](/assets/images/beta.png){:height="50px" width="50px" align="top"}
+
+The anomaly score check is powered by a machine learning algorithm that works with measurements that occur over time. The algorithm learns the patterns of your data – its trends and seasonality – to identify and flag anomalous measurements in time-series data. 
+
+If you have connected Soda Core to a Soda Cloud account, Soda Core pushes check results to your cloud account where Soda Cloud stores all the historic measurements for your checks in a metric store. SodaCL can then use these stored values to establish a baseline of normal measurements against which to evaluate future measurements to identify anomalies. Therefore, you must have a [Soda Cloud account]({% link soda-cloud/overview.md%}) to use change-over-time thresholds.
+
+## Prerequisites
+* a Soda Cloud account, connected to Soda Core
+* Soda Core Scientific package installed
+
+
+## Configuration
+
+The following example demonstrates how to use the anomaly score for the `row_count` metric in a check. You can use any numeric metrics in lieu of `row_count`. By default, anomaly score checks yield warning check results, not failures.
+
+```yaml
+checks for CUSTOMERS:
+  - anomaly score for row_count < default
+```
+<br />
+
+If you wish, you can override the anomaly score. <!--why would you want to do this? what is the .7 a portion of?--> The following check yields a warning check result if the anomaly score for `row_count` exceeds `.7`.
+
+```yaml
+checks for CUSTOMERS:
+  - anomaly score for row_count < .7
+```
+
+<br />
+
+Further, you can use `warn` and `fail` thresholds with the anomaly score. The following example demonstrates how to define the threshold for `row_count` that yields a warning, and the threshold that yields a failed check result. Note that an individual check only ever yields one check result. If your check triggers both a `warn` and a `fail`, the check result only displays the more serious, failed check result. 
+```yaml
+checks for CUSTOMERS:
+  - anomaly score for row_count:
+      warn: when > .8
+      fail: when > .99
+```
+
+## Go further
+
+* Need help? Join the <a href="http://community.soda.io/slack" target="_blank"> Soda community on Slack</a>.
+<br />
+
+---
+{% include docs-footer.md %}

assets/images/check-result.png

@@ -0,0 +1,15 @@
+---
+layout: default
+title: dataset filters
+description: Instead of checking whole tables, you can use SodaCL (Beta) table filters to specify a portion of data in a table against which Soda Core executes a check.
+parent: SodaCL (Beta)
+redirect_from:
+- soda-cl/table-filters.html
+---
+
+# Dataset filters ![beta](/assets/images/beta.png){:height="50px" width="50px" align="top"}
+
+{% include dataset-filters.md %}
+
+---
+{% include docs-footer.md %}

assets/images/check.png

@@ -5,34 +5,10 @@ description: Use a SodaCL (Beta) for each check to specify a list of checks you
 parent: SodaCL (Beta)
 ---
 
-# For each checks ![beta](/assets/images/beta.png){:height="50px" width="50px" align="top"}
+# For each ![beta](/assets/images/beta.png){:height="50px" width="50px" align="top"}
 
-Use a for each check to specify a list of checks you wish to execute on a multiple tables. 
 
-First, in your checks.yml file, specify the list of tables using `for each table T`. The purpose of the `T` is only to ensure that every `for each` check has a unique name. Next, write the checks you wish to execute against the tables.
-
-```yaml
-for each table T:
-  tables:
-    # include the table 
-    - CUSTOMERS
-    # include all tables matching the wildcard expression
-    - new_%
-    # (optional) explicitly add the word include to make the list more readable
-    - include CUSTOMERS
-    # exclude a specific table
-    - exclude fact_survey_response
-    # exclude any tables matching the wildcard expression
-    - exclude prospective_%
-  checks:
-    - row_count > 0
-```
-
-### Notes
-
-* Soda Core resolves all table names in the scan's default data source.
-* You can use `%` as a wildcard in both data source name and table name filters.
-* Soda Core table names matching is case insensitive.
+{% include foreach-config.md %}
 
 ---
 {% include docs-footer.md %}