[SDP] FlowAnalysis and Analyze Logical Command

Author: jaceklaskowski

docs/declarative-pipelines/FlowAnalysis.md

@@ -16,11 +16,11 @@ createFlowFunctionFromLogicalPlan(
   plan: LogicalPlan): FlowFunction
 ```
 
-`createFlowFunctionFromLogicalPlan` takes a [LogicalPlan](../logical-operators/LogicalPlan.md) and creates a [FlowFunction](FlowFunction.md).
+`createFlowFunctionFromLogicalPlan` takes a [LogicalPlan](../logical-operators/LogicalPlan.md) (that represents one of the supported logical commands) and creates a [FlowFunction](FlowFunction.md).
 
 When [executed](FlowFunction.md#call), this `FlowFunction` creates a [FlowAnalysisContext](FlowAnalysisContext.md).
 
-`FlowFunction` uses this `FlowAnalysisContext` to [setConf](#setConf) the given SQL configs (to the [FlowFunction](FlowFunction.md#call)).
+`FlowFunction` uses this `FlowAnalysisContext` to [set the SQL configs](FlowAnalysisContext.md#setConf) (given to the [FlowFunction](FlowFunction.md#call) being defined).
 
 `FlowFunction` [analyze](#analyze) this `LogicalPlan` (with the `FlowAnalysisContext`). This gives the result data (as a `DataFrame`).
 
@@ -37,3 +37,31 @@ In the end, `FlowFunction` creates a [FlowFunctionResult](FlowFunctionResult.md)
     * [CreateView](SqlGraphRegistrationContext.md#CreateView)
     * [CreateStreamingTableAsSelect](SqlGraphRegistrationContext.md#CreateStreamingTableAsSelect)
     * [CreateViewCommand](SqlGraphRegistrationContext.md#CreateViewCommand)
+
+### Analyze Logical Command { #analyze }
+
+```scala
+analyze(
+  context: FlowAnalysisContext,
+  plan: LogicalPlan): DataFrame
+```
+
+!!! note "CTEs"
+    `analyze` resolves pipeline-specific TVFs and CTEs.
+
+    ```sql
+    SELECT ... FROM STREAM(t1)
+    SELECT ... FROM STREAM t1
+    ```
+
+    Developers can define CTEs within their CREATE statements:
+
+    ```sql
+    CREATE STREAMING TABLE a
+    WITH b AS (
+       SELECT * FROM STREAM upstream
+    )
+    SELECT * FROM b
+    ```
+
+`analyze`...FIXME

docs/declarative-pipelines/PipelinesHandler.md

@@ -166,13 +166,16 @@ defineFlow(
 
 `defineFlow` looks up the [GraphRegistrationContext](DataflowGraphRegistry.md#getDataflowGraphOrThrow) for the given `flow` (or throws a `SparkException` if not found).
 
-!!! note "Implicit Flows"
-    An **implicit flow** is a flow with the name of the target dataset (i.e. one defined as part of dataset creation).
+??? note "Implicit Flows"
+    **Implicit Flows** are flows with the name of the target datasets (i.e. one defined as part of dataset creation).
+
+    Implicit flows can be defined with multi-part identifiers (as the corresponding datasets).
+
+    Multi-part identifiers are composed of catalog, schema and table parts (separated by `.` (dot)).
 
 `defineFlow` [creates a flow identifier](GraphIdentifierManager.md#parseTableIdentifier) (for the `flow` name).
 
 ??? note "AnalysisException"
     `defineFlow` reports an `AnalysisException` if the given `flow` is not an implicit flow, but is defined with a multi-part identifier.
 
-In the end, `defineFlow` [registers a flow](GraphRegistrationContext.md#registerFlow).
-
+In the end, `defineFlow` [registers a flow](GraphRegistrationContext.md#registerFlow) (with a proper [FlowFunction](FlowAnalysis.md#createFlowFunctionFromLogicalPlan)).