Skip to content

Latest commit

 

History

History
6 lines (6 loc) · 394 Bytes

reference-with-spark.md

File metadata and controls

6 lines (6 loc) · 394 Bytes

If you are using reference checks with a Spark or Databricks data source to validate the existence of values in two datasets within the same schema, you must first convert your DataFrames into temp views to add them to the Spark session, as in the following example.

# after adding your Spark session to the scan
df.createOrReplaceTempView("df")
df2.createOrReplaceTempView("df2")