|
| 1 | +# WIP - Spark Describe Function with MLRun (non-sparkoperator) |
| 2 | + |
| 3 | +## Run .py file Using Spark |
| 4 | +### Steps: |
| 5 | +1. Deploy spark-operator on the cluster (create service from dashboard). |
| 6 | + This is required at this stage in order to create a configmap for the daemon. |
| 7 | +2. In Jupyter: |
| 8 | + Save the followin code under my.py in fuse (in this case /v3io/users/admin/my.py): |
| 9 | + |
| 10 | +```python |
| 11 | +#!/usr/local/bin/python |
| 12 | + |
| 13 | +# Locate v3iod: |
| 14 | +from subprocess import run |
| 15 | +run(["/bin/bash", "/etc/config/v3io/v3io-spark-operator.sh"]) |
| 16 | + |
| 17 | +# The pyspark code: |
| 18 | +import os |
| 19 | +from pyspark.sql import SparkSession |
| 20 | +os.environ["PYSPARK_SUBMIT_ARGS"] = "--packages mysql:mysql-connector-java:5.1.39 pyspark-shell" |
| 21 | + |
| 22 | +spark = (SparkSession.builder.appName("Spark JDBC to Databases - ipynb") |
| 23 | + .config("spark.driver.extraClassPath", "/v3io/users/admin/mysql-connector-java-5.1.45.jar") |
| 24 | + .config("spark.executor.extraClassPath", "/v3io/users/admin/mysql-connector-java-5.1.45.jar") |
| 25 | + .getOrCreate()) |
| 26 | + |
| 27 | +dfMySQL = (spark.read.format("jdbc") |
| 28 | + .option("url", "jdbc:mysql://mysql-rfam-public.ebi.ac.uk:4497/Rfam") |
| 29 | + .option("dbtable", "Rfam.family") |
| 30 | + .option("user", "rfamro") |
| 31 | + .option("password", "") |
| 32 | + .option("driver", "com.mysql.jdbc.Driver") |
| 33 | + .load()) |
| 34 | + |
| 35 | +dfMySQL.write.format("io.iguaz.v3io.spark.sql.kv").mode("overwrite").option("key", "rfam_id").save("v3io://users/admin/frommysql") |
| 36 | + |
| 37 | +spark.stop() |
| 38 | +``` |
| 39 | + |
| 40 | +3. Make sure that your script has execution permissions. |
| 41 | +4. Execute the following block in a notebook: |
| 42 | + |
| 43 | +```python |
| 44 | +from mlrun import new_function |
| 45 | +from mlrun.platforms.iguazio import mount_v3io, mount_v3iod |
| 46 | +import os |
| 47 | +image_name = 'iguazio/shell:' + os.environ.get("IGZ_VERSION") |
| 48 | +run = new_function(name='my-spark', image=image_name , command='/v3io/users/admin/my.py', kind='job', mode='pass') |
| 49 | +run.apply(mount_v3io(name="v3io-fuse", remote="/", mount_path="/v3io")) |
| 50 | +run.apply(mount_v3iod(namespace="default-tenant", v3io_config_configmap="spark-operator-v3io-config")) |
| 51 | +run.run(artifact_path="/User/artifacts") |
| 52 | +``` |
| 53 | +--- |
| 54 | + |
| 55 | +## Create Simple Read CSV Function Using Spark |
| 56 | +Please refer to the read_csv_spark notebook |
| 57 | + |
| 58 | +--- |
| 59 | + |
| 60 | +## Create Describe Function Using Spark |
| 61 | +Generates profile reports from an Apache Spark DataFrame. |
| 62 | +Based on pandas_profiling, but for Spark's DataFrames instead of pandas. |
| 63 | + |
| 64 | +For each column the following statistics - if relevant for the column type - are presented: |
| 65 | + |
| 66 | +* `Essentials:` type, unique values, missing values |
| 67 | +* `Quantile statistics:` minimum value, Q1, median, Q3, maximum, range, interquartile range |
| 68 | +* `Descriptive statistics:` mean, mode, standard deviation, sum, median absolute deviation, coefficient of variation, kurtosis, skewness |
| 69 | +* `Most frequent values:` for categorical data |
| 70 | + |
| 71 | +``` |
| 72 | +Function params |
| 73 | +
|
| 74 | +:param context: Function context. |
| 75 | +:param dataset: Raw data file (currently needs to be a local file located in v3io://User/bigdata) |
| 76 | +:param bins: Number of bin in histograms |
| 77 | +:param describe_extended: (True) set to False if the aim is to get a simple .describe() infomration |
| 78 | +``` |
| 79 | + |
| 80 | +* All operations are done efficiently, which means that **no** Python UDFs or .map() transformations are used at all; |
| 81 | +* only Spark SQL's Catalyst is used for the retrieval of all statistics. |
| 82 | + |
| 83 | +--- |
| 84 | +### TODO: |
| 85 | +1. Add plots |
| 86 | +2. Add ability to generte html report |
0 commit comments