mlflow-project-best-practices

An example MLFlow project

High Level Architecture

You can Notebook see the Databricks code here: https://github.com/noahgift/mlflow-project-best-practices/blob/main/XGBoost-fake-news-automl.ipynb

Example in Azure CloudShell

Example in Github Codespaces

Curl Example

curl \
  -u token:$DATABRICKS_TOKEN \
  -X POST \
  -H "Content-Type: application/json; format=pandas-records" \
  [email protected] \
  https://adb-2951765055089996.16.azuredatabricks.net/model/Fake-News/1/invocations

Deploy Architecture

Query registered model

• Azure Databricks Model Serving Architecture

Query with Databricks CLI

• Databricks CLI

1. Create a databricks config: touch ~/.databrickscfg
2. Put in host and token
3. Query jobs

databricks jobs list --output JSON | jq 4. List clusters

databricks clusters list --output JSON | jq

5. List contents of DBFS

databricks fs ls dbfs:/

Query for Models with API

• Refer to Quikstart Python
• Refer to Create or Register Model
• Access tracking server external

You need to set the tracking URI.

export MLFLOW_TRACKING_URI=databricks

from pprint import pprint
from mlflow.tracking import MlflowClient
client = MlflowClient()
for rm in client.list_registered_models():
  pprint(dict(rm), indent=4)

Download Model

• Download model artifacts

CLI version

mlflow artifacts download --artifact-uri models:/<name>/<version|stage>

To use Python do the following:

from mlflow.store.artifact.models_artifact_repo import ModelsArtifactRepository

model_uri = MlflowClient.get_model_version_download_uri(model_name, model_version)
ModelsArtifactRepository(model_uri).download_artifacts(artifact_path="")

Register model for AWS Sagemaker

• Register model for AWS Sagemaker

Goal for today

• Databricks Tensorflow run
• Feature Store
• Model Serving

Model Serving

https://docs.databricks.com/applications/mlflow/model-serving.html

Run it with mlserve

 mlflow models serve --model-uri  /workspaces/mlflow-project-best-practices/tf-model

References

• Watch Youtube Walkthrough
• Watch Walkthrough on O'Reilly