End-to-End MLOps Platform with Lakehouse Architecture

Project Overview & Motivation

Topic and Overview

This project demonstrates the implementation of a robust, fault-tolerant MLOps Platform based on the Data Lakehouse Architecture. The primary goal is to bridge the gap between Big Data engineering and Machine Learning operations (MLOps) by automating the flow of data from ingestion to model deployment.

The platform simulates a real-world scenario: detecting fraud in high-volume e-commerce transactions. It ingests raw transaction streams, processes them through a multi-hop "Medallion" architecture (Bronze/Silver), trains a Machine Learning model, and deploys it as a scalable REST API.

Relevance and Target Architecture

In modern Big Data environments, training a model is only a small part of the lifecycle. The bigger challenge lies in building systems that are:

• Scalable: Capable of handling increasing data volumes.

• Reproducible: Ensuring data and model lineage.

• Fault-Tolerant: Resilient to component failures.

To address these challenges, this project leverages Kubernetes as the underlying orchestration layer, ensuring high availability and self-healing capabilities. The system implements a Lakehouse Architecture, combining the flexibility of data lakes (MinIO) with the reliability of data warehouses (Delta Lake ACID transactions).

---

System Architecture

The system follows a modular, microservices-based architecture deployed entirely on Kubernetes. The result is following image:

Components

1. Data Source (Fraud Simulation): A Dockerized Python producer that simulates a continuous stream of e-commerce transactions. It generates synthetic data (valid and fraudulent transactions) and pushes it to the ingestion layer.

2. Ingestion Layer (Apache Kafka): Acts as the high-throughput buffer for incoming data. It decouples the producer from the processing layer, allowing for asynchronous processing and preventing backpressure issues.

3. Orchestration (Apache Airflow): Manages the workflow dependency graph (DAGs). Airflow triggers Spark jobs on Kubernetes using the KubernetesPodOperator, ensuring that each processing step runs in an isolated, clean environment.

4. Processing Layer (Apache Spark): The core compute engine. Spark jobs perform ETL (Extract, Transform, Load) and model training:

   • Bronze Layer: Raw ingestion of data from Kafka to Delta Lake.
   • Silver Layer: Data cleaning, feature engineering, and joining features with labels.
   • Note: The "Gold Layer" is depicted in the architecture for completeness of the Medallion pattern but is conceptually merged into the final training data preparation in this implementation.

5. Storage Layer (MinIO + Delta Lake):

   • MinIO: An S3-compatible object storage server that acts as the centralized Data Lake.
   • Delta Lake: An open-source storage layer that brings ACID transactions (reliability) to the data lake, allowing time travel and schema enforcement.

6. MLOps Layer (MLflow):

   • Tracking Server: Logs model parameters, metrics (e.g., AUC, accuracy), and artifacts (trained models).
   • Model Registry: Manages model versions (e.g., staging vs. production).
   • Model Serving: Deploys the trained model as a REST API endpoint.

Scalability & Fault Tolerance

• Kubernetes: All components (Airflow, Spark, Kafka, MLflow) run as Pods. K8s handles automatic restarting of failed containers and load balancing.
• Spark: Distributed processing allows the system to scale horizontally by adding more executor pods to process larger datasets.
• Kafka: Partitions allow the data stream to be consumed in parallel, ensuring high throughput.

Implementation & Repository Structure

The project is structured to separate infrastructure (IaC), workflow definitions, and application logic.

Repo Structure

.
├── dags/                       # Airflow DAGs (Workflow definitions)
│   └── ingest_labels_dag.py    # Main pipeline definition
├── infrastructure/             # Infrastructure as Code (IaC)
│   ├── helm/                   # Helm Charts for Kubernetes deployment
│   │   ├── platform/           # Umbrella chart connecting all services
│   │   └── producer-chart/     # Custom chart for the data generator
│   └── k8s/                    # Raw Kubernetes manifests (PVCs, Model Deployment)
├── services/                   # Source code for application microservices
│   ├── fraud-simulation/       # Python code for generating dummy data
│   ├── mlflow/                  # Custom Dockerfile to overcome some dependency issues
│   └── spark-jobs/             # PySpark scripts for ETL and Training
├── Makefile                     # Automation shortcuts for setup and deployment
└── README.md                   # Project documentation

Setup and Usage

This project uses a Makefile to simplify complex Kubernetes commands. Follow these steps to spin up the entire platform.

Prerequisites

• Docker Desktop (or Docker Engine)
• Minikube (Kubernetes Cluster)
• Helm (Package Manager for K8s)
• Kubectl (K8s CLI)

Start Instructions

1. Initialize the Cluster: Starts Minikube with sufficient resources (14GB RAM, 8 CPUs recommended for the full stack).

   make setup

2. Mount Data (Important): Keeps the Airflow DAGs synced between your host and the cluster.

   make mount #(Keep this terminal open)

3. Deploy the Platform: This command builds local Docker images (Producer, Spark, MLflow), updates Helm dependencies, creates Kubernetes secrets/storage, and deploys the Helm charts.

   make deploy

4. Access User Interfaces: Open separate terminals to forward ports for the various UIs:

   make airflow-ui  # Access Airflow at localhost:8080
   make minio-ui    # Access MinIO Storage at provided IP
   make mlflow-ui   # Access MLflow Tracking at localhost:5000

5. Running the MLOps Pipeline Trigger the Pipeline: Go to the Airflow UI, unpause the feature_engineering_pipeline DAG, and trigger it manually.

6. Deploy Model Server: Once a model is registered in MLflow, deploy the serving pod:

   make serve

7. Test Prediction: Forward the serving port and send a dummy transaction request:

   # terminal session 1
   make forward-serving
   # terminal session 2
   make predict

Evaluation & Lessons Learned

Challenges

• Helm Chart Instability (Bitnami Legacy): A major pain point was the reliance on public Helm charts. During the development of this project, Bitnami—the standard provider for many open-source Kubernetes charts—fundamentally restructured their public catalog (effective August 2025). This introduced significant instability. Bitnami deprecated support for non-hardened, Debian-based images in their public tier. All existing versioned tags were migrated to a frozen "Bitnami Legacy" repository (docker.io/bitnamilegacy) which no longer receives updates. To overcome this issues it was required to dive deeply into the Charts and to replace the images from Bitnami charts with custom iamges or legacy images. This also opened issues in regards to versioning.

• Spark and Dependency Management: Building the Docker image for Spark jobs proved to be the most labor-intensive aspect of the project, primarily to align Spark, Java, Scala, and Hadoop versions. Unlike modern package managers that automatically resolve transitive dependencies, configuring Spark to communicate with external systems like Kafka and MinIO (S3) which requires to resolve dependencies manually. For instance, enabling Spark 3.5(dropped due to Bitnami Legacy) to talk to Kafka necessitated identifying the exact spark-sql-kafka JAR that matched the specific kafka-clients library, which in turn relied on a precise version of commons-pool. This lack of automatic dependency resolution forced a trial-and-error approach where specific JAR files had to be manually downloaded and placed into the image's class path, often requiring version downgrades to find a compatible combination. This tedious, error-prone process vividly illustrates why enterprise environments heavily favor managed solutions like Databricks over self-managed Spark deployments.

• Modern Python Management with uv: In contrast to the Java/Scala dependency struggles, Python dependency management was significantly streamlined by adopting uv. This tool proved to be a critical enabler, allowing for fast and reliable Python environment creation. Crucially, it solved the problem of outdated system Python versions in the legacy Spark images. By using uv, it was possible to easily inject and manage a modern Python virtual environment (venv) with newer Python versions and libraries directly inside the older, constrained Spark container images, ensuring the usage of MLflow 3.0.

Conclusion

This project demonstrates that while a Lakehouse Architecture on Kubernetes offers immense power and flexibility, the "Do It Yourself" approach requires significant DevOps overhead. The complexity of maintaining compatible versions across distributed systems (Kafka, Spark, K8s) is the primary cost of avoiding managed cloud services. It is also important to keep in mind that this only shows a simple model training. In a bigger scale you also need an Feature Store like Feast with Redis underneath and a streaming path to calculate realtime features. The current approach also misses many things like a catalog format or a virtualization layer to make the whole architecture also accessible to Analysts and Data Scientist.