Fluxly — Lightweight framework for portable, self-contained DAG workflows, decoupled from orchestration.
Fluxly is a lightweight framework for building and running directed acyclic graph (DAG)-based workflows.
The entire workflow acts as a self-contained execution endpoint:
- Run them locally or via CLI commands, API calls, or environment triggers.
- Package them in containers for portability.
- Integrate seamlessly with higher-level orchestrators (Argo, Airflow, Prefect, CI/CD) without extra glue code.
With Fluxly, pipelines are highly structured, enabling safer execution, easier debugging, and better modularity.
Workflows can run standalone or as part of a larger system, making Fluxly both lightweight and flexible.
- Unstructured container pipelines tend to become spaghetti: ad‑hoc scripts across containers, inconsistent inputs/outputs, and scattered retries/timeouts/logging with no shared wrapper.
- Heavyweight orchestrators like Airflow add operational burden (schedulers, DBs, webservers, DAG deployment) when all you need is a simple, portable workflow image.
- Orchestrator‑coupled SDKs (e.g., Prefect) optimize for remote control planes and persistent backends, introducing communication channels and runtime coupling that don’t fit autonomous, fire‑and‑forget jobs.
- Fluxly keeps logic structured and isolated inside a single codebase and container: typed I/O models, explicit DAG, uniform entrypoints (CLI/API/env), and clear node boundaries. Any scheduler can trigger it, but your workflow remains clean and portable.
- Best when each Docker stays simple and self‑sufficient, and you want clarity, low overhead, and no hidden glue.
- In monorepos—or by wrapping Fluxly—you can centralize shared services, typed inputs, validations, outputs, and metadata to fit your org standards. This thin wrapper standardizes containers and removes boilerplate across pipelines.
The package automatically exposes a CLI command and API endpoints for each registered workflow.
Auto-generated CLI commands per workflow.
Auto-generated API endpoints per workflow in Swagger UI.
Comprehensive documentation for Fluxly is available online:
- View the docs: GitHub Pages
- Flexible entrypoints – workflows can be triggered via CLI, API calls, or environment variables.
- DAG-based workflows – define arbitrary connections between nodes and their dependencies.
- Highly structured workflows – strict validation ensures safer pipelines, easier debugging, and predictable behavior.
- Self-orchestrated nodes – each node manages its own execution, retries, and dependencies.
- Lightweight building blocks – workflows are self-contained units that can run independently in any environment.
- Extensible by design – wrap workflows with custom classes to add logging, metrics, or integrations.
- Local-first development – debug and run workflows standalone, then scale seamlessly to CI/CD or external orchestrators.
- Pydantic: Strict data validation and schema for inputs/outputs.
- Typer: Easy wrapper for building great CLIs.
- FastAPI: Modern, high-performance web framework for building APIs.
- Uvicorn: Lightning-fast ASGI server for running FastAPI apps.
- Loguru: Simple, efficient logging for Python.
- Diagrams: Auto-generate DAG diagrams as code.
- Pixi: Cross-platform package and environment management.
- Ruff: Lightning-fast linting.
- Mypy: Static type checking for Python, enforcing type safety and correctness.
- Coverage.py: Measures code coverage of tests for quality assurance.
- Pytest: Testing framework.
- MkDocs: Documentation generator for Markdown-based docs.
- Material for MkDocs: Modern, responsive theme for MkDocs.
- GitHub CI/CD: Continuous integration and deployment.
Install with pip:
pip install fluxly