add docs for uv (#146)

tuulos · web-flow · commit e8240e56d1b5 · 2025-05-21T11:47:14.000-07:00
diff --git a/docs/index.md b/docs/index.md
@@ -33,13 +33,13 @@ Metaflow makes it easy to build and manage real-life data science, AI, and ML pr
 - [Managing Flows in Notebooks and Scripts](metaflow/managing-flows/introduction) 
 - [Debugging Flows](metaflow/debugging)
 - [Visualizing Results](metaflow/visualizing-results/) 
-- [Configuring Flows](metaflow/configuring-flows/introduction) ✨*New*✨
+- [Configuring Flows](metaflow/configuring-flows/introduction)
 
 ## II. Scaling Flows
 
 - [Introduction to Scalable Compute and Data](scaling/introduction)
 - [Computing at Scale](scaling/remote-tasks/introduction) 
-- [Managing Dependencies](scaling/dependencies) 
+- [Managing Dependencies](scaling/dependencies) ✨*New support for `uv`*✨
 - [Dealing with Failures](scaling/failures)
 - [Checkpointing Progress](scaling/checkpoint/introduction) ✨*New*✨
 - [Loading and Storing Data](scaling/data)
diff --git a/docs/scaling/dependencies/README.md b/docs/scaling/dependencies/README.md
@@ -7,10 +7,15 @@ If you are in a hurry:
 - Want to use your own modules and packages in a flow?
   See [Structuring Projects](/scaling/dependencies/project-structure).
 - Want to use a Python library in a flow? See [Managing Libraries](/scaling/dependencies/libraries).
+- Want to use `uv` with Metaflow? See [using `uv`](/scaling/dependencies/uv)
 - Want to use or build a specific Docker image?
   See [Defining Custom Containers](/scaling/dependencies/containers).
 :::
 
+:::info
+New in Metaflow 2.15.8: You can [use `uv` to manage dependencies](/scaling/dependencies/uv).
+:::
+
 A Metaflow flow is defined in a single Python file. Besides the flow code
 itself, a typical flow has many *software dependencies*, that is, other
 [modules](https://docs.python.org/3/tutorial/modules.html),
@@ -98,9 +103,10 @@ python myflow.py package list
 By the default, the package only includes local Python files, not libraries you
 have installed e.g. with `pip install` manually. To include external libraries,
 you need to include them either in [a custom Docker
-image](/scaling/dependencies/containers) or [specify them in a `@pypi` or `@conda`
-decorator](/scaling/dependencies/conda-vs-pypi). This figure illustrates the
-idea:
+image](/scaling/dependencies/containers), [specify them in a `@pypi` or `@conda`
+decorator](/scaling/dependencies/conda-vs-pypi), or [use `uv`](/scaling/dependencies/uv).
+
+This figure illustrates the idea:
 
 ![Remote dependencies](/assets/dependencies-remote.png)
 
diff --git a/docs/scaling/dependencies/conda-vs-pypi.md b/docs/scaling/dependencies/conda-vs-pypi.md
@@ -19,7 +19,7 @@ Let's start by outlining the two major components of the package ecosystem:
 
 2. **A package manager** - a tool that downloads packages from a
    repository and installs them locally. The tools are often specific
-   to a repository. For instance, `pip`, `poetry` and `pipenv` work
+   to a repository. For instance, `uv`, `pip`, `poetry` and `pipenv` work
    with PyPI and `mamba` and `conda` with Conda.
 
 Much confusion arises from the fact that there are many *package managers*,
@@ -32,6 +32,9 @@ Metaflow works with both the *repositories* through [the
 `@pypi` and `@conda`](/scaling/dependencies/libraries) decorators.
 Importantly, when using these decorators, you don't need to use
 *package managers* manually as Metaflow acts as a package manager by itself.
+Alternatively, an increasily popular approach in the Python ecosystem is
+to use `uv` as a package and project manager - and [Metaflow
+integrates with this approach natively](/scaling/dependencies/uv).
 
 ## Virtual environments
 
@@ -43,7 +46,7 @@ are a concept closely related to package managers. In
  environment.
 
 Some package managers like `pip` don't manage virtual environments but they rely
-on another tool, such as `venv` or `pyenv`, to do the job. Others like
+on another tool, such as `uv`, `venv` or `pyenv`, to do the job. Others like
  `poetry` and `mamba` handle virtual environments natively, allowing you to
  create and delete environments as needed.
 
diff --git a/docs/scaling/dependencies/libraries.md b/docs/scaling/dependencies/libraries.md
@@ -6,12 +6,21 @@ packages](/scaling/dependencies/project-structure), this page covers handling
 of 3rd party dependencies that are published as installable Python packages.
 
 Metaflow supports installation of external packages from two Python package
-repositories: [PyPI](https://pypi.org/) and [Conda](https://conda-forge.org/),
-using `@pypi` and `@conda` decorators. If you wonder why we need two
-decorators and when to use which, see
-[Conda vs. PyPI](/scaling/dependencies/conda-vs-pypi). To learn why you
-should use the decorators and not install packages 
-manually, see [Packaging Internals](/scaling/dependencies/internals).
+repositories: [PyPI](https://pypi.org/) and [Conda](https://conda-forge.org/) - learn
+more about their relative benefits in [Conda vs. PyPI](/scaling/dependencies/conda-vs-pypi).
+You can access these package repositories using `@pypi` and `@conda`
+decorators, or you can use [the `uv` package manager](https://github.com/astral-sh/uv)
+to manage PyPI dependencies in a Metaflow project similar to any other Python project.
+
+Currently, the `@pypi` and `@conda` decorators provide the strongest guarantees
+on stability, scalability, and reproducibility of environments, as they snapshot
+all dependencies in the object store, as described in
+[Packaging Internals](/scaling/dependencies/internals). Also, they allow
+each step to specify a separate set of dependencies. However, you can get started
+easily with `uv` - in particular if you are already using it for other Python projects.
+
+This page focuses on the Metaflow's `@pypi` and `@conda` decorators. If you want to
+use `uv` instead, see [using `uv`](/scaling/dependencies/uv).
 
 ## The `@pypi` and `@conda` decorators
 
diff --git a/docs/scaling/dependencies/uv.md b/docs/scaling/dependencies/uv.md
@@ -0,0 +1,48 @@
+
+# Using uv
+
+[`uv` is a fast, modern Python package manager](https://github.com/astral-sh/uv) designed for creating reproducible environments with minimal overhead. In Metaflow, you can use projects managed by `uv` directly by
+specifying `--environment=uv`, enabling you to manage dependencies the same way you would in any contemporary Python project. This makes it easy to develop locally and seamlessly run flows in the cloud with all dependencies included.
+
+`uv` takes a more holistic approach to package and project management in contrast to `conda` and `pip`. As a result,
+when using `--environment=uv`, Metaflow lets `uv` install and handle dependencies using its native approach.
+A downside of this is that `uv` doesn't benefit from [Metaflow's internal package snapshotting](/scaling/dependencies/internals) which provides stronger guarantees for scalability and availability of packages, which matters especially
+when running tasks at large scale.
+
+:::tip
+`uv` is often a convenient starting point. If you run into any issues at scale, you can always switch to `@pypi`
+or `@conda` later.
+:::
+
+## Setting up a Metaflow project with `uv`
+
+Metaflow works with `uv` as any other Python library, so you can use `uv` as usual. Here's an example:
+
+1. Initialize a project: `uv init myflow; cd myflow`.
+
+2. Add packages you need, including Metaflow: `uv add pandas metaflow`.
+
+3. Run a flow `uv run flow.py run`.
+
+## Running tasks remotely
+
+Metaflow will package and ship the `uv` project in the cloud when you specify `--environment=uv`:
+
+```
+uv run flow.py --environment=uv run
+```
+Or, more conveniently, you can set an environment variable
+```
+export METAFLOW_ENVIRONMENT=uv
+```
+so you don't need to add the command line option in every execution.
+
+After this, you can use Metaflow's compute layer to [run the flow in the cloud](/scaling/remote-tasks/requesting-resources) with all the packages defined in your `uv` environment included, e.g. [using Kubernetes](/scaling/remote-tasks/kubernetes) ([or `--with batch`](/scaling/remote-tasks/aws-batch)):
+
+```
+uv run flow.py run --with kubernetes
+```
+
+Remember to set the `environment=uv` when [deploying to production](/production/introduction) as well.
+
+
diff --git a/sidebars.js b/sidebars.js
@@ -163,6 +163,7 @@ const sidebars = {
           items: [
             "scaling/dependencies/project-structure",
             "scaling/dependencies/libraries",
+            "scaling/dependencies/uv",
             "scaling/dependencies/conda-vs-pypi",
             "scaling/dependencies/containers",
             "scaling/dependencies/internals",
@@ -257,6 +258,7 @@ const sidebars = {
             "api/step-decorators/card",
             "api/step-decorators/catch",
             "api/step-decorators/conda",
+            "api/step-decorators/pypi",
             "api/step-decorators/kubernetes",
             "api/step-decorators/resources",
             "api/step-decorators/retry",
