Skip to content

Convert to using uv #1089

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 29 commits into
base: main
Choose a base branch
from
Open

Convert to using uv #1089

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
e320979
Use dependency-group instead of optional-dependencies
tpoliaw Jun 26, 2025
0dfb460
Specify python version upper bound
tpoliaw Jun 26, 2025
091c6c4
Initial workflow rewrite
tpoliaw Jun 27, 2025
9a4050a
WIP Convert dockerfile to use uv
tpoliaw Jun 30, 2025
a3c6058
Remove dev-requirements.txt
tpoliaw Jul 1, 2025
8ba2727
Remove pip install from devcontainer config
tpoliaw Jul 1, 2025
c229d70
Restore postCreateCommand to set up uv environment
tpoliaw Jul 1, 2025
335dd05
Add mount for uv cache
tpoliaw Jul 1, 2025
9e37240
Mount venv in workspace
tpoliaw Jul 4, 2025
e081661
Fix CI
tpoliaw Jul 4, 2025
f47427a
Add shell back to install_requirements
tpoliaw Jul 4, 2025
cf38536
Update ophyd -> ophyd-async
tpoliaw Jul 4, 2025
d69f43f
Move checkout out of install action
tpoliaw Jul 4, 2025
acb1741
Revert change to using uv base image
tpoliaw Jul 7, 2025
6f7af1c
Remove hardcoded project name from devcontainer
tpoliaw Jul 8, 2025
c989b4d
Restrict httpx to <1.0 to avoid breaking changes in pre-release version
tpoliaw Aug 21, 2025
1caa29e
Install requirements in _dist workflow
tpoliaw Sep 2, 2025
8579853
Update uv.lock
tpoliaw Sep 3, 2025
261e064
Attempt to fix dist workflow
tpoliaw Sep 3, 2025
8784091
Update uv.lock. Again
tpoliaw Sep 3, 2025
b49bf2c
Use frozen instead of locked
tpoliaw Sep 3, 2025
33b284e
Use uv to run tox in CI
tpoliaw Sep 3, 2025
4798fa9
Use --frozen for all uv commands
tpoliaw Sep 3, 2025
d3cd892
Skip schema test if git repo is not available
tpoliaw Sep 3, 2025
3ee9de6
Re-use system_test workflow
tpoliaw Sep 3, 2025
ab1cdde
Remove --frozen from build
tpoliaw Sep 3, 2025
d52bb61
Import git exception directly
tpoliaw Sep 3, 2025
2ef4662
Use specific version of helm schema plugin
tpoliaw Sep 3, 2025
522b3ae
Remove pre-release=allow from lock file
tpoliaw Sep 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 14 additions & 3 deletions .devcontainer/devcontainer.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
},
"remoteEnv": {
// Allow X11 apps to run inside the container
"DISPLAY": "${localEnv:DISPLAY}"
"DISPLAY": "${localEnv:DISPLAY}",
},
"customizations": {
"vscode": {
Expand All @@ -33,12 +33,23 @@
]
}
},
"mounts": [
{
"source": "${localEnv:HOME}/.cache/uv",
"target": "/.cache/uv",
"type": "bind"
},
{
"target": "/workspaces/${localWorkspaceFolderBasename}/.venv",
"type": "volume"
}
],
"features": {
// add in eternal history and other bash features
"ghcr.io/diamondlightsource/devcontainer-features/bash-config:1": {}
},
// Create the config folder for the bash-config feature
"initializeCommand": "mkdir -p ${localEnv:HOME}/.config/bash-config",
"initializeCommand": "mkdir -p ${localEnv:HOME}/.config/bash-config ${localEnv:HOME}/.cache/uv",
"runArgs": [
// Allow the container to access the host X11 display and EPICS CA
"--net=host",
Expand All @@ -48,5 +59,5 @@
// Mount the parent as /workspaces so we can pip install peers as editable
"workspaceMount": "source=${localWorkspaceFolder}/..,target=/workspaces,type=bind",
// After the container is created, install the python project in editable form
"postCreateCommand": "pip install $([ -f dev-requirements.txt ] && echo '-c dev-requirements.txt') -e '.[dev]' && pre-commit install"
"postCreateCommand": "uv sync --frozen && uvx pre-commit install && SHELL=bash uv tool update-shell"
Copy link
Contributor

@coretl coretl Sep 10, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some questions:

  • Why --frozen? I would argue it is safer to do it without, as that will potentially update your lockfile if you added a dependency to pyproject.toml which you should at least know about before deciding whether it was intentional or not and committing the result
  • Why uvx rather than uv run? Pre-commit should be listed in the dev dependencies, and this will ensure we get a consistent version of it.
  • Do we use any uv tools? Why did you need to run this command to add the tool dir to the path?

Copy link
Contributor Author

@tpoliaw tpoliaw Sep 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I follow the frozen comment. --frozen doesn't update the lockfile and ensures the environment you have is the same as the one expected. Without it I couldn't get it to not need --prerelease=allow to deal with dodal et al depending on alpha versions of things and I didn't want that to be something we encourage.

I can't remember the motivation for uvx rather than uv run, might have been related to making pre-commit available everywhere without setting the $PATH to the venv as you did for ophyd-async. I will try it without and see what fails.

Also what does

I'm assuming there was meant to be more to this question

Copy link
Contributor

@coretl coretl Sep 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also what does

I'm assuming there was meant to be more to this question

Hah, didn't look before clicking comment, edited the original question

}
20 changes: 9 additions & 11 deletions .github/actions/install_requirements/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,9 @@
name: Install requirements
description: Install a version of python then call pip install and report what was installed
name: Setup UV
description: Setup UV and install the project
inputs:
python-version:
description: Python version to install, default is from Dockerfile
default: "dev"
pip-install:
description: Parameters to pass to pip install
default: "$([ -f dev-requirements.txt ] && echo '-c dev-requirements.txt') -e .[dev]"

runs:
using: composite
Expand All @@ -21,15 +18,16 @@ runs:
echo "PYTHON_VERSION=$PYTHON_VERSION" >> "$GITHUB_ENV"
shell: bash

- name: Setup python
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
- name: Install uv
uses: astral-sh/setup-uv@v5
with:
python-version: ${{ env.PYTHON_VERSION }}
enable-cache: true

- name: Install packages
run: pip install ${{ inputs.pip-install }}
- name: Install project
run: uv sync --frozen --all-extras --dev
shell: bash

- name: Report what was installed
run: pip freeze
- name: List dependency tree
run: uv export --frozen --format requirements.txt --no-hashes
shell: bash
8 changes: 6 additions & 2 deletions .github/workflows/_dist.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,13 @@ jobs:
# Need this to get version number from last tag
fetch-depth: 0

- name: Install
uses: ./.github/actions/install_requirements

- name: Build sdist and wheel
run: >
export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) &&
pipx run build
uv build

- name: Upload sdist and wheel as artifacts
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
Expand All @@ -24,11 +27,12 @@ jobs:
path: dist

- name: Check for packaging errors
run: pipx run twine check --strict dist/*
run: uvx twine check --strict dist/*

- name: Install produced wheel
uses: ./.github/actions/install_requirements
with:
# TODO: figure out what's going on
Copy link
Contributor

@coretl coretl Sep 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't checked dists yet, what problems do you see?

Copy link
Contributor Author

@tpoliaw tpoliaw Sep 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

None but I wasn't sure what I was looking for. The comment was more that I had no idea what the step was meant to be doing.

pip-install: dist/*.whl

- name: Test module --version works using the installed wheel
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/_system_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,4 +50,4 @@ jobs:
run: blueapi -c ${{ github.workspace }}/tests/system_tests/config.yaml serve &

- name: Run tests
run: tox -e system-test
run: uv run --frozen tox -e system-test
Copy link
Contributor

@coretl coretl Sep 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, maybe --frozen does make sense here, but maybe --locked would make even more sense? What do you reckon?

Copy link
Contributor Author

@tpoliaw tpoliaw Sep 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--locked fails if any packages are updatable so would fail pretty much every run, --frozen makes it use the versions in the lock file.

Copy link
Contributor

@coretl coretl Sep 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's not my reading of it:
https://docs.astral.sh/uv/concepts/projects/sync/#checking-the-lockfile

uv will not consider lockfiles outdated when new versions of packages are released — the lockfile needs to be explicitly updated if you want to upgrade dependencies.

Copy link
Contributor Author

@tpoliaw tpoliaw Sep 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might have been to do with the pre-release stuff again. It looks like dodal has been updated now so it doesn't depend on alpha releases but with the current lock file:

❯ git status
On branch uv_trial
Your branch is up to date with 'origin/uv_trial'.

nothing to commit, working tree clean

❯ uv sync --locked
Resolving despite existing lockfile due to change in pre-release mode: `allow` vs. `if-necessary-or-explicit`
Resolved 224 packages in 65ms
The lockfile at `uv.lock` needs to be updated, but `--locked` was provided. To update the lockfile, run `uv lock`.

❯ echo $?
1

❯ uv sync --frozen
Audited 221 packages in 1ms

❯ echo $?
0

65 changes: 0 additions & 65 deletions .github/workflows/_test.yml
View file
Open in desktop

This file was deleted.

10 changes: 6 additions & 4 deletions .github/workflows/_tox.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,11 @@ jobs:

- name: Install python packages
uses: ./.github/actions/install_requirements

- name: Install helm plugins
run: helm plugin install https://github.com/losisin/helm-values-schema-json.git

# The version here matches the one used by pre-commit - ideally this wouldn't need to be manually
# kept in sync.
run: helm plugin install https://github.com/losisin/helm-values-schema-json.git --version 2.2.1

- name: Run tox
run: tox -e ${{ inputs.tox }}
run: uv run --frozen tox -e ${{ inputs.tox }}
38 changes: 24 additions & 14 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,27 +16,37 @@ jobs:
tox: pre-commit,type-checking

test:
runs-on: ubuntu-latest
strategy:
matrix:
runs-on: ["ubuntu-latest"] # can add windows-latest, macos-latest
python-version: ["3.11", "3.12"]
include:
# Include one that runs in the dev environment
- runs-on: "ubuntu-latest"
python-version: "dev"
fail-fast: false
uses: ./.github/workflows/_test.yml
with:
runs-on: ${{ matrix.runs-on }}
python-version: ${{ matrix.python-version }}
secrets:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
steps:
- name: Checkout
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
with:
# Need this to get version number from last tag
fetch-depth: 0
- name: Setup project
uses: ./.github/actions/install_requirements
with:
python-version: ${{ matrix.python-version }}

- name: Unit tests
run: uv run --frozen tox -e tests

- name: Upload coverage to Codecov
uses: codecov/codecov-action@18283e04ce6e62d37312384ff67231eb8fd56d24 # v5
with:
name: ${{ matrix.python-version }}/ubuntu-latest
files: cov.xml
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

system-test:
uses: ./.github/workflows/_system_test.yml

container:
needs: test
needs: [test, system-test]
if: always()
uses: ./.github/workflows/_container.yml
with:
Expand Down Expand Up @@ -69,7 +79,7 @@ jobs:
id-token: write

release:
needs: [dist, test, docs]
needs: [dist, test, docs, container]
if: github.ref_type == 'tag'
uses: ./.github/workflows/_release.yml
permissions:
Expand Down
25 changes: 14 additions & 11 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,31 +3,33 @@
# Version SHA has been removed, see: https://github.com/DiamondLightSource/blueapi/issues/1053
ARG PYTHON_VERSION=3.11
FROM python:${PYTHON_VERSION} AS developer
Copy link
Contributor

@coretl coretl Sep 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I switched to ubuntu and use a uv managed python, this seems to work now

Copy link
Contributor Author

@tpoliaw tpoliaw Sep 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds sensible, I will look into that. Does it re-use the downloaded python or does it need to download it again every time it's built?

Copy link
Contributor

@coretl coretl Sep 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Difficult to tell, the download takes a fraction of a second on my machine, so I miss the output. I think it is downloaded each time, but I can't be sure

COPY --from=ghcr.io/astral-sh/uv:0.7.17 /uv /uvx /bin/

# Add any system dependencies for the developer/build environment here
RUN apt-get update && apt-get install -y --no-install-recommends \
graphviz \
&& rm -rf /var/lib/apt/lists/*

# Install helm for the dev container. This is the recommended
# Install helm for the dev container. This is the recommended
# approach per the docs: https://helm.sh/docs/intro/install
RUN curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3; \
chmod 700 get_helm.sh; \
./get_helm.sh; \
rm get_helm.sh
RUN helm plugin install https://github.com/losisin/helm-values-schema-json.git

# Set up a virtual environment and put it in PATH
RUN python -m venv /venv
ENV PATH=/venv/bin:$PATH
RUN mkdir -p /.cache/uv; chmod 777 /.cache/uv
ENV UV_CACHE_DIR=/.cache/uv
RUN SHELL=/usr/bin/bash uv tool update-shell

# The build stage installs the context into the venv
FROM developer AS build
RUN mkdir -p /.cache/pip; chmod o+wrX /.cache/pip

# Requires buildkit 0.17.0
COPY --chmod=o+wrX . /workspaces/blueapi

WORKDIR /workspaces/blueapi
RUN touch dev-requirements.txt && pip install --upgrade pip && pip install -c dev-requirements.txt .
RUN uv sync --frozen


FROM build AS debug
Expand All @@ -43,8 +45,8 @@ RUN DEBIAN_FRONTEND=noninteractive apt install libnss-ldapd -y
RUN sed -i 's/files/ldap files/g' /etc/nsswitch.conf

# Make editable and debuggable
RUN pip install debugpy
RUN pip install -e .
RUN uv tool install debugpy
Copy link
Contributor

@coretl coretl Sep 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, this is why you need uv tools to be on your path, but I don't see why you need it in the devcontainer.json too?

Copy link
Contributor Author

@tpoliaw tpoliaw Sep 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You might not, I don't use devcontainers so might have been holding it wrong when I tested it. I'll try again without.

RUN uv tool install --editable .

# Alternate entrypoint to allow devcontainer to attach
ENTRYPOINT [ "/bin/bash", "-c", "--" ]
Expand All @@ -58,9 +60,10 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
# Git required for installing packages at runtime
git \
&& rm -rf /var/lib/apt/lists/*
COPY --from=build --chmod=o+wrX /venv/ /venv/
COPY --from=build --chmod=o+wrX /.cache/pip /.cache/pip
ENV PATH=/venv/bin:$PATH
COPY --from=build --chmod=777 /.cache/uv /.cache/uv
COPY --from=build --chmod=777 /workspaces/blueapi /workspaces/blueapi
ENV PATH=/workspaces/blueapi/.venv/bin:$PATH
ENV UV_CACHE_DIR=/.cache/uv
ENV PYTHONPYCACHEPREFIX=/tmp/blueapi_pycache

# For this pod to understand finding user information from LDAP
Expand Down
Loading