Convert to using uv #1089

tpoliaw · 2025-06-27T15:36:30Z

Use dependency-group instead of optional-dependencies
Specify python version upper bound
Initial workflow rewrite

abbiemery

Please can you change/add to the docs to reflect the new uv commands?

Update dependencies manually while awaiting #1089 to pick up updates to ScanSpec that allow us to use Pydantic 2.11 and changes to our own observability-utils and bluesky-stomp.

Until all dependencies support python 3.13

It is no longer required now we have uv.lock

Significantly cuts devcontainer start-up time but requires cache directory to already exist on local machine.

Using the uv base image means the python version is not compatible with the epicscorelibs library.

codecov · 2025-09-03T14:20:16Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.54%. Comparing base (6136b90) to head (2ef4662).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1089   +/-   ##
=======================================
  Coverage   94.54%   94.54%           
=======================================
  Files          41       41           
  Lines        2566     2566           
=======================================
  Hits         2426     2426           
  Misses        140      140

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

pyright complains otherwise

coretl · 2025-09-10T15:04:59Z

.devcontainer/devcontainer.json

    "workspaceMount": "source=${localWorkspaceFolder}/..,target=/workspaces,type=bind",
    // After the container is created, install the python project in editable form
-    "postCreateCommand": "pip install $([ -f dev-requirements.txt ] && echo '-c dev-requirements.txt') -e '.[dev]' && pre-commit install"
+    "postCreateCommand": "uv sync --frozen && uvx pre-commit install && SHELL=bash uv tool update-shell"


Some questions:

Why --frozen? I would argue it is safer to do it without, as that will potentially update your lockfile if you added a dependency to pyproject.toml which you should at least know about before deciding whether it was intentional or not and committing the result

Why uvx rather than uv run? Pre-commit should be listed in the dev dependencies, and this will ensure we get a consistent version of it.

Do we use any uv tools? Why did you need to run this command to add the tool dir to the path?

Not sure I follow the frozen comment. --frozen doesn't update the lockfile and ensures the environment you have is the same as the one expected. Without it I couldn't get it to not need --prerelease=allow to deal with dodal et al depending on alpha versions of things and I didn't want that to be something we encourage.

I can't remember the motivation for uvx rather than uv run, might have been related to making pre-commit available everywhere without setting the $PATH to the venv as you did for ophyd-async. I will try it without and see what fails.

Also what does

I'm assuming there was meant to be more to this question

Also what does

I'm assuming there was meant to be more to this question

Hah, didn't look before clicking comment, edited the original question

coretl · 2025-09-10T15:05:43Z

.github/actions/install_requirements/action.yml

I've deleted this action in my version as I use a .python-version to determine the default python, then uv run everywhere

That'd be useful, I'll see if that's a viable option here

coretl · 2025-09-10T15:06:43Z

.github/workflows/_dist.yml

      - name: Install produced wheel
        uses: ./.github/actions/install_requirements
        with:
+          # TODO: figure out what's going on


I haven't checked dists yet, what problems do you see?

None but I wasn't sure what I was looking for. The comment was more that I had no idea what the step was meant to be doing.

coretl · 2025-09-10T15:07:42Z

.github/workflows/_system_test.yml


      - name: Run tests
-        run: tox -e system-test
+        run: uv run --frozen tox -e system-test


Hmm, maybe --frozen does make sense here, but maybe --locked would make even more sense? What do you reckon?

--locked fails if any packages are updatable so would fail pretty much every run, --frozen makes it use the versions in the lock file.

That's not my reading of it:
https://docs.astral.sh/uv/concepts/projects/sync/#checking-the-lockfile

uv will not consider lockfiles outdated when new versions of packages are released — the lockfile needs to be explicitly updated if you want to upgrade dependencies.

Might have been to do with the pre-release stuff again. It looks like dodal has been updated now so it doesn't depend on alpha releases but with the current lock file:

❯ git status On branch uv_trial Your branch is up to date with 'origin/uv_trial'. nothing to commit, working tree clean ❯ uv sync --locked Resolving despite existing lockfile due to change in pre-release mode: `allow` vs. `if-necessary-or-explicit` Resolved 224 packages in 65ms The lockfile at `uv.lock` needs to be updated, but `--locked` was provided. To update the lockfile, run `uv lock`. ❯ echo $? 1 ❯ uv sync --frozen Audited 221 packages in 1ms ❯ echo $? 0

coretl · 2025-09-10T15:08:54Z

Dockerfile

 # or docker with user namespaces.
 # Version SHA has been removed, see: https://github.com/DiamondLightSource/blueapi/issues/1053
 ARG PYTHON_VERSION=3.11
 FROM python:${PYTHON_VERSION} AS developer


I switched to ubuntu and use a uv managed python, this seems to work now

Sounds sensible, I will look into that. Does it re-use the downloaded python or does it need to download it again every time it's built?

Difficult to tell, the download takes a fraction of a second on my machine, so I miss the output. I think it is downloaded each time, but I can't be sure

coretl · 2025-09-10T15:10:35Z

Dockerfile

 # Make editable and debuggable
-RUN pip install debugpy
-RUN pip install -e .
+RUN uv tool install debugpy


Ah, this is why you need uv tools to be on your path, but I don't see why you need it in the devcontainer.json too?

You might not, I don't use devcontainers so might have been holding it wrong when I tested it. I'll try again without.

coretl · 2025-09-10T15:12:43Z

pyproject.toml

 license.file = "LICENSE"
 readme = "README.md"
-requires-python = ">=3.11"
+requires-python = ">=3.11,<3.13"


The current best practice appears to be never to cap python version for libraries:
https://iscinumpy.dev/post/bound-version-constraints/#tldr

I would argue it probably makes sense here too...

This was probably a hang over from epics(?) not supporting 3.13 and can be removed now, especially if we're using .python-version. I'm pretty sure uv ignores upper bounds anyway.

dodal, ophyd-async etc no longer need it.

abbiemery requested changes Jul 1, 2025

View reviewed changes

tpoliaw force-pushed the uv_trial branch 2 times, most recently from 1fd9b2f to 79dc252 Compare July 4, 2025 11:00

tpoliaw requested a review from coretl July 4, 2025 11:04

DiamondJoseph mentioned this pull request Jul 14, 2025

chore(deps): Update dependencies #1122

Merged

tpoliaw force-pushed the uv_trial branch from c7caf8a to 5ef5171 Compare July 22, 2025 15:07

tpoliaw force-pushed the uv_trial branch from 049ac3d to 0231fea Compare August 21, 2025 14:18

tpoliaw and others added 18 commits September 3, 2025 11:23

Use dependency-group instead of optional-dependencies

e320979

Specify python version upper bound

0dfb460

Until all dependencies support python 3.13

Initial workflow rewrite

091c6c4

WIP Convert dockerfile to use uv

9a4050a

Remove dev-requirements.txt

a3c6058

It is no longer required now we have uv.lock

Remove pip install from devcontainer config

8ba2727

Restore postCreateCommand to set up uv environment

c229d70

Add mount for uv cache

335dd05

Significantly cuts devcontainer start-up time but requires cache directory to already exist on local machine.

Mount venv in workspace

9e37240

Fix CI

e081661

Add shell back to install_requirements

f47427a

Update ophyd -> ophyd-async

cf38536

Move checkout out of install action

d69f43f

Revert change to using uv base image

acb1741

Using the uv base image means the python version is not compatible with the epicscorelibs library.

Remove hardcoded project name from devcontainer

6f7af1c

Restrict httpx to <1.0 to avoid breaking changes in pre-release version

c989b4d

Install requirements in _dist workflow

1caa29e

Update uv.lock

8579853

tpoliaw force-pushed the uv_trial branch from 0231fea to 8579853 Compare September 3, 2025 10:35

tpoliaw added 3 commits September 3, 2025 11:43

Attempt to fix dist workflow

261e064

Update uv.lock. Again

8784091

Use frozen instead of locked

b49bf2c

tpoliaw force-pushed the uv_trial branch 2 times, most recently from 9ef69b0 to 91bf416 Compare September 3, 2025 13:35

Use uv to run tox in CI

33b284e

tpoliaw force-pushed the uv_trial branch from 91bf416 to 33b284e Compare September 3, 2025 13:36

tpoliaw added 2 commits September 3, 2025 14:40

Use --frozen for all uv commands

4798fa9

Skip schema test if git repo is not available

d3cd892

tpoliaw force-pushed the uv_trial branch 4 times, most recently from 0724855 to 102809e Compare September 3, 2025 14:52

Re-use system_test workflow

3ee9de6

tpoliaw force-pushed the uv_trial branch from 102809e to 3ee9de6 Compare September 3, 2025 14:54

tpoliaw added 3 commits September 3, 2025 15:57

Remove --frozen from build

ab1cdde

Import git exception directly

d52bb61

pyright complains otherwise

Use specific version of helm schema plugin

2ef4662

tpoliaw linked an issue Sep 3, 2025 that may be closed by this pull request

Trial-adopt migration to uv from copier template #957

Open

3 tasks

tpoliaw marked this pull request as ready for review September 10, 2025 13:50

tpoliaw requested a review from a team as a code owner September 10, 2025 13:50

coretl reviewed Sep 10, 2025

View reviewed changes

Remove pre-release=allow from lock file

522b3ae

dodal, ophyd-async etc no longer need it.

Uh oh!

Convert to using uv #1089

Are you sure you want to change the base?

Convert to using uv #1089

Uh oh!

Conversation

tpoliaw commented Jun 27, 2025

Uh oh!

abbiemery left a comment

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

coretl Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Sep 3, 2025 •

edited

Loading

coretl Sep 10, 2025 •

edited

Loading