Version-less pip recipe install does not upgrade an already-installed package (Python RPC engine)

[timtebeek]

What is the problem?

Installing a pip-based recipe package without a version specifier through the Moderne CLI / OpenRewrite Python RPC engine does not upgrade an already-installed package. The engine reuses whatever version happens to be present in the recipe-install directory instead of resolving the latest from the index.

This is a regression. Engine versions ≤ 8.81.0 performed no in-engine pip install at all; the short-circuit guard responsible for this was introduced around 8.84.9.

Where

rewrite-python/rewrite/src/rewrite/rpc/server.py

handle_install_recipes() pip-installs a recipe package into --recipe-install-dir before activating it, guarded by _is_package_installed:

if _recipe_install_dir is not None and not _is_package_installed(package_name, version):
    _pip_install_recipe_package(package_name, version, _recipe_install_dir)

def _is_package_installed(package_name, version):
    try:
        installed = importlib.metadata.version(package_name)
    except Exception:
        return False
    return version is None or installed == version

Root cause

There are actually two coupled defects on the version-less path:

1. The install is skipped entirely.
When version is None (a version-less request, which per the "Adds or updates a pip package" command header and by analogy with jar install g:a defaulting to LATEST should install the newest compatible version), _is_package_installed returns True as soon as any version is installed. So _pip_install_recipe_package is never called, and the engine activates the stale, already-present version. The --upgrade flag inside _pip_install_recipe_package is dead code on this path.

The same flaw applies to non-exact PEP 440 specifiers (>=1.0, ~=1.4): installed == version is a string compare that can never match a comparator spec, so those currently do fall through to pip — but the intent (never treat an arbitrary installed version as "satisfied") should be made explicit so a future refactor doesn't reintroduce the short-circuit.

2. Even once pip runs, the reported version can stay stale.
_pip_install_recipe_package uses pip install --target <dir> --upgrade. pip install --target cannot uninstall — it overwrites the package's module files but leaves the previous version's {name}-{version}.dist-info directory sitting next to the newly installed one (dist-info dir names embed the version, so they don't collide). importlib.metadata.version() then enumerates the install dir and returns whichever dist-info it finds first — frequently the stale lower version. The result: even after the latest code is installed, the version reported back in the RPC response (and used for activation/attribution) can be the old one.

Reproduced with a local wheel index offering 0.7.0 and 0.9.2:

install 0.7.0            -> target/orw_dummy_recipe-0.7.0.dist-info
pip install --upgrade    -> target/orw_dummy_recipe-0.9.2.dist-info  (0.7.0 dist-info NOT removed)
importlib.metadata.version("orw-dummy-recipe")  ->  "0.7.0"   # stale

Impact (as seen from the CLI)

For a user running a version-less recipe install through the Moderne CLI against the Python engine:

• The recipe package is never upgraded. Once any version is installed into the recipe-install dir, subsequent version-less installs are silently no-ops — the CLI reports success while continuing to run an outdated recipe package.
• Fixes and new recipes never arrive. Publishing a new recipe version to the index has no effect for anyone who already has an older version cached; they keep running the stale one until the install dir is manually cleared or an exact newer version is pinned.
• Silent / misleading. There is no error — the install "succeeds" and reports the old version, so the staleness is invisible unless the user notices behavior that a newer release was supposed to change.
• Even the two-part scenario (clear dir won't help long-term): because of defect #2, once an upgrade does eventually run into a dir that still holds an older dist-info, the version echoed back to the CLI can still be the stale one.

This diverges from the documented/expected semantics that a version-less install resolves and installs the latest compatible version, mirroring jar install g:a → LATEST.

Steps to reproduce

1. Configure the Python RPC engine with --recipe-install-dir.
2. Install a recipe package pinned to an older version (e.g. 0.7.0).
3. Issue a version-less install of the same package ({'packageName': ..., 'version': None}) with a newer version (0.9.2) available on the index.
4. Observe the engine returns/activates 0.7.0 rather than 0.9.2.

Affected versions

Engine ≥ ~8.84.9 (guard introduced). Not present in ≤ 8.81.0 (which did no in-engine pip install).

[timtebeek]

Proposed solution

Two small, targeted changes in rewrite-python/rewrite/src/rewrite/rpc/server.py, addressing both defects above.

1. _is_package_installed — never short-circuit a version-less or non-exact request

Only skip pip when the request pins an exact version that is already installed. A version-less request (None) or a non-exact PEP 440 specifier (>=, ~=, !=, …) must always run pip so the index decides the resolved version.

def _is_package_installed(package_name: str, version: Optional[str]) -> bool:
    # Skipping pip is only safe when the request pins an exact version that is
    # already installed. A version-less request ("install the latest") must
    # always run pip so the index resolves the newest compatible version —
    # mirroring `jar install g:a` defaulting to LATEST. A non-exact PEP 440
    # specifier (">=1.0", "~=1.4") must likewise never be treated as satisfied
    # by whatever arbitrary version happens to be installed.
    if version is None:
        return False
    if version.startswith("=="):
        exact = version[2:]
    elif version[0] in "<>!~=":
        return False  # some other comparator: not an exact pin
    else:
        exact = version  # bare version == exact match (mirrors the "==" spec built for it)
    try:
        import importlib.metadata
        installed = importlib.metadata.version(package_name)
    except Exception:
        return False
    return installed == exact

2. Clear stale dist-info before installing so the reported version is correct

pip install --target can't uninstall, so add a helper that removes any existing *.dist-info for the package before the install; pip then re-creates exactly one, and importlib.metadata.version() resolves to the version it just installed. The package's module files are left as-is — pip overwrites them during the reinstall.

def _remove_stale_dist_info(package_name: str, target_dir: Path) -> None:
    import re, shutil
    if not target_dir.exists():
        return
    normalized = re.sub(r"[-_.]+", "_", package_name).lower()
    for dist_info in target_dir.glob("*.dist-info"):
        name_part = dist_info.name[: -len(".dist-info")].rsplit("-", 1)[0]
        if re.sub(r"[-_.]+", "_", name_part).lower() == normalized:
            shutil.rmtree(dist_info, ignore_errors=True)

Called at the top of _pip_install_recipe_package (before the pip install --upgrade --target … invocation). --upgrade stays — it's still needed so pip replaces the populated target dir, and now it genuinely fetches the latest on the version-less path instead of being dead code.

Tests

Regression coverage added in tests/rpc/test_server.py, following the module's existing conventions:

• _is_package_installed unit coverage: None, exact pin match/mismatch, ==, comparator specifiers, and the package-absent path.
• An end-to-end handle_install_recipes test using a self-contained local wheel index (hand-rolled 0.7.0 + 0.9.2 wheels + PIP_NO_INDEX/PIP_FIND_LINKS env vars — no network). It pre-installs 0.7.0, calls handle_install_recipes({'recipes': {'packageName': ..., 'version': None}}), and asserts the active/returned version is 0.9.2.

Both new tests fail on current main (the version-less test logs InstallRecipes: 0 new and reports 0.7.0) and pass with the fix — confirming they're genuine regressions. Full module test run: tests/rpc/test_server.py 16 passed, tests/test_marketplace.py 18 passed.

I have this implemented and tested locally and can open a PR.

[timtebeek]

Pending a fix we've pinned the versions in the docs; optionally this can be reverted after a fix here

• Pin Python recipe pip install commands to an explicit version moderneinc/moderne-docs#826
• Pin Python recipe pip install commands to an explicit version rewrite-recipe-markdown-generator#351