"Unsatisfiable Requirement" edge case error when `juliacall` is upgraded but Julia registries are not

[brian-dellabetta]

Hello, I have a potential issue originally posted as a Discussion a few weeks back. I haven't gotten any responses, so I'm moving it to an issue in hopes of getting more traction. If my solution of adding another env var that, if set, calls Pkg.registry.update() inside the initialization script seems reasonable, I can try my hand at a PR.

Discussed in #422

^{Originally posted by brian-dellabetta November 9, 2023}
Hi there,
we have come across an edge case when upgrading PythonCall/juliacall, and I am curious if anyone knows of a workaround or clean solution.

Initial Condition:

• User has juliacall==0.9.14 installed in their Python env
• User has PythonCall.jl@0.9.14 installed in the Julia project that juliacall links to.
• User has not updated their julia registry to know that a new version of PythonCall.jl@0.9.15 available.

Steps to reproduce:

• User does pip install juliacall==0.9.15
• User runs from juliacall import Main as jl
  This will result in the following error:

ERROR: Unsatisfiable requirements detected for package PythonCall [6099a3de]:
 PythonCall [6099a3de] log:
 ├─possible versions are: 0.1.0-0.9.14 or uninstalled
 ├─restricted to versions * by <redacted>, leaving only versions: 0.1.0-0.9.14
 │ └─<redacted> log:
 │   ├─possible versions are: 0.1.0 or uninstalled
 │   └─<redacted> is fixed to version 0.1.0
 └─restricted to versions 0.9.15 by an explicit requirement — no versions left
Stacktrace:
  [1] check_constraints(graph::Pkg.Resolve.Graph)
    @ Pkg.Resolve ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/Resolve/graphtype.jl:998
  [2] Pkg.Resolve.Graph(compat::Dict{Base.UUID, Dict{VersionNumber, Dict{Base.UUID, Pkg.Versions.VersionSpec}}}, compat_weak::Dict{Base.UUID, Dict{VersionNumber, Set{Base.UUID}}}, uuid_to_name::Dict{Base.UUID, String}, reqs::Dict{Base.UUID, Pkg.Versions.VersionSpec}, fixed::Dict{Base.UUID, Pkg.Resolve.Fixed}, verbose::Bool, julia_version::VersionNumber)
    @ Pkg.Resolve ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/Resolve/graphtype.jl:345
  [3] deps_graph(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, uuid_to_name::Dict{Base.UUID, String}, reqs::Dict{Base.UUID, Pkg.Versions.VersionSpec}, fixed::Dict{Base.UUID, Pkg.Resolve.Fixed}, julia_version::VersionNumber, installed_only::Bool)
    @ Pkg.Operations ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:587
  [4] resolve_versions!(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, julia_version::VersionNumber, installed_only::Bool)
    @ Pkg.Operations ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:407
  [5] targeted_resolve(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, preserve::Pkg.Types.PreserveLevel, julia_version::VersionNumber)
    @ Pkg.Operations ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1357
  [6] tiered_resolve(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, julia_version::VersionNumber, try_all_installed::Bool)
    @ Pkg.Operations ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1346
  [7] _resolve(io::Base.TTY, env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, preserve::Pkg.Types.PreserveLevel, julia_version::VersionNumber)
    @ Pkg.Operations ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1367
  [8] develop(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, new_git::Set{Base.UUID}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1407
  [9] develop
    @ ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1399 [inlined]
 [10] develop(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; shared::Bool, preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform, kwargs::Base.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.TTY}}})
    @ Pkg.API ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/API.jl:222
 [11] develop(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/API.jl:156
 [12] develop(pkgs::Vector{Pkg.Types.PackageSpec})
    @ Pkg.API ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/API.jl:145
 [13] top-level scope
    @ none:1

All I need to do is run jl.seval("import Pkg; Pkg.registry.update()") but I can't get access to jl, I hit the error above first. Any suggestions on how to programmatically resolve this, ideally in Python? Hoping there's a way without having to edit juliacall source code to allow for an init/startup script. If that is the case, I can create an issue and take a shot at a PR. It may be as simple as adding another env var option that injects Pkg.registry.update() in the init script somewhere around this line.

Thanks!

[cjdoris]

I'd be OK with adding an unconditional Pkg.registry.update() in the right place. I thought Julia always updated the registries the first time you Pkg.add anything though??

[brian-dellabetta]

Thanks @cjdoris , I checked in julia and ] add PythonCall does in fact trigger a registry update even if PythonCall already exists in the project.

This may really be an issue for juliapkg, I noticed that PythonCall's version gets pinned in the juliapkg.json, but it is very strange that this is occurring:

[juliapkg] Locating Julia ^1.6.1
[juliapkg] Using Julia 1.9.3 at /home/<redacted>/.julia/environments/pyjuliapkg/pyjuliapkg/install/bin/julia
[juliapkg] Using Julia project at /home/<redacted>/.julia/environments/pyjuliapkg
[juliapkg] Installing packages:
           julia> import Pkg
           julia> Pkg.develop([<redacted>])
           julia> Pkg.add([Pkg.PackageSpec(name="PythonCall", uuid="6099a3de-0909-46bc-b1f4-468b9a2dfc0d")])
           julia> Pkg.resolve()
           julia> Pkg.precompile()
   Resolving package versions...
ERROR: Unsatisfiable requirements detected for package PythonCall [6099a3de]:
 <full error posted above>

I'm not entirely sure how juliapkg uses the .json files internally, maybe there is some subtlety. My concern with the explicit call to Pkg.registry.update() is that it should only occur in online mode, i.e. PYTHON_JULIAPKG_OFFLINE=no, but that would be using a juliapkg env var inside the juliacall source code.

Any thoughts? Next time a version of PythonCall/juliacall is released, I can check. Unfortunately this edge case is rather hard to reproduce.

[cjdoris]

Actually this would be a change to juliapkg so no worries there.

It doesn't sound like that would fix things anyway, if the registry does update. You can roll back to an old version of the registry by cloning the registry repo and checking out an old commit. Can you do that and see if you can reproduce the issue?

[brian-dellabetta]

Sure, I will try this sometime next week

[brian-dellabetta]

Nm i had a chance to look just now. Steps to reproduce:

1. Revert to old General registry commit

cd /tmp
git clone git@github.com:JuliaRegistries/General.git
cd General
git reset 46f90400dd 
git checkout -- . && git clean -fd

2. Create conda environment, set up julia_env, and install PythonCall into julia_env

conda create -n pyjlcheck
conda activate pyjlcheck
pip install juliacall==0.9.14
python -c 'from juliacall import Main as jl'

3. Set General registry to old cloned repo
   julia --project=~/anaconda3/envs/pyjlcheck/julia_env

] registry rm General
] registry add /tmp/General

4. Pull latest changes from general registry and upgrade juliacall version

cd /tmp/General
git pull
pip install juliacall==0.9.15

5. Run juliacall

from juliacall import Main as jl

errors out with full logs:

>>> from juliacall import Main as jl
[juliapkg] Locating Julia ^1.6.1
[juliapkg] Using Julia 1.9.4 at /Users/brian/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/bin/julia
[juliapkg] Using Julia project at /Users/brian/anaconda3/envs/pyjlcheck/julia_env
[juliapkg] Installing packages:
           julia> import Pkg
           julia> Pkg.add([Pkg.PackageSpec(name="PythonCall", uuid="6099a3de-0909-46bc-b1f4-468b9a2dfc0d")])
           julia> Pkg.resolve()
   Resolving package versions...
ERROR: Unsatisfiable requirements detected for package PythonCall [6099a3de]:
 PythonCall [6099a3de] log:
 ├─possible versions are: 0.1.0-0.9.14 or uninstalled
 └─restricted to versions 0.9.15 by an explicit requirement — no versions left
Stacktrace:
  [1] check_constraints(graph::Pkg.Resolve.Graph)
    @ Pkg.Resolve ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/Resolve/graphtype.jl:998
  [2] Pkg.Resolve.Graph(compat::Dict{Base.UUID, Dict{VersionNumber, Dict{Base.UUID, Pkg.Versions.VersionSpec}}}, compat_weak::Dict{Base.UUID, Dict{VersionNumber, Set{Base.UUID}}}, uuid_to_name::Dict{Base.UUID, String}, reqs::Dict{Base.UUID, Pkg.Versions.VersionSpec}, fixed::Dict{Base.UUID, Pkg.Resolve.Fixed}, verbose::Bool, julia_version::VersionNumber)
    @ Pkg.Resolve ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/Resolve/graphtype.jl:345
  [3] deps_graph(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, uuid_to_name::Dict{Base.UUID, String}, reqs::Dict{Base.UUID, Pkg.Versions.VersionSpec}, fixed::Dict{Base.UUID, Pkg.Resolve.Fixed}, julia_version::VersionNumber, installed_only::Bool)
    @ Pkg.Operations ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:587
  [4] resolve_versions!(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, julia_version::VersionNumber, installed_only::Bool)
    @ Pkg.Operations ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:407
  [5] targeted_resolve(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, preserve::Pkg.Types.PreserveLevel, julia_version::VersionNumber)
    @ Pkg.Operations ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1362
  [6] tiered_resolve(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, julia_version::VersionNumber, try_all_installed::Bool)
    @ Pkg.Operations ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1351
  [7] _resolve(io::Base.TTY, env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, preserve::Pkg.Types.PreserveLevel, julia_version::VersionNumber)
    @ Pkg.Operations ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1372
  [8] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, new_git::Set{Base.UUID}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1389
  [9] add
    @ ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1378 [inlined]
 [10] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform, kwargs::Base.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.TTY}}})
    @ Pkg.API ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/API.jl:275
 [11] add(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/API.jl:156
 [12] add(pkgs::Vector{Pkg.Types.PackageSpec})
    @ Pkg.API ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/API.jl:145
 [13] top-level scope
    @ none:1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/brian/anaconda3/lib/python3.8/site-packages/juliacall/__init__.py", line 228, in <module>
    init()
  File "/Users/brian/anaconda3/lib/python3.8/site-packages/juliacall/__init__.py", line 153, in init
    CONFIG['exepath'] = exepath = juliapkg.executable()
  File "/Users/brian/anaconda3/lib/python3.8/site-packages/juliapkg/deps.py", line 296, in executable
    resolve()
  File "/Users/brian/anaconda3/lib/python3.8/site-packages/juliapkg/deps.py", line 275, in resolve
    run([exe, '--project='+project, '--startup-file=no', '-e', '; '.join(script)], check=True)
  File "/Users/brian/anaconda3/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/Users/brian/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/bin/julia', '--project=/Users/brian/anaconda3/envs/pyjlcheck/julia_env', '--startup-file=no', '-e', 'import Pkg; Pkg.add([Pkg.PackageSpec(name="PythonCall", uuid="6099a3de-0909-46bc-b1f4-468b9a2dfc0d")]); Pkg.resolve()']' returned non-zero exit status 1.

I can confirm that taking that command

/Users/brian/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/bin/julia --project=/Users/brian/anaconda3/envs/pyjlcheck/julia_env --startup-file=no -e 'import Pkg; Pkg.add([Pkg.PackageSpec(name="PythonCall", uuid="6099a3de-0909-46bc-b1f4-468b9a2dfc0d")]); Pkg.resolve()'

fails but adding Pkg.Registry.update() works (though we'd probably only want to do this in online mode)

/Users/brian/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/bin/julia --project=/Users/brian/anaconda3/envs/pyjlcheck/julia_env --startup-file=no -e 'import Pkg; Pkg.Registry.update(); Pkg.add([Pkg.PackageSpec(name="PythonCall", uuid="6099a3de-0909-46bc-b1f4-468b9a2dfc0d")]); Pkg.resolve()'

[brian-dellabetta]

@cjdoris should we just change it to auto-update registry? I can create a PR, I think it just amounts to changing this line to

script = ['import Pkg', 'Pkg.Registry.update()']

It's already in the if not STATE['offline']: check so we don't need to worry about offline mode

[brian-dellabetta]

PR submitted to resolve this in pyjuliapkg -- JuliaPy/pyjuliapkg#22