"Unsatisfiable Requirement" edge case error when `juliacall` is upgraded but Julia registries are not #428

brian-dellabetta · 2023-11-27T16:47:53Z

Hello, I have a potential issue originally posted as a Discussion a few weeks back. I haven't gotten any responses, so I'm moving it to an issue in hopes of getting more traction. If my solution of adding another env var that, if set, calls Pkg.registry.update() inside the initialization script seems reasonable, I can try my hand at a PR.

Discussed in #422

^{Originally posted by brian-dellabetta November 9, 2023}
Hi there,
we have come across an edge case when upgrading PythonCall/juliacall, and I am curious if anyone knows of a workaround or clean solution.

Initial Condition:

User has juliacall==0.9.14 installed in their Python env
User has [email protected] installed in the Julia project that juliacall links to.
User has not updated their julia registry to know that a new version of [email protected] available.

Steps to reproduce:

User does pip install juliacall==0.9.15
User runs from juliacall import Main as jl
This will result in the following error:

ERROR: Unsatisfiable requirements detected for package PythonCall [6099a3de]:
 PythonCall [6099a3de] log:
 ├─possible versions are: 0.1.0-0.9.14 or uninstalled
 ├─restricted to versions * by <redacted>, leaving only versions: 0.1.0-0.9.14
 │ └─<redacted> log:
 │   ├─possible versions are: 0.1.0 or uninstalled
 │   └─<redacted> is fixed to version 0.1.0
 └─restricted to versions 0.9.15 by an explicit requirement — no versions left
Stacktrace:
  [1] check_constraints(graph::Pkg.Resolve.Graph)
    @ Pkg.Resolve ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/Resolve/graphtype.jl:998
  [2] Pkg.Resolve.Graph(compat::Dict{Base.UUID, Dict{VersionNumber, Dict{Base.UUID, Pkg.Versions.VersionSpec}}}, compat_weak::Dict{Base.UUID, Dict{VersionNumber, Set{Base.UUID}}}, uuid_to_name::Dict{Base.UUID, String}, reqs::Dict{Base.UUID, Pkg.Versions.VersionSpec}, fixed::Dict{Base.UUID, Pkg.Resolve.Fixed}, verbose::Bool, julia_version::VersionNumber)
    @ Pkg.Resolve ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/Resolve/graphtype.jl:345
  [3] deps_graph(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, uuid_to_name::Dict{Base.UUID, String}, reqs::Dict{Base.UUID, Pkg.Versions.VersionSpec}, fixed::Dict{Base.UUID, Pkg.Resolve.Fixed}, julia_version::VersionNumber, installed_only::Bool)
    @ Pkg.Operations ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:587
  [4] resolve_versions!(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, julia_version::VersionNumber, installed_only::Bool)
    @ Pkg.Operations ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:407
  [5] targeted_resolve(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, preserve::Pkg.Types.PreserveLevel, julia_version::VersionNumber)
    @ Pkg.Operations ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1357
  [6] tiered_resolve(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, julia_version::VersionNumber, try_all_installed::Bool)
    @ Pkg.Operations ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1346
  [7] _resolve(io::Base.TTY, env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, preserve::Pkg.Types.PreserveLevel, julia_version::VersionNumber)
    @ Pkg.Operations ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1367
  [8] develop(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, new_git::Set{Base.UUID}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1407
  [9] develop
    @ ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1399 [inlined]
 [10] develop(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; shared::Bool, preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform, kwargs::Base.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.TTY}}})
    @ Pkg.API ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/API.jl:222
 [11] develop(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/API.jl:156
 [12] develop(pkgs::Vector{Pkg.Types.PackageSpec})
    @ Pkg.API ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/API.jl:145
 [13] top-level scope
    @ none:1

All I need to do is run jl.seval("import Pkg; Pkg.registry.update()") but I can't get access to jl, I hit the error above first. Any suggestions on how to programmatically resolve this, ideally in Python? Hoping there's a way without having to edit juliacall source code to allow for an init/startup script. If that is the case, I can create an issue and take a shot at a PR. It may be as simple as adding another env var option that injects Pkg.registry.update() in the init script somewhere around this line.

Thanks!

The text was updated successfully, but these errors were encountered:

cjdoris · 2023-11-29T09:05:51Z

I'd be OK with adding an unconditional Pkg.registry.update() in the right place. I thought Julia always updated the registries the first time you Pkg.add anything though??

brian-dellabetta · 2023-11-29T20:40:21Z

Thanks @cjdoris , I checked in julia and ] add PythonCall does in fact trigger a registry update even if PythonCall already exists in the project.

This may really be an issue for juliapkg, I noticed that PythonCall's version gets pinned in the juliapkg.json, but it is very strange that this is occurring:

[juliapkg] Locating Julia ^1.6.1
[juliapkg] Using Julia 1.9.3 at /home/<redacted>/.julia/environments/pyjuliapkg/pyjuliapkg/install/bin/julia
[juliapkg] Using Julia project at /home/<redacted>/.julia/environments/pyjuliapkg
[juliapkg] Installing packages:
           julia> import Pkg
           julia> Pkg.develop([<redacted>])
           julia> Pkg.add([Pkg.PackageSpec(name="PythonCall", uuid="6099a3de-0909-46bc-b1f4-468b9a2dfc0d")])
           julia> Pkg.resolve()
           julia> Pkg.precompile()
   Resolving package versions...
ERROR: Unsatisfiable requirements detected for package PythonCall [6099a3de]:
 <full error posted above>

I'm not entirely sure how juliapkg uses the .json files internally, maybe there is some subtlety. My concern with the explicit call to Pkg.registry.update() is that it should only occur in online mode, i.e. PYTHON_JULIAPKG_OFFLINE=no, but that would be using a juliapkg env var inside the juliacall source code.

Any thoughts? Next time a version of PythonCall/juliacall is released, I can check. Unfortunately this edge case is rather hard to reproduce.

cjdoris · 2023-11-30T08:34:57Z

Actually this would be a change to juliapkg so no worries there.

It doesn't sound like that would fix things anyway, if the registry does update. You can roll back to an old version of the registry by cloning the registry repo and checking out an old commit. Can you do that and see if you can reproduce the issue?

brian-dellabetta · 2023-12-01T17:49:52Z

Sure, I will try this sometime next week

brian-dellabetta · 2023-12-01T18:50:41Z

Nm i had a chance to look just now. Steps to reproduce:

Revert to old General registry commit

cd /tmp
git clone [email protected]:JuliaRegistries/General.git
cd General
git reset 46f90400dd 
git checkout -- . && git clean -fd

Create conda environment, set up julia_env, and install PythonCall into julia_env

conda create -n pyjlcheck
conda activate pyjlcheck
pip install juliacall==0.9.14
python -c 'from juliacall import Main as jl'

Set General registry to old cloned repo
julia --project=~/anaconda3/envs/pyjlcheck/julia_env

] registry rm General
] registry add /tmp/General

Pull latest changes from general registry and upgrade juliacall version

cd /tmp/General
git pull
pip install juliacall==0.9.15

Run juliacall

from juliacall import Main as jl

errors out with full logs:

>>> from juliacall import Main as jl
[juliapkg] Locating Julia ^1.6.1
[juliapkg] Using Julia 1.9.4 at /Users/brian/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/bin/julia
[juliapkg] Using Julia project at /Users/brian/anaconda3/envs/pyjlcheck/julia_env
[juliapkg] Installing packages:
           julia> import Pkg
           julia> Pkg.add([Pkg.PackageSpec(name="PythonCall", uuid="6099a3de-0909-46bc-b1f4-468b9a2dfc0d")])
           julia> Pkg.resolve()
   Resolving package versions...
ERROR: Unsatisfiable requirements detected for package PythonCall [6099a3de]:
 PythonCall [6099a3de] log:
 ├─possible versions are: 0.1.0-0.9.14 or uninstalled
 └─restricted to versions 0.9.15 by an explicit requirement — no versions left
Stacktrace:
  [1] check_constraints(graph::Pkg.Resolve.Graph)
    @ Pkg.Resolve ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/Resolve/graphtype.jl:998
  [2] Pkg.Resolve.Graph(compat::Dict{Base.UUID, Dict{VersionNumber, Dict{Base.UUID, Pkg.Versions.VersionSpec}}}, compat_weak::Dict{Base.UUID, Dict{VersionNumber, Set{Base.UUID}}}, uuid_to_name::Dict{Base.UUID, String}, reqs::Dict{Base.UUID, Pkg.Versions.VersionSpec}, fixed::Dict{Base.UUID, Pkg.Resolve.Fixed}, verbose::Bool, julia_version::VersionNumber)
    @ Pkg.Resolve ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/Resolve/graphtype.jl:345
  [3] deps_graph(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, uuid_to_name::Dict{Base.UUID, String}, reqs::Dict{Base.UUID, Pkg.Versions.VersionSpec}, fixed::Dict{Base.UUID, Pkg.Resolve.Fixed}, julia_version::VersionNumber, installed_only::Bool)
    @ Pkg.Operations ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:587
  [4] resolve_versions!(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, julia_version::VersionNumber, installed_only::Bool)
    @ Pkg.Operations ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:407
  [5] targeted_resolve(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, preserve::Pkg.Types.PreserveLevel, julia_version::VersionNumber)
    @ Pkg.Operations ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1362
  [6] tiered_resolve(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, julia_version::VersionNumber, try_all_installed::Bool)
    @ Pkg.Operations ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1351
  [7] _resolve(io::Base.TTY, env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, preserve::Pkg.Types.PreserveLevel, julia_version::VersionNumber)
    @ Pkg.Operations ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1372
  [8] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, new_git::Set{Base.UUID}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1389
  [9] add
    @ ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1378 [inlined]
 [10] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform, kwargs::Base.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.TTY}}})
    @ Pkg.API ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/API.jl:275
 [11] add(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/API.jl:156
 [12] add(pkgs::Vector{Pkg.Types.PackageSpec})
    @ Pkg.API ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/API.jl:145
 [13] top-level scope
    @ none:1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/brian/anaconda3/lib/python3.8/site-packages/juliacall/__init__.py", line 228, in <module>
    init()
  File "/Users/brian/anaconda3/lib/python3.8/site-packages/juliacall/__init__.py", line 153, in init
    CONFIG['exepath'] = exepath = juliapkg.executable()
  File "/Users/brian/anaconda3/lib/python3.8/site-packages/juliapkg/deps.py", line 296, in executable
    resolve()
  File "/Users/brian/anaconda3/lib/python3.8/site-packages/juliapkg/deps.py", line 275, in resolve
    run([exe, '--project='+project, '--startup-file=no', '-e', '; '.join(script)], check=True)
  File "/Users/brian/anaconda3/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/Users/brian/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/bin/julia', '--project=/Users/brian/anaconda3/envs/pyjlcheck/julia_env', '--startup-file=no', '-e', 'import Pkg; Pkg.add([Pkg.PackageSpec(name="PythonCall", uuid="6099a3de-0909-46bc-b1f4-468b9a2dfc0d")]); Pkg.resolve()']' returned non-zero exit status 1.

I can confirm that taking that command

/Users/brian/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/bin/julia --project=/Users/brian/anaconda3/envs/pyjlcheck/julia_env --startup-file=no -e 'import Pkg; Pkg.add([Pkg.PackageSpec(name="PythonCall", uuid="6099a3de-0909-46bc-b1f4-468b9a2dfc0d")]); Pkg.resolve()'

fails but adding Pkg.Registry.update() works (though we'd probably only want to do this in online mode)

/Users/brian/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/bin/julia --project=/Users/brian/anaconda3/envs/pyjlcheck/julia_env --startup-file=no -e 'import Pkg; Pkg.Registry.update(); Pkg.add([Pkg.PackageSpec(name="PythonCall", uuid="6099a3de-0909-46bc-b1f4-468b9a2dfc0d")]); Pkg.resolve()'

brian-dellabetta · 2023-12-06T22:31:54Z

@cjdoris should we just change it to auto-update registry? I can create a PR, I think it just amounts to changing this line to

script = ['import Pkg', 'Pkg.Registry.update()']

It's already in the if not STATE['offline']: check so we don't need to worry about offline mode

brian-dellabetta · 2023-12-21T13:23:10Z

PR submitted to resolve this in pyjuliapkg -- JuliaPy/pyjuliapkg#22

brian-dellabetta mentioned this issue Dec 21, 2023

explicitly update julia registry during online startup JuliaPy/pyjuliapkg#22

Merged

cjdoris closed this as completed in JuliaPy/pyjuliapkg#22 Mar 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"Unsatisfiable Requirement" edge case error when `juliacall` is upgraded but Julia registries are not #428

"Unsatisfiable Requirement" edge case error when `juliacall` is upgraded but Julia registries are not #428

brian-dellabetta commented Nov 27, 2023

Initial Condition:

Steps to reproduce:

cjdoris commented Nov 29, 2023

brian-dellabetta commented Nov 29, 2023

cjdoris commented Nov 30, 2023

brian-dellabetta commented Dec 1, 2023

brian-dellabetta commented Dec 1, 2023 •

edited

Loading

brian-dellabetta commented Dec 6, 2023 •

edited

Loading

brian-dellabetta commented Dec 21, 2023

"Unsatisfiable Requirement" edge case error when juliacall is upgraded but Julia registries are not #428

"Unsatisfiable Requirement" edge case error when juliacall is upgraded but Julia registries are not #428

Comments

brian-dellabetta commented Nov 27, 2023

Discussed in #422

Initial Condition:

Steps to reproduce:

cjdoris commented Nov 29, 2023

brian-dellabetta commented Nov 29, 2023

cjdoris commented Nov 30, 2023

brian-dellabetta commented Dec 1, 2023

brian-dellabetta commented Dec 1, 2023 • edited Loading

brian-dellabetta commented Dec 6, 2023 • edited Loading

brian-dellabetta commented Dec 21, 2023

"Unsatisfiable Requirement" edge case error when `juliacall` is upgraded but Julia registries are not #428

"Unsatisfiable Requirement" edge case error when `juliacall` is upgraded but Julia registries are not #428

brian-dellabetta commented Dec 1, 2023 •

edited

Loading

brian-dellabetta commented Dec 6, 2023 •

edited

Loading