[BUG] Installation issue: `const PyCall = Base.require` #287

flxmr · 2023-03-31T23:12:57Z

Describe the bug
I am using Python installed in a spack environment and want to use PySR (the following works with the system-python). So I installed Julia, added it to my PATH and then I do the installation instructions, which fail:

>>> import pysr
>>> pysr.install()
[ Info: Julia version info
Julia Version 1.8.5
< all the diagnostics >
┌ Info: PyCall is already installed and compatible with Python executable.
│ 
│ PyCall:
│     python: /software_modules/spack/var/spack/environments/ml-23-03/.spack-env/view/bin/python
│     libpython: /software_modules/spack/opt/spack/linux-fedora36-zen/gcc-8.5.0/python-3.10.10-ldbbqz6242q36jrznozylpqzburhgu2v/lib/libpython3.10.so.1.0
│ Python:
│     python: /software_modules/spack/var/spack/environments/ml-23-03/.spack-env/view/bin/python
└     libpython: /software_modules/spack/opt/spack/linux-fedora36-zen/gcc-8.5.0/python-3.10.10-ldbbqz6242q36jrznozylpqzburhgu2v/lib/libpython3.10.so.1.0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/.local/lib/python3.10/site-packages/pysr/julia_helpers.py", line 85, in install
    Main, init_log = init_julia(julia_project, quiet=quiet, return_aux=True)
  File "/home/.local/lib/python3.10/site-packages/pysr/julia_helpers.py", line 192, in init_julia
    Julia(**julia_kwargs)
  File "/home/.local/lib/python3.10/site-packages/julia/core.py", line 519, in __init__
    self._call("const PyCall = Base.require({0})".format(PYCALL_PKGID))
  File "/home/.local/lib/python3.10/site-packages/julia/core.py", line 555, in _call
    self.check_exception(src)
  File "/home/.local/lib/python3.10/site-packages/julia/core.py", line 609, in check_exception
    raise JuliaError(u'Exception \'{}\' occurred while calling julia code:\n{}'
julia.core.JuliaError: Exception 'InitError' occurred while calling julia code:
const PyCall = Base.require(Base.PkgId(Base.UUID("438e738f-606a-5dbb-bf0a-cddfbfd45ab0"), "PyCall"))

Curiously it works calling the julia-library before:

Python 3.10.10 (main, Mar 31 2023, 22:33:35) [GCC 8.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import julia
>>> julia.Julia()
<julia.core.LegacyJulia object at 0x7f7fff0a0550>
>>> import pysr
>>> pysr.install()

(I can run the example in this REPL)

Closing that REPL and trying to run the example again, I arrive at:

>>> model.fit(X, y)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/.local/lib/python3.10/site-packages/pysr/sr.py", line 1834, in fit
    self._run(X, y, mutated_params, weights=weights, seed=seed)
  File "/home/.local/lib/python3.10/site-packages/pysr/sr.py", line 1525, in _run
    Main = init_julia(self.julia_project, julia_kwargs=julia_kwargs)
  File "/home/.local/lib/python3.10/site-packages/pysr/julia_helpers.py", line 192, in init_julia
    Julia(**julia_kwargs)
  File "/home/.local/lib/python3.10/site-packages/julia/core.py", line 519, in __init__
    self._call("const PyCall = Base.require({0})".format(PYCALL_PKGID))
  File "/home/.local/lib/python3.10/site-packages/julia/core.py", line 555, in _call
    self.check_exception(src)
  File "/home/.local/lib/python3.10/site-packages/julia/core.py", line 609, in check_exception
    raise JuliaError(u'Exception \'{}\' occurred while calling julia code:\n{}'
julia.core.JuliaError: Exception 'InitError' occurred while calling julia code:
const PyCall = Base.require(Base.PkgId(Base.UUID("438e738f-606a-5dbb-bf0a-cddfbfd45ab0"), "PyCall"))

This is weird, do you have any idea what could be the problem? Should I raise this with the Pycall-Developers?

Version (please include the following information):

OS: Linux
Julia 1.8.5
Python 3.10.10
pip-install
PySR 0.12.1

The text was updated successfully, but these errors were encountered:

MilesCranmer · 2023-04-01T07:55:19Z

One thing to try, which is a bit of a hack but might fix it — manually install and build PyCall in the base Julia environment:

julia -e 'using Pkg; Pkg.install("PyCall"); Pkg.build("PyCall")'

Run this line in bash in any directory, doesn’t matter which. (Make sure to have the same Python available on your path as if you were using pysr.install())

This should add PyCall to the base env which might fix this. Let me know if this works. If not I would then try the solutions in #257

Cheers,
Miles

dbl001 · 2023-05-07T17:40:09Z

Perhaps this is the same issue ...
I can run the Feynman problems from iPython in a terminal, however, I get this error running the code from inside 'Pycharm'


/Users/davidlaxer/anaconda3/envs/pysr_test/bin/python /Users/davidlaxer/PySR/test.py 
DEBUG (28160) 
DEBUG (28160) Debug-level logging is enabled for PyJulia.
DEBUG (28160) PyJulia version: 0.6.0
DEBUG (28160) 
/Users/davidlaxer/anaconda3/envs/pysr_test/lib/python3.11/site-packages/julia/juliainfo.py:93: UserWarning: julia warned:
The latest version of Julia in the `1.9` channel is 1.9.0-rc3+0.x64.apple.darwin14. You currently have `1.9.0-rc2+0.x64.apple.darwin14` installed. Run:

  juliaup update

to install Julia 1.9.0-rc3+0.x64.apple.darwin14 and update the `1.9` channel to that version.
  warnings.warn("{} warned:\n{}".format(julia, stderr))
DEBUG (28160) pyprogramname = None
DEBUG (28160) sys.executable = /Users/davidlaxer/anaconda3/envs/pysr_test/bin/python
DEBUG (28160) bindir = /Users/davidlaxer/.julia/juliaup/julia-1.9.0-rc2+0.x64.apple.darwin14/bin
DEBUG (28160) libjulia_path = /Users/davidlaxer/.julia/juliaup/julia-1.9.0-rc2+0.x64.apple.darwin14/lib/libjulia.1.9.dylib
DEBUG (28160) is_compatible_python = None
DEBUG (28160) use_custom_sysimage = False
DEBUG (28160) compiled_modules = 'no'
DEBUG (28160) argv_list = [b'/Users/davidlaxer/anaconda3/envs/pysr_test/bin/python', b'--compiled-modules=no', b'--threads=auto']
DEBUG (28160) argc = c_int(3)
DEBUG (28160) jl_parse_opts called
DEBUG (28160) argc = c_int(0)
DEBUG (28160) calling jl_init_with_image(/Users/davidlaxer/.julia/juliaup/julia-1.9.0-rc2+0.x64.apple.darwin14/bin, /Users/davidlaxer/.julia/juliaup/julia-1.9.0-rc2+0.x64.apple.darwin14/lib/julia/sys.dylib)
DEBUG (28160) seems to work...
┌ Error: Failed to import PyCall
│   exception =
│    ArgumentError: Package PyCall [438e738f-606a-5dbb-bf0a-cddfbfd45ab0] is required but does not seem to be installed:
│     - Run `Pkg.instantiate()` to install all recorded dependencies.
│    
│    Stacktrace:
│     [1] _require(pkg::Base.PkgId, env::Nothing)
│       @ Base ./loading.jl:1741
│     [2] _require_prelocked(uuidkey::Base.PkgId, env::Nothing)
│       @ Base ./loading.jl:1623
│     [3] _require_prelocked
│       @ ./loading.jl:1621 [inlined]
│     [4] macro expansion
│       @ ./lock.jl:267 [inlined]
│     [5] require(uuidkey::Base.PkgId)
│       @ Base ./loading.jl:1618
│     [6] top-level scope
│       @ none:3
└ @ Main none:5
DEBUG (28160) exception occured? 5841618960
Traceback (most recent call last):
  File "/Users/davidlaxer/PySR/test.py", line 11, in <module>
    julia = Julia(compiled_modules=False, debug=True, threads='auto')
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/davidlaxer/anaconda3/envs/pysr_test/lib/python3.11/site-packages/julia/core.py", line 702, in __init__
    self.__julia = Julia(*args, **kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/davidlaxer/anaconda3/envs/pysr_test/lib/python3.11/site-packages/julia/core.py", line 510, in __init__
    self._call("""
  File "/Users/davidlaxer/anaconda3/envs/pysr_test/lib/python3.11/site-packages/julia/core.py", line 555, in _call
    self.check_exception(src)
  File "/Users/davidlaxer/anaconda3/envs/pysr_test/lib/python3.11/site-packages/julia/core.py", line 609, in check_exception
    raise JuliaError(u'Exception \'{}\' occurred while calling julia code:\n{}'
julia.core.JuliaError: Exception 'ArgumentError' occurred while calling julia code:

            const PyCall = try
                Base.require(Base.PkgId(Base.UUID("438e738f-606a-5dbb-bf0a-cddfbfd45ab0"), "PyCall"))
            catch err
                @error "Failed to import PyCall" exception = (err, catch_backtrace())
                rethrow()
            end
            

SYSTEM: caught exception of type :ArgumentError while trying to print a failed Task notice; giving up

Process finished with exit code 1

Here's the test script:

import os

# Set the number of threads for Julia
os.environ["JULIA_NUM_THREADS"] = "4"

import pysr

from julia import Julia
julia = Julia(compiled_modules=False, debug=True, threads='auto')

from julia import Main
from julia.tools import redirect_output_streams

#redirect_output_streams()

pysr.install()

from pysr.sr import pysr, best
from pysr.feynman_problems import FeynmanProblem
from pysr.feynman_problems import mk_problems
from pysr.feynman_problems import run_on_problem
from pysr.feynman_problems import do_feynman_experiments
from pysr.feynman_problems import do_feynman_experiments_parallel


import sympy
from sympy.core.random import _randint
import numpy as np
from matplotlib import pyplot as plt
from pysr import PySRRegressor
import torch
from torch import nn, optim
from torch.nn import functional as F
from torch.utils.data import DataLoader, TensorDataset
import pytorch_lightning as pl
from sklearn.model_selection import train_test_split


do_feynman_experiments()

Environment:

% julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.9.0-rc3 (2023-04-26)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using Pkg

julia> Pkg.add("PyCall")
   Resolving package versions...
  No Changes to `~/anaconda3/envs/pysr_test/share/julia/environments/pysr_test/Project.toml`
  No Changes to `~/anaconda3/envs/pysr_test/share/julia/environments/pysr_test/Manifest.toml`

julia> Pkg.update("PyCall")
  No Changes to `~/anaconda3/envs/pysr_test/share/julia/environments/pysr_test/Project.toml`
  No Changes to `~/anaconda3/envs/pysr_test/share/julia/environments/pysr_test/Manifest.toml`

julia> 
(pysr_test) davidlaxer@bluediamond julia % ipython          
Python 3.11.0 | packaged by conda-forge | (main, Oct 25 2022, 06:24:51) [Clang 14.0.4 ]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.7.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from julia.api import JuliaInfo

In [2]: info = JuliaInfo.load()

In [3]: info.is_compatible_python()
Out[3]: False

In [4]: info.python
Out[4]: '/Users/davidlaxer/anaconda3/envs/pysr_test/bin/python'

In [5]: info.sysimage
Out[5]: '/Users/davidlaxer/anaconda3/envs/pysr_test/share/julia/juliaup/julia-1.9.0-rc3+0.x64.apple.darwin14/lib/julia/sys.dylib'

dbl001 · 2023-05-20T02:51:42Z

In my environment, PyCharm was invoking julia in:

/Users/davidlaxer/.julia/juliaup/julia-1.9.0+0.x64.apple.darwin14/bin/julia

instead of from my conda virtual environment:

/Users/davidlaxer/anaconda3/envs/pysr_test//bin/julia

The julia environment in: /Users/davidlaxer/.julia/juliaup/julia-1.9.0+0.x64.apple.darwin14/bin/julia did not have 'pycall' installed.

MilesCranmer · 2023-05-20T12:16:00Z

Hm, this might be a different issue. Did you install PySR with pip instead of conda?

dbl001 · 2023-05-20T13:39:40Z

Look like: $ python setup.py develop

% conda list pysr
# packages in environment at /Users/davidlaxer/anaconda3/envs/pysr_test:
#
# Name                    Version                   Build  Channel
pysr                      0.12.3                    dev_0    <develop>
(pysr_test) davidlaxer@bluediamond julia % pip show pysr
Name: pysr
Version: 0.11.11
Summary: Simple and efficient symbolic regression
Home-page: https://github.com/MilesCranmer/pysr
Author: Miles Cranmer
Author-email: [email protected]
License: 
Location: /Users/davidlaxer/anaconda3/envs/pysr_test/lib/python3.11/site-packages
Requires: julia, numpy, pandas, scikit-learn, sympy
Required-by:

MilesCranmer · 2023-05-20T14:50:57Z

Okay I think that's a different issue. If using conda's Julia then you likely would also need to use conda's PySR (https://github.com/conda-forge/pysr-feedstock) so it gets the paths correctly.

MilesCranmer · 2024-02-12T09:02:37Z

Presumably fixed by #535, so closing.

flxmr added the bug Something isn't working label Mar 31, 2023

flxmr assigned MilesCranmer Mar 31, 2023

MilesCranmer changed the title ~~[BUG] Installation doesn't work~~ [BUG] Installation issue: const PyCall = Base.require Apr 1, 2023

MilesCranmer removed their assignment Apr 20, 2023

MilesCranmer mentioned this issue May 10, 2023

[BUG] Errors when building PyCall.jl #257

Closed

MilesCranmer mentioned this issue Jul 2, 2023

[WIP] Use preferences.jl to set python executable JuliaPy/PyCall.jl#945

Draft

MilesCranmer closed this as completed Feb 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] Installation issue: `const PyCall = Base.require` #287

[BUG] Installation issue: `const PyCall = Base.require` #287

flxmr commented Mar 31, 2023

MilesCranmer commented Apr 1, 2023

dbl001 commented May 7, 2023 •

edited

Loading

dbl001 commented May 20, 2023

MilesCranmer commented May 20, 2023

dbl001 commented May 20, 2023

MilesCranmer commented May 20, 2023

MilesCranmer commented Feb 12, 2024

[BUG] Installation issue: const PyCall = Base.require #287

[BUG] Installation issue: const PyCall = Base.require #287

Comments

flxmr commented Mar 31, 2023

MilesCranmer commented Apr 1, 2023

dbl001 commented May 7, 2023 • edited Loading

dbl001 commented May 20, 2023

MilesCranmer commented May 20, 2023

dbl001 commented May 20, 2023

MilesCranmer commented May 20, 2023

MilesCranmer commented Feb 12, 2024

[BUG] Installation issue: `const PyCall = Base.require` #287

[BUG] Installation issue: `const PyCall = Base.require` #287

dbl001 commented May 7, 2023 •

edited

Loading