(BREAKING) Rewrite Julia interface with PyJulia -> JuliaCall; other changes

[MilesCranmer]

I managed to get a PythonCall.jl/JuliaCall version of PySR working!

Tagging @mkitti @cjdoris in case you are curious. Any suggestions about implementation are very appreciated :)

Detailed changes:

• (BREAKING) Changed PyJulia with JuliaCall
  • Swapping eval -> seval
  • Manually converting to Vector when calling SymbolicRegression.jl functions (otherwise would get passed as PyList{Any}; see Working with type instabilities, coming from PyJulia JuliaPy/PythonCall.jl#441)
  • Wrapped equation_search code with jl.PythonCall.GC.disable() to avoid multithreading-related segfaults (Bus error with multithreading linear algebra JuliaPy/PythonCall.jl#298)
  • Manually convert np.str_ to str before passing to variable_names, otherwise it becomes a PyArray and not a String (@cjdoris might be worth adding a workaround, it seems like PyJulia does this automatically)
• (BREAKING) The user no longer needs to run python -m pysr install. The install process is done by JuliaCall at import time.
  • Removed code related to pysr.install() and python -m pysr install because JuliaCall now handles this.
  • python -m pysr install will not give a warning and do nothing.
• (BREAKING) Remove the feynman problems dataset. I don't think anybody used it, and doesn't seem good to have in a library itself.
• (BREAKING) Deprecated julia_project argument (ignored; no effect). The user now needs to set this up by customizing juliapkg.json.
• Dependencies are now managed by pyjuliapkg rather than the custom code we made. Simplifies things a lot!
• Rather than storing the raw julia variables in PySRRegressor, I am now storing a serialized version of them. This means you can now pickle the search state and warm-start the search in another Python process. It feels much safer to do it like this.
  • Not breaking! Because self.raw_julia_state_ will actually deserialize it automatically for you
• SymbolicRegression is now available to import from PySR:

from pysr import SymbolicRegression as SR
x1 = SR.Node(feature=1)  # Create expressions manually

• SymbolicRegression options are accessible in model.julia_options_ (generated from a serialized format for pickle safety)
• Deprecated various kwargs to match SymbolicRegression.jl (old names will still work):
  • ncyclesperiteration => ncycles_per_iteration
  • loss => elementwise_loss
  • full_objective => loss_function

TODO:

• Fix remaining test issues
• Decide if I want to import JuliaCall lazily (at first run) or if it gets installed from simply import pysr.
  • Will do at import.
• Add back pysr.install with a warning message.
• Add other unnecessary functions with simple deprecation messages.
• Fixes needed:
  • Using development version of package opens Julia REPL JuliaPy/PythonCall.jl#451
    • Workaround implemented
  • Main.seval can't parse multiple toplevel blocks JuliaPy/PythonCall.jl#433
    • Workaround implemented, though still broken on Julia < 1.9.0
  • jl.println and jl.display do not work in Jupyter notebook JuliaPy/PythonCall.jl#232
    • Has workaround (implemented)
  • Add pyjuliacall pyjuliapkg via grayskull conda-forge/staged-recipes#25097
• Remove click dependency. (Leaving for backwards compatibility)
• Test custom julia_project with development version of SymbolicRegression.jl.
  • Decide if we need to deprecate this in favor of telling the user to customize the juliapkg.json file.
  • (We will deprecate this)
• Use the git version of SymbolicRegression.jl rather than registry version.
• Test large distributed run on a slurm cluster.
• Find other unused pieces of code and remove them.
• Ensure coverage is pretty good.
• Make sure we test all new code.
• Set environment variable for -O3.
• Raise warning if Julia was started without multiple threads, without optimization.
• Add an integration test for restarting a warm start fit in a new Python session.
• Get all tests passing
• Make sure that the old Julia versions are actually being tested and PythonCall.jl isn't just installing Julia 1.10 on them (maybe with an env var?)
• Update Colab notebook (and test it)
• Update examples in docs to use seval
• Make SymbolicRegression available (and save the Options)
• Unittest behavior in:
  • jupyter notebook
  • IPython [skipping for now...]

Post:

• Submit as an integration test to PythonCall

[cjdoris]

Could you make an issue about converting np.str_ please?

[MilesCranmer]

Still some random issues it seems... Not sure if these are a blocker because PyJulia has had worse issues but still some build weirdness. I ran into this one time (can't reproduce):

[ins] In [1]: from pysr import PySRRegressor, jl
ERROR: ArgumentError: Package PythonCall [6099a3de-0909-46bc-b1f4-468b9a2dfc0d] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

Stacktrace:
  [1] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1920
  [2] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1806
  [3] #invoke_in_world#3
    @ Base ./essentials.jl:921 [inlined]
  [4] invoke_in_world
    @ Base ./essentials.jl:918 [inlined]
  [5] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1797
  [6] macro expansion
    @ Base ./loading.jl:1784 [inlined]
  [7] macro expansion
    @ Base ./lock.jl:267 [inlined]
  [8] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1747
  [9] #invoke_in_world#3
    @ Base ./essentials.jl:921 [inlined]
 [10] invoke_in_world
    @ Base ./essentials.jl:918 [inlined]
 [11] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1740
 [12] top-level scope
    @ none:8---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
Cell In[1], line 1
----> 1 from pysr import PySRRegressor, jl

File ~/PermaDocuments/SymbolicRegressionMonorepo/.venv/lib/python3.10/site-packages/pysr/__init__.py:4
      1 # This must be imported as early as possible to prevent
      2 # library linking issues caused by numpy/pytorch/etc. importing
      3 # old libraries:
----> 4 from .julia_import import jl, SymbolicRegression  # isort:skip
      6 from . import sklearn_monkeypatch
      7 from .deprecated import best, best_callable, best_row, best_tex, install, pysr

File ~/PermaDocuments/SymbolicRegressionMonorepo/.venv/lib/python3.10/site-packages/pysr/julia_import.py:59
     53 elif os.environ["PYSR_AUTOLOAD_EXTENSIONS"] not in {"no", "yes", ""}:
     54     warnings.warn(
     55         "PYSR_AUTOLOAD_EXTENSIONS environment variable is set to something other than 'yes' or 'no' or ''."
     56     )
---> 59 from juliacall import Main as jl  # type: ignore
     62 # Finally, overwrite the seval function to use Meta.parseall
     63 # instead of Meta.parse.
     64 # TODO: Overwrite this once PythonCall.jl is updated:
     65 def seval(s: str):

File ~/PermaDocuments/SymbolicRegressionMonorepo/.venv/lib/python3.10/site-packages/juliacall/__init__.py:228
    224         raise Exception('PythonCall.jl did not start properly')
    226     CONFIG['inited'] = True
--> 228 init()
    230 def load_ipython_extension(ip):
    231     import juliacall.ipython

File ~/PermaDocuments/SymbolicRegressionMonorepo/.venv/lib/python3.10/site-packages/juliacall/__init__.py:224, in init()
    222 res = jl_eval(script.encode('utf8'))
    223 if res is None:
--> 224     raise Exception('PythonCall.jl did not start properly')
    226 CONFIG['inited'] = True

Exception: PythonCall.jl did not start properly

but seems like I couldn't solve it without deleting my environment and starting over.

[MilesCranmer]

Thanks both for your tips again! I'm very excited to get this through!! 😃