Use Julia 1.12, Pkg workspaces and JuliaC sans trimming #2629

visr · 2025-10-09T14:45:19Z

Supersedes #2440.

Let's see about the tests. From the original PR this point stands:

One PR should give 1.12 compat since julia_init is removed there. We need to update our Rust wrapper for that as well.

This may apply to ribasim_api package as well as the Rust wrapper.
See also https://github.com/JuliaLang/PackageCompiler.jl/pull/1047/files and https://docs.julialang.org/en/v1/manual/embedding/

visr · 2025-10-09T19:53:22Z

Ugh:

Full SciMLBasePythonCallExt precompile error

ERROR: The following 1 direct dependency failed to precompile:

SciMLBasePythonCallExt --code-coverage=@/home/runner/work/Ribasim/Ribasim/core --color=auto --check-bounds=yes --warn-overwrite=yes --depwarn=yes --inline=yes --startup-file=no --track-allocation=none 

Failed to precompile SciMLBasePythonCallExt [2797fd30-2078-5027-980c-4c2c8a19c528] to "/home/runner/.julia/compiled/v1.12/SciMLBasePythonCallExt/jl_n1a9h2".
CondaPkg Initialising pixi
             │ /home/runner/.julia/artifacts/cefba4912c2b400756d043a2563ef77a0088866b/bin/pixi
             │ init
             │ --format pixi
             └ /home/runner/work/Ribasim/Ribasim/core/.CondaPkg
✔ Created /home/runner/work/Ribasim/Ribasim/core/.CondaPkg/pixi.toml
    CondaPkg Wrote /home/runner/work/Ribasim/Ribasim/core/.CondaPkg/pixi.toml
             │ [dependencies]
             │ 
             │ [project]
             │ name = ".CondaPkg"
             │ platforms = ["linux-64"]
             │ channels = ["conda-forge"]
             │ channel-priority = "strict"
             └ description = "automatically generated by CondaPkg.jl"
    CondaPkg Installing packages
             │ /home/runner/.julia/artifacts/cefba4912c2b400756d043a2563ef77a0088866b/bin/pixi
             │ install
             └ --manifest-path /home/runner/work/Ribasim/Ribasim/core/.CondaPkg/pixi.toml
 WARN Using local manifest /home/runner/work/Ribasim/Ribasim/core/.CondaPkg/pixi.toml rather than /home/runner/work/Ribasim/Ribasim/pixi.toml from environment variable `PIXI_PROJECT_MANIFEST`
✔ The default environment has been installed.
ERROR: LoadError: InitError: Python executable "/home/runner/work/Ribasim/Ribasim/core/.CondaPkg/.pixi/envs/default/bin/python" does not exist.
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:44
  [2] init_context()
    @ PythonCall.C ~/.julia/packages/PythonCall/mkWc2/src/C/context.jl:108
  [3] __init__()
    @ PythonCall.C ~/.julia/packages/PythonCall/mkWc2/src/C/C.jl:26
  [4] run_module_init(mod::Module, i::Int64)
    @ Base ./loading.jl:1415
  [5] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1403
  [6] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any}; register::Bool)
    @ Base ./loading.jl:1291
  [7] _include_from_serialized
    @ ./loading.jl:1246 [inlined]
  [8] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128, stalecheck::Bool; reasons::Dict{String, Int64}, DEPOT_PATH::Vector{String})
    @ Base ./loading.jl:2087
  [9] _require_search_from_serialized
    @ ./loading.jl:1981 [inlined]
 [10] __require_prelocked(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:2599
 [11] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:2465
 [12] macro expansion
    @ ./loading.jl:2393 [inlined]
 [13] macro expansion
    @ ./lock.jl:376 [inlined]
 [14] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2358
 [15] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2334
 [16] top-level scope
    @ ~/.julia/packages/SciMLBase/YE7xF/ext/SciMLBasePythonCallExt.jl:3
 [17] include(mod::Module, _path::String)
    @ Base ./Base.jl:306
 [18] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
    @ Base ./loading.jl:2997
 [19] top-level scope
    @ stdin:5
 [20] eval(m::Module, e::Any)
    @ Core ./boot.jl:489
 [21] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)
    @ Base ./loading.jl:2843
 [22] include_string
    @ ./loading.jl:2853 [inlined]
 [23] exec_options(opts::Base.JLOptions)
    @ Base ./client.jl:315
 [24] _start()
    @ Base ./client.jl:550

I've seen this pop up before. We don't use SciMLBasePythonCallExt, but it gets included since we use both SciMLBase and PythonCall. (though only in root?)

Trying the Null option from https://github.com/JuliaPy/CondaPkg.jl in fa7c122

…resized concurrently")`

visr · 2025-10-10T07:46:51Z

PackageCompiler crashes on

PackageCompiler: compiling fresh sysimage (incremental=false)

On Linux we get ProcessSignaled(6)

On Windows

[12820] signal 22: SIGABRT
ProcessExited(3) on Windows

It could be a TeamCity issue, or a PackageCompiler issue.

I wanted to do it in a separate PR, but I'm tempted to directly switch to https://github.com/JuliaLang/JuliaC.jl/ (without trimming) here. Based on an early effort in #2460.

See #2629 But now setting this for all tasks at once, because we never want to create new enviornments. This failed again on the same PR in the JuliaC precompilation step.

visr · 2025-10-12T08:55:37Z

Status: this produces a fully functional binary of the same size, and all tests pass. Building the binary seems quite a bit faster.

However I did notice that running the basic model tool 1m30s total (not just computation time), vs 30s. JuliaC doesn't support using our precompile.jl script, so I removed that. I tried changing our preferences to:

[preferences.Ribasim]
precompile_workload = true
specialize = true

This caused the compilation to take longer, but seemed to have no effect on latency. Possibly I need to try again or just hardcode this for a bit.

So this needs some more investigation to get the precompile workload to be effective to avoid the 3x latency regression.

visr · 2025-10-12T10:05:37Z

Ah nevermind I set the preferences in the root TOML instead of the core TOML. Now the latency is about the same, as expected. I will fix up the preferences mechanism without precompile.jl.

The main thing that is different now is that the CPU target is set to native. I'm worried that that will make it less portable to other machines, so we'll need to do something about that before merging. See JuliaLang/JuliaC.jl#33.

EDIT: both done in 2d698c8

evetion · 2025-10-13T11:49:34Z

core/test/libribasim_test.jl

@@ -1,4 +1,8 @@
-@testitem "libribasim" begin
+@testmodule libribasim begin
+    include("../../build/libribasim.jl")


Maybe a norm/joinpath here?

Why? The include docs mention

The argument path is normalized using normpath which will resolve relative path tokens such as .. and convert / to the appropriate path separator.

evetion · 2025-10-13T11:50:20Z

python/ribasim_api/tests/conftest.py


-@pytest.fixture(scope="session")
-def basic_transient(basic) -> ribasim.Model:
-    return basic_transient_model(basic)


What did this temporal model do?

It wasn't used in the tests, so I removed it.

This reverts commit 5356f1e.

This reverts #2629 and #2644. The main reason is that we were only able to build working binaries with cpu_target native. That meant the TeamCity binaries were much less portable to other machines than before. Once that is resolved (we made an upstream issue in JuliaC), we can go back to 1.12.

visr added 4 commits October 9, 2025 16:00

Set Julia version to 1.12.0 and update Manifest

d95802e

Set up Pkg workspace

4c2c7a8

Merge branch 'main' into julia-1.12.0

35bef97

Try to avoid SciMLBasePythonCallExt precompile error

fa7c122

visr added 3 commits October 9, 2025 22:16

Add Glob to test project as well

5554b19

Fix `nested task error: ConcurrencyViolationError("Vector can not be …

172c075

…resized concurrently")`

Add missing test dependencies

2226329

visr added 5 commits October 10, 2025 10:20

Remove SSL_CERT workaround

707ce10

Merge branch 'main' into julia-1.12.0

37fde32

Merge branch 'main' into julia-1.12.0

e00eb44

Fix Aqua test

df4be3e

Move out libribasim

fae6cf4

visr added a commit that referenced this pull request Oct 10, 2025

Set JULIA_CONDAPKG_BACKEND to Null

0742aea

See #2629 But now setting this for all tasks at once, because we never want to create new enviornments. This failed again on the same PR in the JuliaC precompilation step.

visr mentioned this pull request Oct 10, 2025

Set JULIA_CONDAPKG_BACKEND to Null #2635

Merged

visr added a commit that referenced this pull request Oct 10, 2025

Set JULIA_CONDAPKG_BACKEND to Null (#2635)

a593d47

See #2629 But now setting this for all tasks at once, because we never want to create new enviornments. This failed again on the same PR in the JuliaC precompilation step.

visr added 6 commits October 10, 2025 20:20

Unpack the module

6bee1f2

Merge branch 'main' into julia-1.12.0

69f4705

Use JuliaC to compile

8437510

Fixes

60f88ae

oops

d90e88f

Remove init_julia

b9dbd69

visr changed the title ~~Use Julia 1.12 and Pkg workspaces; release edition~~ Use Julia 1.12, Pkg workspaces and JuliaC sans trimming Oct 12, 2025

Set preferences to release options during build

2d698c8

visr marked this pull request as ready for review October 13, 2025 11:24

Merge branch 'main' into julia-1.12.0

2c5c806

evetion approved these changes Oct 13, 2025

View reviewed changes

visr merged commit 5356f1e into main Oct 13, 2025
19 of 24 checks passed

visr deleted the julia-1.12.0 branch October 13, 2025 11:58

visr added a commit that referenced this pull request Oct 16, 2025

Revert "Use Julia 1.12, Pkg workspaces and JuliaC sans trimming (#2629)"

8191b56

This reverts commit 5356f1e.

visr mentioned this pull request Oct 16, 2025

Go back to Julia 1.11 #2649

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Use Julia 1.12, Pkg workspaces and JuliaC sans trimming #2629

Use Julia 1.12, Pkg workspaces and JuliaC sans trimming #2629

Uh oh!

visr commented Oct 9, 2025

Uh oh!

visr commented Oct 9, 2025 •

edited

Loading

Uh oh!

visr commented Oct 10, 2025 •

edited

Loading

Uh oh!

visr commented Oct 12, 2025

Uh oh!

visr commented Oct 12, 2025 •

edited

Loading

Uh oh!

evetion Oct 13, 2025

Uh oh!

visr Oct 13, 2025

Uh oh!

evetion Oct 13, 2025

Uh oh!

visr Oct 13, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Use Julia 1.12, Pkg workspaces and JuliaC sans trimming #2629

Use Julia 1.12, Pkg workspaces and JuliaC sans trimming #2629

Uh oh!

Conversation

visr commented Oct 9, 2025

Uh oh!

visr commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

visr commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

visr commented Oct 12, 2025

Uh oh!

visr commented Oct 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

evetion Oct 13, 2025

Choose a reason for hiding this comment

Uh oh!

visr Oct 13, 2025

Choose a reason for hiding this comment

Uh oh!

evetion Oct 13, 2025

Choose a reason for hiding this comment

Uh oh!

visr Oct 13, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

visr commented Oct 9, 2025 •

edited

Loading

visr commented Oct 10, 2025 •

edited

Loading

visr commented Oct 12, 2025 •

edited

Loading