Jupyternim fails to install on macOS

[AngelEzquerra]

Installation on macOS fails:

 Installing jupyternim@0.8.0
   Building jupyternim/jupyternim using c backend
   Warning: Symlink already exists in /Users/myuser/.nimble/bin/jupyternim. Replacing.
   Success: jupyternim installed successfully.
    Prompt: Missing directory /Users/myuser/.nimble/pkgs/jupyternim-0.8.0/src/jupyternimpkg. Continue? [y/N]
    Answer: y
    Prompt: Missing directory /Users/myuser/.nimble/pkgs/jupyternim-0.8.0/src/jupyternimspec. Continue? [y/N]
    Answer: y

Then when I try to run jupyternim from a shell I get a "command not found" error.

[stisa]

Thanks for the report. Unfortunately I don't have a mac to test, my best guess is a problem with compatibility with nimPNG (1717fa4).

You could try passing --verbose to nimble when you install to see if there's more information.

[AngelEzquerra]

This is the output I got when I ran nimble install --verbose jupyternim:

    Reading official package list
Downloading https://github.com/stisa/jupyternim using git
    Cloning latest tagged version: 0.8.0
  Verifying dependencies for jupyternim@0.8.0
    Reading official package list
   Checking for zmq@>= 1.2.1 & < 1.3.0
      Info: Dependency on zmq@>= 1.2.1 & < 1.3.0 already satisfied
  Verifying dependencies for zmq@1.2.1
    Reading official package list
   Checking for hmac@>= 0.2.0 & < 0.3.0
      Info: Dependency on hmac@>= 0.2.0 & < 0.3.0 already satisfied
  Verifying dependencies for hmac@0.2.0
    Reading official package list
   Checking for nimSHA2@any version
      Info: Dependency on nimSHA2@any version already satisfied
  Verifying dependencies for nimSHA2@0.1.1
    Reading official package list
   Checking for sha1@any version
      Info: Dependency on sha1@any version already satisfied
  Verifying dependencies for sha1@1.1
    Reading official package list
   Checking for nimcrypto@>= 0.5.4
      Info: Dependency on nimcrypto@>= 0.5.4 already satisfied
  Verifying dependencies for nimcrypto@0.5.4
    Reading official package list
   Checking for nimSHA2@#b8f666069dff1ed0c5142dd1ca692f0e71434716
      Info: Dependency on nimSHA2@#b8f666069dff1ed0c5142dd1ca692f0e71434716 already satisfied
  Verifying dependencies for nimSHA2@#b8f666069dff1ed0c5142dd1ca692f0e71434716
    Reading official package list
   Checking for nimPNG@#2dccf3e19ceb3fdf3f3f415e717d9d336d473a19
      Info: Dependency on nimPNG@#2dccf3e19ceb3fdf3f3f415e717d9d336d473a19 already satisfied
  Verifying dependencies for nimPNG@#2dccf3e19ceb3fdf3f3f415e717d9d336d473a19
 Installing jupyternim@0.8.0
   Building jupyternim/jupyternim using c backend
  Executing /opt/homebrew/Cellar/nim/1.6.6/nim/bin/nim c --colors:on --noNimblePath -d:release -d:NimblePkgVersion=0.8.0 --path:/Users/myuser/.nimble/pkgs/zmq-1.2.1 --path:/Users/myuser/.nimble/pkgs/hmac-0.2.0 --path:/Users/myuser/.nimble/pkgs/nimSHA2-0.1.1 --path:/Users/myuser/.nimble/pkgs/sha1-1.1 --path:/Users/myuser/.nimble/pkgs/nimcrypto-0.5.4 --path:'/Users/myuser/.nimble/pkgs/nimSHA2-#b8f666069dff1ed0c5142dd1ca692f0e71434716' --path:'/Users/myuser/.nimble/pkgs/nimPNG-#2dccf3e19ceb3fdf3f3f415e717d9d336d473a19' -o:/var/folders/lz/6ktvh__54hz3rjmrry69dbqc0000gn/T/nimble_15599/githubcom_stisajupyternim/jupyternim /var/folders/lz/6ktvh__54hz3rjmrry69dbqc0000gn/T/nimble_15599/githubcom_stisajupyternim/src/jupyternim.nim
Hint: used config file '/opt/homebrew/Cellar/nim/1.6.6/nim/config/nim.cfg' [Conf]
Hint: used config file '/opt/homebrew/Cellar/nim/1.6.6/nim/config/config.nims' [Conf]
.................................................................................................................................................................
/private/var/folders/lz/6ktvh__54hz3rjmrry69dbqc0000gn/T/nimble_15599/githubcom_stisajupyternim/src/jupyternimpkg/sockets.nim(113, 6) Hint: 'startCodeServer' is declared but not used [XDeclaredButNotUsed]
/private/var/folders/lz/6ktvh__54hz3rjmrry69dbqc0000gn/T/nimble_15599/githubcom_stisajupyternim/src/jupyternimpkg/sockets.nim(185, 7) Hint: 'MagicsStrings' is declared but not used [XDeclaredButNotUsed]
/private/var/folders/lz/6ktvh__54hz3rjmrry69dbqc0000gn/T/nimble_15599/githubcom_stisajupyternim/src/jupyternimpkg/sockets.nim(393, 6) Hint: 'parseNimsuggest' is declared but not used [XDeclaredButNotUsed]
/private/var/folders/lz/6ktvh__54hz3rjmrry69dbqc0000gn/T/nimble_15599/githubcom_stisajupyternim/src/jupyternimpkg/sockets.nim(73, 7) Hint: 'codeserver' is declared but not used [XDeclaredButNotUsed]
...
/private/var/folders/lz/6ktvh__54hz3rjmrry69dbqc0000gn/T/nimble_15599/githubcom_stisajupyternim/src/jupyternim.nim(8, 1) Hint: duplicate import of 'strutils'; previous import here: /private/var/folders/lz/6ktvh__54hz3rjmrry69dbqc0000gn/T/nimble_15599/githubcom_stisajupyternim/src/jupyternim.nim(2, 18) [DuplicateModuleImport]
.
CC: stdlib_digitsutils.nim
CC: stdlib_assertions.nim
CC: stdlib_formatfloat.nim
CC: stdlib_dollars.nim
CC: stdlib_io.nim
CC: stdlib_system.nim
CC: ../../../../../../../../../Users/myuser/.nimble/pkgs/zmq-1.2.1/zmq/bindings.nim
CC: stdlib_parseutils.nim
CC: stdlib_unicode.nim
CC: stdlib_math.nim
CC: stdlib_algorithm.nim
CC: stdlib_strutils.nim
CC: stdlib_strformat.nim
CC: ../../../../../../../../../Users/myuser/.nimble/pkgs/zmq-1.2.1/zmq/connections.nim
CC: stdlib_pathnorm.nim
CC: stdlib_posix.nim
CC: stdlib_options.nim
CC: stdlib_times.nim
CC: stdlib_os.nim
CC: stdlib_hashes.nim
CC: stdlib_tables.nim
CC: stdlib_asyncfutures.nim
CC: stdlib_monotimes.nim
CC: stdlib_nativesockets.nim
CC: stdlib_kqueue.nim
CC: stdlib_selectors.nim
CC: stdlib_asyncdispatch.nim
CC: stdlib_streams.nim
CC: stdlib_lexbase.nim
CC: stdlib_parsejson.nim
CC: stdlib_json.nim
CC: stdlib_strtabs.nim
CC: stdlib_cpuinfo.nim
CC: stdlib_osproc.nim
CC: stdlib_base64.nim
CC: ../../../../../../../../../Users/myuser/.nimble/pkgs/nimSHA2-#b8f666069dff1ed0c5142dd1ca692f0e71434716/nimSHA2.nim
CC: ../../../../../../../../../Users/myuser/.nimble/pkgs/hmac-0.2.0/hmac.nim
CC: stdlib_sysrand.nim
CC: stdlib_random.nim
CC: jupyternimpkg/utils.nim
CC: jupyternimpkg/messages.nim
CC: stdlib_mimetypes.nim
CC: jupyternimpkg/display.nim
CC: jupyternimpkg/sockets.nim
CC: stdlib_exitprocs.nim
CC: stdlib_dynlib.nim
CC: jupyternim.nim
Hint:  [Link]
Hint: gc: refc; opt: speed; options: -d:release
91693 lines; 5.132s; 180.199MiB peakmem; proj: /private/var/folders/lz/6ktvh__54hz3rjmrry69dbqc0000gn/T/nimble_15599/githubcom_stisajupyternim/src/jupyternim.nim; out: /var/folders/lz/6ktvh__54hz3rjmrry69dbqc0000gn/T/nimble_15599/githubcom_stisajupyternim/jupyternim [SuccessX]
    Prompt: jupyternim@0.8.0 already exists. Overwrite? [y/N]
    Answer: y
    Prompt: Missing directory /Users/myuser/.nimble/pkgs/jupyternim-0.8.0/src/jupyternimpkg. Continue? [y/N]
    Answer: y
    Prompt: Missing directory /Users/myuser/.nimble/pkgs/jupyternim-0.8.0/src/jupyternimspec. Continue? [y/N]
    Answer: y
    Prompt: Missing directory /Users/myuser/.nimble/pkgs/jupyternim-0.7.0/src/jupyternimpkg. Continue? [y/N]
    Answer: y
    Prompt: Missing directory /Users/myuser/.nimble/pkgs/jupyternim-0.7.0/src/jupyternimspec. Continue? [y/N]
    Answer: y
   Creating symlink: /Users/myuser/.nimble/pkgs/jupyternim-0.7.0/jupyternim -> /Users/myuser/.nimble/bin/jupyternim
    Copying file /var/folders/lz/6ktvh__54hz3rjmrry69dbqc0000gn/T/nimble_15599/githubcom_stisajupyternim/src/jupyternimpkg/sockets.nim to /Users/myuser/.nimble/pkgs/jupyternim-0.8.0/jupyternimpkg/sockets.nim
    Copying file /var/folders/lz/6ktvh__54hz3rjmrry69dbqc0000gn/T/nimble_15599/githubcom_stisajupyternim/src/jupyternimpkg/initialCodeCell.nim to /Users/myuser/.nimble/pkgs/jupyternim-0.8.0/jupyternimpkg/initialCodeCell.nim
    Copying file /var/folders/lz/6ktvh__54hz3rjmrry69dbqc0000gn/T/nimble_15599/githubcom_stisajupyternim/src/jupyternimpkg/utils.nim to /Users/myuser/.nimble/pkgs/jupyternim-0.8.0/jupyternimpkg/utils.nim
    Copying file /var/folders/lz/6ktvh__54hz3rjmrry69dbqc0000gn/T/nimble_15599/githubcom_stisajupyternim/src/jupyternimpkg/messages.nim to /Users/myuser/.nimble/pkgs/jupyternim-0.8.0/jupyternimpkg/messages.nim
    Copying file /var/folders/lz/6ktvh__54hz3rjmrry69dbqc0000gn/T/nimble_15599/githubcom_stisajupyternim/src/jupyternimpkg/codeserver.nim to /Users/myuser/.nimble/pkgs/jupyternim-0.8.0/jupyternimpkg/codeserver.nim
    Copying file /var/folders/lz/6ktvh__54hz3rjmrry69dbqc0000gn/T/nimble_15599/githubcom_stisajupyternim/src/jupyternimpkg/display.nim to /Users/myuser/.nimble/pkgs/jupyternim-0.8.0/jupyternimpkg/display.nim
    Copying file /var/folders/lz/6ktvh__54hz3rjmrry69dbqc0000gn/T/nimble_15599/githubcom_stisajupyternim/src/jupyternimspec/kernel.json to /Users/myuser/.nimble/pkgs/jupyternim-0.8.0/jupyternimspec/kernel.json
    Copying file /var/folders/lz/6ktvh__54hz3rjmrry69dbqc0000gn/T/nimble_15599/githubcom_stisajupyternim/src/jupyternimspec/kernel.js to /Users/myuser/.nimble/pkgs/jupyternim-0.8.0/jupyternimspec/kernel.js
    Copying file /var/folders/lz/6ktvh__54hz3rjmrry69dbqc0000gn/T/nimble_15599/githubcom_stisajupyternim/src/jupyternimspec/logo-64x64.png to /Users/myuser/.nimble/pkgs/jupyternim-0.8.0/jupyternimspec/logo-64x64.png
    Copying file /var/folders/lz/6ktvh__54hz3rjmrry69dbqc0000gn/T/nimble_15599/githubcom_stisajupyternim/src/jupyternimspec/logo-32x32.png to /Users/myuser/.nimble/pkgs/jupyternim-0.8.0/jupyternimspec/logo-32x32.png
    Copying file /var/folders/lz/6ktvh__54hz3rjmrry69dbqc0000gn/T/nimble_15599/githubcom_stisajupyternim/jupyternim.nimble to /Users/myuser/.nimble/pkgs/jupyternim-0.8.0/jupyternim.nimble
    Copying file /var/folders/lz/6ktvh__54hz3rjmrry69dbqc0000gn/T/nimble_15599/githubcom_stisajupyternim/jupyternim to /Users/myuser/.nimble/pkgs/jupyternim-0.8.0/jupyternim
   Creating symlink: /Users/myuser/.nimble/pkgs/jupyternim-0.8.0/jupyternim -> /Users/myuser/.nimble/bin/jupyternim
   Success: jupyternim installed successfully.
 Attempting to execute hook installAfter in /Users/myuser/.nimble/pkgs/jupyternim-0.8.0/jupyternim.nimble

To install Jupyter Nim kernel, run:

  jupyternim

Running jupyternim fails with a command not found error.

[stisa]

That seems like it installs fine but not in the correct folder (might be a nimble issue?). Can you check if you can find jupyternim in your nimble pkgs folder? If it's there, running the jupyternim executable should register the kernel for you, but you'll have to manually add it to path.

You can also try the manual install https://github.com/stisa/jupyternim#long-version if nimble failed.

This part is interesting:

Prompt: jupyternim@0.8.0 already exists. Overwrite? [y/N]
    Answer: y
    Prompt: Missing directory /Users/myuser/.nimble/pkgs/jupyternim-0.8.0/src/jupyternimpkg. Continue? [y/N]
    Answer: y
    Prompt: Missing directory /Users/myuser/.nimble/pkgs/jupyternim-0.8.0/src/jupyternimspec. Continue? [y/N]
    Answer: y
    Prompt: Missing directory /Users/myuser/.nimble/pkgs/jupyternim-0.7.0/src/jupyternimpkg. Continue? [y/N]
    Answer: y
    Prompt: Missing directory /Users/myuser/.nimble/pkgs/jupyternim-0.7.0/src/jupyternimspec. Continue? [y/N]
    Answer: y

not sure why nimble can't find the 0.8 folder when it says it's installed, and no idea why it's trying to remove 0.7 too.

[Vindaar]

For what it's worth I just tried jupyternim for the first time on my laptop running Void Linux. Installation here succeeded correctly and the binary is in ~/.nimble/bin.

Tomorrow evening or Saturday I'll test it on Windows and will report back.

[AngelEzquerra]

I can find a jupyternim-0.8.0 (and a 0.7.0) folder in the .nimble/pkgs folder. If I run jupyternim from the 0.8.0 folder I get the following error:

could not load: libzmq.dylib
(compile with -d:nimDebugDlOpen for more information)

[AngelEzquerra]

BTW, nim-zmq is installed, and zmq is also installed (via brew).

[AngelEzquerra]

I downloaded the jupyternim repo into my M1 Mac and I compiled it using the -d:nimDebugDlOpen option (i.e. by doing nimble build -d:nimDebugDlOpen). Then I set DYLD_LIBRARY_PATH=/opt/homebrew/lib and run the jupyternim executable that I built (i.e. by doing DYLD_LIBRARY_PATH=/opt/homebrew/lib ./jupyternim). The result was the following error:

dlopen(libzmq.dylib, 0x0002): tried: '/opt/homebrew/lib/libzmq.dylib' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), 'libzmq.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibzmq.dylib' (no such file), '/usr/lib/libzmq.dylib' (no such file, not in dyld cache), 'libzmq.dylib' (no such file), '/usr/local/lib/libzmq.dylib' (no such file), '/usr/lib/libzmq.dylib' (no such file, not in dyld cache)
could not load: libzmq.dylib

Apparently jupyternim (or nim-zmq?) tries use an x86_64 libzmq.dylib on an M1 Mac for some reason. If I do not set DYLD_LIBRARY_PATH the error is simply that libzmq.dylib is not found (i.e. jupyternim does not look for libzmq.dylib on the homebrew lib path.

[stisa]

Thanks again for the report.
Which version of zmq do you have? I believe nim-zmq is for v4.2.x. Might be a conflict there? I assume the normal python kernel works fine, right?

That error you report sounds like nim finds libzmq.dylib in '/opt/homebrew/lib' but it finds an arm64 while trying to compile for x86. You could try to pass --cpu:arm64 to make sure nim is compiling for the right target architecture.
Unfortunately I don't have access to a Mac, but I'll give this a try on an arm64 pc I have around.

PS sorry for the slowness, with the holidays coming up I don't have much free time

[stisa]

Also, what does running file on the jupyternim executable return? (file jupyernim) I remember some issues with choosenim having spotty support for arm, causing binaries to be for the wrong arch, so if that wasn't fixed and that's how you installed nim that could also be an issue

[AngelEzquerra]

Yes, the regular Jupyter lab works fine with a Python kernel.

I do now now which zmq version I have installed. I am not sure how to tell. I did have nim-zmq 1.2.1 installed. I upgraded to the latest version (1.3.1) but I have the same problem.

Passing --cpu:arm64 did not help.

The command file jupyternim returns:

./jupyternim: Mach-O 64-bit executable x86_64
which seems a little odd?

[stisa]

The command file jupyternim returns:

./jupyternim: Mach-O 64-bit executable x86_64

Interesting, what about nim? Running nim -v (or file nim I guess) will show you which architecture your nim is built for. I think that if you are running the amd64 version of nim instead of the arm64, it will default to building amd64 programs. Which would explain why your jupyternim is an x64 executable.

Not sure why nimble build --cpu:arm64 would build an x64 executable tough

To check your zmq version I guess you could check it with brew?

[AngelEzquerra]

nim --version returns:

Nim Compiler Version 1.6.10 [MacOSX: amd64]
Compiled at 2022-11-23
Copyright (c) 2006-2021 by Andreas Rumpf

active boot switches: -d:release

file nim returns:

nim: Mach-O 64-bit executable x86_64

This is a bit odd and might be related to the output I get when trying to run jupyternim compiled with the -d:nimDebugDlOpen flag and setting DYLD_LIBRARY_PATH=/opt/homebrew/lib:

DYLD_LIBRARY_PATH=/opt/homebrew/lib ./jupyternim
dlopen(libzmq.dylib, 0x0002): tried: '/opt/homebrew/lib/libzmq.dylib' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), 'libzmq.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibzmq.dylib' (no such file), '/usr/lib/libzmq.dylib' (no such file, not in dyld cache), 'libzmq.dylib' (no such file), '/usr/local/lib/libzmq.dylib' (no such file), '/usr/lib/libzmq.dylib' (no such file, not in dyld cache)
could not load: libzmq.dylib

So it seems there are 2 problems:

1. jupyternim (or nimzmq?) does not look for the zmq library in the homebrew folder (and I think it should since it is so common to use brew as a package manager)
2. For some reason it expects that library to be x86_64 while it should use arm64

[stisa]

Nim Compiler Version 1.6.10 [MacOSX: amd64]

So for 2. it looks like your nim is compiled for x86_64, my best guess is this is causing it to compile everything into x86_64 binaries, so it looks for the x64 version of zmq (seems like other people are having similar problems e.g. nim-lang/Nim#19376 (comment) nim-lang/csources_v1#8 (comment) ). No idea why compiling with --cpu:arm64 still results in x86_64 binaries though.
It might be worth opening an issue in the nim repo to see if they can provide a Mac arm64 build instead of just x86_64.
Unfortunately I don't know enough about how nim builds mac binaries to help, I think running the following directly will build an arm64 version:

git clone https://github.com/nim-lang/Nim  
cd Nim  
sh build_all.sh  

If then nim -v reports arm64 it should work fine with the arm64 zmq from brew.

For 1., agreed it should check that path too, I'll add it to my todo list