Compiling and running scryer as a WebAssembly binary?

[rla]

Hello @mthom! Would it be possible to compile and run scryer as a WebAssembly binary? I worked some time ago on SWI-Prolog WebAssembly port but it turned out to be rather difficult due to:

• Clib/POSIX platform expectations (like getting stuck with the Emscripten's broken readdir and other functions);
• Fighting with the build system (it requires intermediate binary to build itselt);
• Awful difficulties with cross-compiling C as usual.

Also, it was quite inconvenient to provide API for JavaScript. It takes hundreds of lines of handwritten code to bind JS functions to low-level FLI, a SWI-Prolog's FFI.

More info: SWI-Prolog/roadmap#43

It seems like Rust has some support for WebAssembly (https://developer.mozilla.org/en-US/docs/WebAssembly/Rust_to_wasm).

What do you think:

• Would it be possible to build scryer as a WebAssembly binary?
• Could scryer be decoupled from filesystem/POSIX/threading/signals requirements and run without filesystem, clib or any syscalls at all?
• Would it be possible to use wasm-bindgen to generate useful API? (It could be something quick-and-dirty like terms serialized as JSON strings at first)

[cduret]

I have compiled scryer without signal (nix package) under windows OS

[tniessen]

Would it be possible to build scryer as a WebAssembly binary?

I did that one or two months ago, based on c342d18. It seems possible, but it certainly isn't trivial.

Compilation problems

1. The rug dependency doesn't work with WebAssembly. I exchanged a few emails with @tspiteri, and it seemed challenging to cross-compile GMP. Workaround: Remove the rug dependency and change the default features to ["num"], which uses num-rug-adapter.

2. I couldn't get the crossterm crate to work, which is not really a surprise given the lack of a terminal in WebAssembly, but there might be a solution that I missed. Workaround: Remove crossterm and adapt get_single_char in src/prolog/machine/system_calls.rs to use an imported function raw_mode_single_char instead (which needs to be implemented in JavaScript):

   extern "C" {
     pub fn raw_mode_single_char() -> u32;
   }
   
   pub fn get_single_char() -> char {
     unsafe {
       return char::from_u32(raw_mode_single_char()).unwrap();
     }
   }

3. ProcessTime won't work in WebAssembly. Workaround: Replace or remove ProcessTime::now() in src/prolog/machine/system_calls.rs.

4. Signal handling doesn't work in WebAssembly. Workaround: Remove signal handling logic in src/main.rs. (Probably not necessary if you are building the project as a library instead of an executable binary.)

Cross-compiling

I don't remember the exact command I used to compile it, but I used rustup's nightly toolchain for the wasm32-wasi target.

Syscalls in the browser

I implemented necessary syscalls to an extent that allowed to load a prolog source file from a virtual in-browser file system, and to see output written to the stdout file descriptor. Apart from that, I left most of the wasi syscalls unimplemented, but someone with more time could probably get most of them to work.

Result

I got this tiny "Hello world" to work eventually:

:- initialization(hello_world).

hello_world :-
    write('Hello, World!'), nl.

Partial screenshot of the web page (with some debugging info):

I didn't try much more after that, this had already taken me many hours and I mostly did it to gain some experience with Rust → WebAssembly.

[rla]

@tniessen, thank you! That's awesome. Despite these issues, it seems like porting scryer would be easier.

• SWI-Prolog wasm version also does not support GMP;
• Unless using threads, terminal input cannot work anyway since the blocking read call would also block the browser event loop including input events to read user input. SWI-Prolog wasm version has exactly the same issue.
• I see that all IO is collected to system_calls.rs module, this also simplifies things.

@tniessen, do you happen to still have your changes somewhere? It would be a nice starting point.

[tniessen]

Unless using threads, terminal input cannot work anyway since the blocking read call would also block the browser event loop including input events to read user input. SWI-Prolog wasm version has exactly the same issue.

@rla It might be possible to avoid this in some browsers with only minimal modifications in scryer-prolog. I wrote synchronous-channel a while back to allow WebAssembly in worker threads to retrieve information (such as terminal input) from the main thread. I did not test the library in browsers, but it should work. So the idea would be for the main thread to write terminal input to a SynchronousChannel, and a worker thread running the WebAssembly code could read it from the SynchronousChannel. The library supports both blocking and non-blocking read and write operations.

do you happen to still have your changes somewhere? It would be a nice starting point.

I'll try to find and commit them, but it might take a few days.

[jacobfriedman]

Very interested in a wasm module with TCP/UDP bindings. I'm trying to build the SWI-Prolog WASM just to see where I can find those calls... any idea if I'll be able to sniff those calls out with a nodeJS WASI wrapper?

[jacobfriedman]

@tniessen when you're ready to dedicate some time to get a WASM going, I'm right here with @rla . I'd like to get an environment up... 'one to bind them all' with prolog - scryer's the most lean implementation I've seen. Please let me know if you're willing to work together :)

[tniessen]

Sorry @jacobfriedman and @rla, I am super busy with university, OSS, and everything else right now, but it's still on my list. I'd love to collaborate on this, and I'll try to find time soon-ish, I hope.

[tniessen]

@jacobfriedman @rla I am trying to get it to work again but recent changes have only made it more difficult.

• crossterm doesn't work, this affects keyboard events, workaround in Compiling and running scryer as a WebAssembly binary? #615 (comment)
• hostname doesn't work (not important)
• rustyline doesn't work, this affects essentially everything related to user input
• ring doesn't work (but might with some hacks)
• openssl doesn't work (but might with some hacks)
• native-tls doesn't work
• sodiumoxide doesn't work (but might with some hacks)
• slice-dequeue doesn't work

I am not sure why there are so many different crypto libraries in the list of dependencies.

TCP (the sockets library) won't work in WebAssembly, assuming it's supposed to run in a browser.

[jacobfriedman]

Well, a red flag for me:
"I am not sure why there are so many different crypto libraries in the list of dependencies."
@mthom is there a workaround?

What I've been seeing is that many libraries just wrap C instead of implementing in the native language. I wonder if these dependencies can be replaced with native (rust) drop-ins.

Are you using emscripten to compile, or compiling straight to WASM?

[triska]

Thank you a lot for resuming work on this project!

The cryptographic crates are used to support various cryptographic predicates in library(crypto), and also TLS connections in library(sockets).

The big bouquet of crates is needed because none of the existing crates covers all the functionality that library(crypto) exposes. Some of these crates are more convenient to use than others, and all of them have unique functionality that only they provide, at least in this simplicity.

If you do not need this functionality, you can, in your WASM branch, simply remove all cryptographic predicates from system_calls.rs, starting from

scryer-prolog/src/machine/system_calls.rs

Line 4749 in 10e92ee

&SystemClauseType::CryptoRandomByte => {

up to and including

scryer-prolog/src/machine/system_calls.rs

Line 5197 in 10e92ee

&SystemClauseType::Curve25519ScalarMult => {

and then simply remove the inclusion of these crates. This could be useful to start with this project. If at all possible, it would be great to preserve the cryptographic functionality as far as practical also in the WASM port.

Please let me know if you have any questions about this.

[jacobfriedman]

Thank you for the help! @tniessen will you take the wheel on this, as I'm relocating this week? If so please fork/branch & post the URL here :)

[triska]

One additional remark: If you plan to get one of the cryptographic crates working, I suggest to focus on ring. This is the most useful and well designed of the cryptographic crates, and porting it would enable a lot of the cryptographic functionality, most notably secure hashes, encryption and authentication.

The other crates are used for more rarely used hashes, and low-level reasoning over elliptic curves. I plan to get rid of specifically the openssl dependency as soon as a native Rust crate provides the needed functionality for elliptic curves.

[infogulch]

simply remove all cryptographic predicates from system_calls.rs ...

and then simply remove the inclusion of these crates

I wonder if this would be a good use-case for a cfg flag (crypto perhaps?), enabled by default, that if disabled omits the crypto-related system calls and their dependencies. This way, if the project is being built for WASM the cfg flag could simply be disabled.