[RFC] Support more WASI proposals(beyond wasi-libc p2)

[lum1n0us]

The goal of raising this issue is to enhance support for more WASI capabilities.

wasm32-wasi-p2-module, or wasm32-wasi-t2, is a new concept highlighted in a slide in Wasm CG June 2024. The initial idea can be found in a discussion.

P2-module is expected to enhance core modules by integrating advanced features of the WASI-p2, such as networking and encryption capabilities, which are anticipated to be highly appealing to users.

While there hasn't been an official release yet, developers can create wasm-wasi-p2-modules using the latest versions of wasi-sdk(22 and 23). This is done by compiling with the options --target=wasm32-wasip2 -fuse-ld=lld. The output has a mixed version of imports, wasi_snapshost_preview1 and wasi:[email protected]:

Import[15]:
 - func[0] sig=2 <__imported_wasi_snapshot_preview1_args_get> <- wasi_snapshot_preview1.args_get
 - func[1] sig=2 <__imported_wasi_snapshot_preview1_args_sizes_get> <- wasi_snapshot_preview1.args_sizes_get
 - func[2] sig=3 <__imported_wasi_snapshot_preview1_fd_close> <- wasi_snapshot_preview1.fd_close
 - func[3] sig=2 <__imported_wasi_snapshot_preview1_fd_fdstat_get> <- wasi_snapshot_preview1.fd_fdstat_get
 - func[4] sig=4 <__imported_wasi_snapshot_preview1_fd_seek> <- wasi_snapshot_preview1.fd_seek
 - func[5] sig=5 <__imported_wasi_snapshot_preview1_fd_write> <- wasi_snapshot_preview1.fd_write
 - func[6] sig=6 <__imported_wasi_snapshot_preview1_proc_exit> <- wasi_snapshot_preview1.proc_exit
 - func[7] sig=6 <__wasm_import_poll_pollable_drop> <- wasi:io/[email protected].[resource-drop]pollable
 - func[8] sig=6 <__wasm_import_streams_input_stream_drop> <- wasi:io/[email protected].[resource-drop]input-stream
 - func[9] sig=6 <__wasm_import_streams_output_stream_drop> <- wasi:io/[email protected].[resource-drop]output-stream
 - func[10] sig=6 <__wasm_import_udp_udp_socket_drop> <- wasi:sockets/[email protected].[resource-drop]udp-socket
 - func[11] sig=6 <__wasm_import_udp_incoming_datagram_stream_drop> <- wasi:sockets/[email protected].[resource-drop]incoming-datagram-stream
 - func[12] sig=6 <__wasm_import_udp_outgoing_datagram_stream_drop> <- wasi:sockets/[email protected].[resource-drop]outgoing-datagram-stream
 - func[13] sig=6 <__wasm_import_tcp_tcp_socket_drop> <- wasi:sockets/[email protected].[resource-drop]tcp-socket
 - func[14] sig=3 <__wasi_preview1_adapter_close_badfd> <- wasi_snapshot_preview1.adapter_close_badfd

The imports are same with wasm32-wasi-p2.

When implementing, the original wasi library at core/iwas/libraries/libc-wasi won't be touched. The new stuff will be stored at core/iwasm/libraries/libc-wasi-p2-module. If later, it is found to be able to shared with wasm32-wasi-p2, might be renamed to libc-wasi-p2.

[yamt]

P2-module is expected to enhance core modules by integrating advanced features of the WASI-p1, such as networking and encryption capabilities, which are anticipated to be highly appealing to users.

do you mean WASI-p2?

The output has a mixed version of imports, wasi_snapshost_preview1 and wasi:[email protected]

wasi_snapshost_preview1 will eventually be replaced by wasip2 things, right?

[lum1n0us]

do you mean WASI-p2?

IIUC, if using --target=wasm32-wasip2, the output is a component. If using --target=wasm32-wasip2-module, the output is a core module. The difference is which linker is used, component-ld or ld.

wasi_snapshost_preview1 will eventually be replaced by wasip2 things, right?

I don't know. I am hoping somebody might be able to answer that.

[yamt]

do you mean WASI-p2?

IIUC, if using --target=wasm32-wasip2, the output is a component. If using --target=wasm32-wasip2-module, the output is a core module. The difference is which linker is used, component-ld or ld.

my question was about your sentence "by integrating advanced features of the WASI-p1".

[lum1n0us]

Yes. Advanced features of the WASI-p2.

[loganek]

Thanks for kicking off the discussion. From our perspective, at least for time being, we don't need a full P2 support (i.e. WASI + Component Model) and just having "T2" would be sufficient. From what I understand, both T2 and preview1 can co-exist as they're in completely different namespaces, so one can choose to build WAMR with both T2 and preview1 enabled, and core module compiled to either target will work (ofc mixing T2 and preview1 calls within the same module might be dangerous, so users should be careful with that). And I think that might be a reasonable plan to first focus on just P2 imports, and only after that provide a full component model support (if ever needed).

Regarding potential deprecation of preview1 in WAMR - at that stage we'd really need to keep it around; however, we'll invest in the adapter https://github.com/loganek/wasi-snapshot-preview2-to-preview1-adapter/ and as soon as it's stable, my team will no longer need preview1 support in WAMR - if there are no other teams relying on that, perhaps it's something we might choose to deprecate over time.

[lum1n0us]

And I think that might be a reasonable plan to first focus on just P2 imports, and only after that provide a full component model support (if ever needed).

Yes, that's correct. In my understanding, there's a stronger demand for new features in WASI compared to the need for a component model. Therefore, priority should be given to handling p2 imports.

[yamt]

The output has a mixed version of imports, wasi_snapshost_preview1 and wasi:[email protected]

wasi_snapshost_preview1 will eventually be replaced by wasip2 things, right?

in case it was not clear, i meant about the use of the adapter in wasi-libc wasip2 build.
i didn't mean the deprecation of wasip1.

[lum1n0us]

I'm not entirely certain if this is an actual issue or just a misunderstanding on my part.

There are roughly 25+ WASI proposals, yet only a handful are supported by wasi-libc, even in P2. This means that a developer cannot utilize the capabilities of the new proposals without directly using binders (often created by wit-bindgen) or libraries built on top of these binders. Compared to using established libraries, working with raw binders demands a deeper understanding of the intricacies involved in type conversion, memory management, and communication between the host and WebAssembly. This can be confusing and challenging.

Despite the fact that more than 75% of the new proposals focus on networking and cloud applications, the trends and practices established now could influence future proposals.

When we look at the Rust/WebAssembly ecosystem, programmers have a wealth of libraries at their disposal, with crates available for almost all WASI proposals on crates.io. On the other hand, C/WebAssembly programmers face a scarcity of libraries, making it difficult to find C libraries related to WASI proposals.

The goal of raising this issue is to enhance support for WASI capabilities. Initially, I believed that all that was needed was to back wasi-libc (P2). However, it now appears that there is also a need to develop C libraries related to WASI proposals that based on binders.

[AlixANNERAUD]

Any updates on the support for WASI P2 in WAMR ?