-
Notifications
You must be signed in to change notification settings - Fork 64
Update distributing components section with docs on wkg #220
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 2 commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
@@ -1,7 +1,157 @@ | ||||||
# Distributing Components | ||||||
# Distributing and Fetching Components and WIT | ||||||
|
||||||
Modern applications rely extensively on third-party packages - so extensively that distributing packages is almost an industry in itself. Traditionally, these have been specific to a language. For example, JavaScript developers are used to using packages from NPM, and Rust developers use `crates.io`. Some runtimes support binary distribution and linking, enabling limited cross-language interop; for example, Maven packages can be written in any language that targets the Java runtime. Services like this are variously referred to as "package managers" or "registries." | ||||||
|
||||||
Publishing and distribution are not defined by the core component model, but will form an important part of the component ecosystem. For example, if you're writing JavaScript, and want to pull in a highly optimised machine learning algorithm written in C and compiled to Wasm, you should be able to invoke it from a registry, just as easily as you would add a NPM package from the NPM registry. | ||||||
Publishing and distribution are not defined by the core component model, but form important part of the component ecosystem. For example, if you're writing JavaScript, and want to pull in a highly optimised machine learning algorithm written in C and compiled to Wasm, you can pull it from a registry, ideally just as easily as you would add a NPM package from the NPM registry. | ||||||
|
||||||
Publishing and distribution is a work in progress. The proposed registry protocol is [warg](https://warg.io/), but this is still in development, and there are no public warg registries as yet. You can find more information about the development of the registry protocol [here](https://github.com/bytecodealliance/governance/blob/main/SIGs/SIG-Registries/proposal.md). | ||||||
You can get involved with improving the packaging and hosting of Wasm components by joining the [Bytecode Alliance Packaging Special Interest Group (SIG)](https://github.com/bytecodealliance/governance/blob/main/SIGs/sig-packaging/proposal.md). | ||||||
|
||||||
## The `wkg` Registry Tool | ||||||
|
||||||
The [`wasm-pkg-tools` project](https://github.com/bytecodealliance/wasm-pkg-tools) enables fetching and publishing Wasm components to OCI registries. It contains a `wkg` CLI tool that eases distributing and fetching components and WIT packages. `wkg` contains several subcommand: | ||||||
|
||||||
- `wkg oci` - is a CLI wrapper around the [oci-wasm](https://github.com/bytecodealliance/rust-oci-wasm) crate which enables pushing/pulling Wasm artifacts to/from any OCI registry | ||||||
- `wkg publish` - pushes *library* components or WIT packages | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm not quite sure what the emphasis on "library" is aiming at. It implies that there are certain components I can't publish but it's not quite clear to me what the distinction is. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I wanted to emphasize that wkg publish is used for publishing components that are meant to be consumed rather than executed, but i guess you could publish a component that implements There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Comparing to
Then this line could refer back by saying e.g. "publish components or WIT packages by package name" or something. (I'm not wedded to any particular structure or phrasing - just wanting to provide some context for this command.) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is a good distinction of "by package name". I incorporated your suggestions |
||||||
- `wkg get` - pulls *library* components or WIT packages | ||||||
- `wkg wit` - commands for interacting with WIT files and dependencies | ||||||
- `wkg config` - interact with the `wkg` configuration | ||||||
|
||||||
The following sections detail a subset of actions that can be performed with `wkg`. | ||||||
|
||||||
## `wkg` Configuration Files | ||||||
|
||||||
The `wkg` tool uses a configuration file to understand where to publish and fetch specific packages to and from. It provides the ability to configure: | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe provide some context along the lines of "When you use wkg, you don't interact with physical locations, only with package names." (And then hopefully the config file can be understood in terms of "mapping package naming to physical location" kind of thing.) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Added this line about mapping package name to location. I like it! |
||||||
|
||||||
- a default registry for all packages at the top level of the file | ||||||
kate-goldenring marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
- a default registry for all packages of a specific namespace under `[namespace_registries]`. This section can be used to configure the registry of all `wasi` namespaced packages, such as `wasi:clocks` and `wasi:http`. | ||||||
- an override for package of a specific namespace under `[package_registry_overrides]`. This section can be used to fetch/publish a specific package of a namespace from/to a different location than all other packages of that namespace. For example, maybe you want to fetch `wasi:http` from a different registry. | ||||||
- credentials for a registry under `[registry."<registry-name>".oci]` | ||||||
- and more! See the [`wkg` docs for more configuration options](https://github.com/bytecodealliance/wasm-pkg-tools?tab=readme-ov-file#configuration). | ||||||
|
||||||
For example, to fetch WASI packages, such as `wasi:clocks` and `wasi:http`, a line can be added under the `namespace_registries` section for the `wasi` namespace. Specifically, the example below configures `wkg` to fetch WASI packages from the [WebAssembly OCI GitHub Container Registry](https://github.com/orgs/WebAssembly/packages), where the latest interfaces are published upon WASI releases. To edit your `wkg` config file, simply run `wkg config --edit`. | ||||||
kate-goldenring marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
> Remember, all package names consist of the a namespace field followed by the package field. The package name `wasi:clocks` has a namespace of `wasi` and package field of `clocks`. In this way, the following configuration ensures that `wkg` will know to route fetches and publishes of any `wasi:x` to the configured location. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is super helpful and I wonder if we should briefly mention this before we get into
kate-goldenring marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
```toml | ||||||
# $XDG_CONFIG_HOME/wasm-pkg/config.toml | ||||||
default_registry = "ghcr.io" | ||||||
|
||||||
[namespace_registries] | ||||||
# Instruct wkg to use the OCI protocol to fetch packages with the `wasi` namespace from ghcr.io/webassembly | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
(the reasoning here is to align the order of the sentence with the order of the entries in the config line, so as to help the reader map the two) |
||||||
wasi = { registry = "wasi", metadata = { preferredProtocol = "oci", "oci" = {registry = "ghcr.io", namespacePrefix = "webassembly/" } } } | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I can copy this but I'm not sure I understand it. My guess is that this is talking about "packages with the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I am still not sure I get what |
||||||
``` | ||||||
|
||||||
As a more generic example, The following configuration, instructs `wkg` to use [ttl.sh](https://ttl.sh/) OCI registry for all packages with the `docs` namespace. | ||||||
|
||||||
```toml | ||||||
# $XDG_CONFIG_HOME/wasm-pkg/config.toml | ||||||
default_registry = "ghcr.io" | ||||||
|
||||||
[namespace_registries] | ||||||
# Instruct wkg to use the OCI protocol to fetch packages with the `foo` namespace from ttl.sh/wasm-components | ||||||
docs = { registry = "docs-registry", metadata = { preferredProtocol = "oci", "oci" = {registry = "ttl.sh", namespacePrefix = "wasm-components/" } } } | ||||||
``` | ||||||
|
||||||
> Note: the registry name can be referenced in the `package_registry_overrides` section of the `wkg` config to provide overrides for specific packages of a namespace. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. AHA NOW I GET IT! Thank you! |
||||||
|
||||||
|
||||||
## Distributing WIT and Library Components using `wkg publish` | ||||||
|
||||||
Once you've [configured `wkg`](#wkg-configuration-files) to know where to publish packages to, you can use the `wkg publish` command to publish library *components* or *interfaces* to be consumed by others. | ||||||
|
||||||
Imagine you have defined the following `adder` world in WIT: | ||||||
|
||||||
```wit | ||||||
package docs:[email protected]; | ||||||
|
||||||
interface add { | ||||||
add: func(a: u32, b: u32) -> u32; | ||||||
} | ||||||
|
||||||
world adder { | ||||||
export add; | ||||||
} | ||||||
``` | ||||||
|
||||||
You can publish this *WIT* using `wkg` by wrapping it up as a Wasm component. Yes, you heard that right! We are packaging WIT as Wasm. | ||||||
|
||||||
```sh | ||||||
# Package the contents of add WIT directory as Wasm | ||||||
wkg wit build --wit-dir tutorial/wit/adder | ||||||
# Publish the produced component | ||||||
wkg publish docs:[email protected] | ||||||
``` | ||||||
|
||||||
If you had configured `wkg` as described in the [`wkg` configuration section](#wkg-configuration-files), this would publish the component to `ttl.sh/wasm-components/docs/adder:0.1.0`. This WIT can then be fetched using `wkg get`, specifying the format `wit`: | ||||||
|
||||||
```sh | ||||||
wkg get --format wit docs:[email protected] | ||||||
``` | ||||||
|
||||||
Instead of publishing the WIT interface, you could publish the built component by running: | ||||||
|
||||||
```sh | ||||||
wkg publish adder.wasm --package docs:[email protected] | ||||||
``` | ||||||
|
||||||
This component can then be fetched by running: | ||||||
kate-goldenring marked this conversation as resolved.
Show resolved
Hide resolved
kate-goldenring marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
```sh | ||||||
wkg get docs:[email protected] | ||||||
``` | ||||||
|
||||||
## More Generic Operations with `wkg oci` | ||||||
|
||||||
The `wkg oci` subcommand is a CLI wrapper around the [oci-wasm](https://github.com/bytecodealliance/rust-oci-wasm) crate which enables pushing/pulling Wasm artifacts to/from any OCI registry. Unlike `wkg publish` and `wkg get` it is not limited to library components, as providing the WIT package is not required. | ||||||
kate-goldenring marked this conversation as resolved.
Show resolved
Hide resolved
kate-goldenring marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
A component is pushed to an OCI registry using `wkg oci pull`. The example below pulls a component from a GitHub Container Registry. | ||||||
kate-goldenring marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
```sh | ||||||
wkg oci push ghcr.io/user/component:0.1.0 component.wasm | ||||||
``` | ||||||
|
||||||
To pull a component, run: | ||||||
|
||||||
```sh | ||||||
wkg oci pull ghcr.io/user/component:0.1.0 -o component.wasm | ||||||
``` | ||||||
|
||||||
## Fetching WIT Package Dependencies using `wkg` | ||||||
|
||||||
Sometimes fetching a single package is not sufficient because it depends on other packages. For example, the following world describes a simple Wasm service which requires `wasi:http/proxy`: | ||||||
|
||||||
```wit | ||||||
package foo:wasi-http-service; | ||||||
|
||||||
world target-world { | ||||||
include wasi:http/[email protected]; | ||||||
} | ||||||
``` | ||||||
|
||||||
You may be tempted to simply get the `wasi:http` package with `wkg get --format wit wasi:[email protected] -o wit/deps/http/`. However, `wasi:http` depends on other WASI packages such as `wasi:clocks` and `wasi:io`. To make sure to fetch a package and all its dependencies, use `wkg wit fetch`, which will read the package containing the world(s) you have defined in the given wit directory (`wit` by default). It will then fetch the | ||||||
dependencies and write them to the `deps` directory along with a lock file. | ||||||
|
||||||
After placing the above file in `./wit`, run the following to fetch the dependencies: | ||||||
|
||||||
```sh | ||||||
wkg wit fetch | ||||||
itowlson marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
``` | ||||||
|
||||||
The `./wit` directory will be populated as follows: | ||||||
```sh | ||||||
wit | ||||||
├── deps | ||||||
│ ├── wasi-cli-0.2.3 | ||||||
│ │ └── package.wit | ||||||
│ ├── wasi-clocks-0.2.3 | ||||||
│ │ └── package.wit | ||||||
│ ├── wasi-http-0.2.3 | ||||||
│ │ └── package.wit | ||||||
│ ├── wasi-io-0.2.3 | ||||||
│ │ └── package.wit | ||||||
│ └── wasi-random-0.2.3 | ||||||
│ └── package.wit | ||||||
└── world.wit | ||||||
``` | ||||||
|
||||||
Now, you can use the language toolchain of your choice to generate bindings and create your component. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The opening material talked about distributing and fetching components but most of the focus seems to be on WITs. Can we give an example of consuming a library (implementation) package from a registry? Perhaps it would be useful to provide a bit of conceptual framework around "okay things in registries may be interface or implementation and it's all represented as Wasm and it gets distributed and fetched the same way you just need to say if you want the interface or implementation when you pull/get/fetch it" or whatever is, uh, true. |
Uh oh!
There was an error while loading. Please reload this page.