Skip to content

Update distributing components section with docs on wkg #220

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Apr 7, 2025
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion component-model/src/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
- [Authoring Components](./creating-and-consuming/authoring.md)
- [Composing Components](./creating-and-consuming/composing.md)
- [Running Components](./creating-and-consuming/running.md)
- [Distributing Components](./creating-and-consuming/distributing.md)
- [Distributing and Fetching Components and WIT](./creating-and-consuming/distributing.md)
- [Tutorial](./tutorial.md)

# Runtime Support
Expand Down
156 changes: 153 additions & 3 deletions component-model/src/creating-and-consuming/distributing.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,157 @@
# Distributing Components
# Distributing and Fetching Components and WIT

Modern applications rely extensively on third-party packages - so extensively that distributing packages is almost an industry in itself. Traditionally, these have been specific to a language. For example, JavaScript developers are used to using packages from NPM, and Rust developers use `crates.io`. Some runtimes support binary distribution and linking, enabling limited cross-language interop; for example, Maven packages can be written in any language that targets the Java runtime. Services like this are variously referred to as "package managers" or "registries."

Publishing and distribution are not defined by the core component model, but will form an important part of the component ecosystem. For example, if you're writing JavaScript, and want to pull in a highly optimised machine learning algorithm written in C and compiled to Wasm, you should be able to invoke it from a registry, just as easily as you would add a NPM package from the NPM registry.
Publishing and distribution are not defined by the core component model, but form important part of the component ecosystem. For example, if you're writing JavaScript, and want to pull in a highly optimised machine learning algorithm written in C and compiled to Wasm, you can pull it from a registry, ideally just as easily as you would add a NPM package from the NPM registry.

Publishing and distribution is a work in progress. The proposed registry protocol is [warg](https://warg.io/), but this is still in development, and there are no public warg registries as yet. You can find more information about the development of the registry protocol [here](https://github.com/bytecodealliance/governance/blob/main/SIGs/SIG-Registries/proposal.md).
You can get involved with improving the packaging and hosting of Wasm components by joining the [Bytecode Alliance Packaging Special Interest Group (SIG)](https://github.com/bytecodealliance/governance/blob/main/SIGs/sig-packaging/proposal.md).

## The `wkg` Registry Tool

The [`wasm-pkg-tools` project](https://github.com/bytecodealliance/wasm-pkg-tools) enables fetching and publishing Wasm components to OCI registries. It contains a `wkg` CLI tool that eases distributing and fetching components and WIT packages. `wkg` contains several subcommand:

- `wkg oci` - is a CLI wrapper around the [oci-wasm](https://github.com/bytecodealliance/rust-oci-wasm) crate which enables pushing/pulling Wasm artifacts to/from any OCI registry
- `wkg publish` - pushes *library* components or WIT packages
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not quite sure what the emphasis on "library" is aiming at. It implies that there are certain components I can't publish but it's not quite clear to me what the distinction is.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to emphasize that wkg publish is used for publishing components that are meant to be consumed rather than executed, but i guess you could publish a component that implements wasi:cli/run.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comparing to wkg oci this invites the question "pushes to where" but honestly I am not sure there is a good answer that fits in a single line! The distinction is kind of "as a CM package, addressable by package name" vs "as an OCI thing, addressable by physical location" but I'm not sure how to wrap that up. Maybe we should have another para between the intro para and the list along the lines of:

The usual way of using wkg is to address packages by their package name - for example, example:[email protected]. When using wkg this way, you don't need to know about the physical location of the package - wkg configuration handles that for you. If you need to, though, you can also use wkg to work with OCI artifacts directly, addressing them by OCI references.

Then this line could refer back by saying e.g. "publish components or WIT packages by package name" or something.

(I'm not wedded to any particular structure or phrasing - just wanting to provide some context for this command.)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good distinction of "by package name". I incorporated your suggestions

- `wkg get` - pulls *library* components or WIT packages
- `wkg wit` - commands for interacting with WIT files and dependencies
- `wkg config` - interact with the `wkg` configuration

The following sections detail a subset of actions that can be performed with `wkg`.

## `wkg` Configuration Files

The `wkg` tool uses a configuration file to understand where to publish and fetch specific packages to and from. It provides the ability to configure:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe provide some context along the lines of "When you use wkg, you don't interact with physical locations, only with package names." (And then hopefully the config file can be understood in terms of "mapping package naming to physical location" kind of thing.)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added this line about mapping package name to location. I like it!


- a default registry for all packages at the top level of the file
- a default registry for all packages of a specific namespace under `[namespace_registries]`. This section can be used to configure the registry of all `wasi` namespaced packages, such as `wasi:clocks` and `wasi:http`.
- an override for package of a specific namespace under `[package_registry_overrides]`. This section can be used to fetch/publish a specific package of a namespace from/to a different location than all other packages of that namespace. For example, maybe you want to fetch `wasi:http` from a different registry.
- credentials for a registry under `[registry."<registry-name>".oci]`
- and more! See the [`wkg` docs for more configuration options](https://github.com/bytecodealliance/wasm-pkg-tools?tab=readme-ov-file#configuration).

For example, to fetch WASI packages, such as `wasi:clocks` and `wasi:http`, a line can be added under the `namespace_registries` section for the `wasi` namespace. Specifically, the example below configures `wkg` to fetch WASI packages from the [WebAssembly OCI GitHub Container Registry](https://github.com/orgs/WebAssembly/packages), where the latest interfaces are published upon WASI releases. To edit your `wkg` config file, simply run `wkg config --edit`.

> Remember, all package names consist of the a namespace field followed by the package field. The package name `wasi:clocks` has a namespace of `wasi` and package field of `clocks`. In this way, the following configuration ensures that `wkg` will know to route fetches and publishes of any `wasi:x` to the configured location.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is super helpful and I wonder if we should briefly mention this before we get into wkg at all. But in any event good to have here!


```toml
# $XDG_CONFIG_HOME/wasm-pkg/config.toml
default_registry = "ghcr.io"

[namespace_registries]
# Instruct wkg to use the OCI protocol to fetch packages with the `wasi` namespace from ghcr.io/webassembly
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Instruct wkg to use the OCI protocol to fetch packages with the `wasi` namespace from ghcr.io/webassembly
# Tell wkg that packages with the `wasi` namespace are in an OCI registry under ghcr.io/webassembly

(the reasoning here is to align the order of the sentence with the order of the entries in the config line, so as to help the reader map the two)

wasi = { registry = "wasi", metadata = { preferredProtocol = "oci", "oci" = {registry = "ghcr.io", namespacePrefix = "webassembly/" } } }
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can copy this but I'm not sure I understand it. My guess is that this is talking about "packages with the wasi prefix" and saying "they should be got from the wasi registry and then... saying where the wasi registry is? It might be good to explain the config file at a high level (with a link for details) alongside the recommendation?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am still not sure I get what registry = "wasi" is doing here but I'm going to stop worrying about it - if I care that deeply I can read the wkg documentation! The rest of it clicked much more readily for me - thank you!

```

As a more generic example, The following configuration, instructs `wkg` to use [ttl.sh](https://ttl.sh/) OCI registry for all packages with the `docs` namespace.

```toml
# $XDG_CONFIG_HOME/wasm-pkg/config.toml
default_registry = "ghcr.io"

[namespace_registries]
# Instruct wkg to use the OCI protocol to fetch packages with the `foo` namespace from ttl.sh/wasm-components
docs = { registry = "docs-registry", metadata = { preferredProtocol = "oci", "oci" = {registry = "ttl.sh", namespacePrefix = "wasm-components/" } } }
```

> Note: the registry name can be referenced in the `package_registry_overrides` section of the `wkg` config to provide overrides for specific packages of a namespace.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AHA NOW I GET IT! Thank you!



## Distributing WIT and Library Components using `wkg publish`

Once you've [configured `wkg`](#wkg-configuration-files) to know where to publish packages to, you can use the `wkg publish` command to publish library *components* or *interfaces* to be consumed by others.

Imagine you have defined the following `adder` world in WIT:

```wit
package docs:[email protected];

interface add {
add: func(a: u32, b: u32) -> u32;
}

world adder {
export add;
}
```

You can publish this *WIT* using `wkg` by wrapping it up as a Wasm component. Yes, you heard that right! We are packaging WIT as Wasm.

```sh
# Package the contents of add WIT directory as Wasm
wkg wit build --wit-dir tutorial/wit/adder
# Publish the produced component
wkg publish docs:[email protected]
```

If you had configured `wkg` as described in the [`wkg` configuration section](#wkg-configuration-files), this would publish the component to `ttl.sh/wasm-components/docs/adder:0.1.0`. This WIT can then be fetched using `wkg get`, specifying the format `wit`:

```sh
wkg get --format wit docs:[email protected]
```

Instead of publishing the WIT interface, you could publish the built component by running:

```sh
wkg publish adder.wasm --package docs:[email protected]
```

This component can then be fetched by running:

```sh
wkg get docs:[email protected]
```

## More Generic Operations with `wkg oci`

The `wkg oci` subcommand is a CLI wrapper around the [oci-wasm](https://github.com/bytecodealliance/rust-oci-wasm) crate which enables pushing/pulling Wasm artifacts to/from any OCI registry. Unlike `wkg publish` and `wkg get` it is not limited to library components, as providing the WIT package is not required.

A component is pushed to an OCI registry using `wkg oci pull`. The example below pulls a component from a GitHub Container Registry.

```sh
wkg oci push ghcr.io/user/component:0.1.0 component.wasm
```

To pull a component, run:

```sh
wkg oci pull ghcr.io/user/component:0.1.0 -o component.wasm
```

## Fetching WIT Package Dependencies using `wkg`

Sometimes fetching a single package is not sufficient because it depends on other packages. For example, the following world describes a simple Wasm service which requires `wasi:http/proxy`:

```wit
package foo:wasi-http-service;

world target-world {
include wasi:http/[email protected];
}
```

You may be tempted to simply get the `wasi:http` package with `wkg get --format wit wasi:[email protected] -o wit/deps/http/`. However, `wasi:http` depends on other WASI packages such as `wasi:clocks` and `wasi:io`. To make sure to fetch a package and all its dependencies, use `wkg wit fetch`, which will read the package containing the world(s) you have defined in the given wit directory (`wit` by default). It will then fetch the
dependencies and write them to the `deps` directory along with a lock file.

After placing the above file in `./wit`, run the following to fetch the dependencies:

```sh
wkg wit fetch
```

The `./wit` directory will be populated as follows:
```sh
wit
├── deps
│ ├── wasi-cli-0.2.3
│ │ └── package.wit
│ ├── wasi-clocks-0.2.3
│ │ └── package.wit
│ ├── wasi-http-0.2.3
│ │ └── package.wit
│ ├── wasi-io-0.2.3
│ │ └── package.wit
│ └── wasi-random-0.2.3
│ └── package.wit
└── world.wit
```

Now, you can use the language toolchain of your choice to generate bindings and create your component.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The opening material talked about distributing and fetching components but most of the focus seems to be on WITs. Can we give an example of consuming a library (implementation) package from a registry?

Perhaps it would be useful to provide a bit of conceptual framework around "okay things in registries may be interface or implementation and it's all represented as Wasm and it gets distributed and fetched the same way you just need to say if you want the interface or implementation when you pull/get/fetch it" or whatever is, uh, true.