Project Asman: Public Compute Queues for everyone everywhere all at once

• client: https://container.mtfm.io/
• basic demo

Open-source publicly available compute queues: run queues of containerized work on your laptop, workstations, compute clusters.

• Every compute job is represented as a unique URL that contains all the context required to run it.
• Every queue is a unique URL. Queues are unguessable, instantly created, both durable and disposable.

Background

A common problem in sharing scientific workflows is that compute isn’t portable. Workflow engines, such as Nextflow, or code environments such Jupyter Notebooks have the compute tightly connected to the overall environment.

This means you can’t just simply share code and data and expect others to run it easily—differences in environments, libraries, and hardware often get in the way, and most environments are highly complex to set up. These systems were not built from the beginning to be web-first shareable, and require an all-or-nothing approach to using those systems.

Existing systems for work queues exist, but they are either built for different specific purposes, such as CI/CD pipelines (github actions and google cloud build), or they are internal and language specific.

Project Asman solves this by providing a web-based queue API. A lightweight web or CLI client submits Docker jobs to a queue. If the queue does not exist, it is immediately created. Any connected worker—yours, your institution’s, or a collaborator’s—can pick up the job, run it, and return the results. Supporting small teams to collaborate was a driving force for this project.

The API is efficient, open-source, and built for flexibility: workers can run locally, on a cluster, or be dynamically scaled via cloud providers. Docker images can be pulled directly or built from a Git repo.

This repo includes:

• The queue and worker infrastructure
• Example cloud deployments
• Support for metapage-style workflows using containerized metaframes

Online docs: Online docs notion source

Quick setup / getting started

1) Run a worker (optional)

👉 You can skip this step if you use the queue public1 where we supply a small amount of available compute.

❗ Replace my-unique-queue-name with a unique id or uuid or queue name, it can be anything

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /tmp:/tmp metapage/metaframe-docker-worker:0.54.30 run --max-job-duration=20m --data-directory=/tmp/worker-metapage-io-remote --cpus=2 my-unique-queue-name

2) Run a compute job

Go to the web client configured with a simple job.

• Click on the queue button at the bottom right. Enter your queue name (same as the worker queue)
• Click Run Job and the web client will submit the job to the queue, which will be picked up by the

The URL hash content uniquely defines the container job definition.

Developers

The API runs as cloudflare or deno workers: highly efficient and cost-effective server endpoints that simply record the current job queue.

Blob storage is via an S3 compatible API. Important: in the public version, all jobs are deleted after a week. This keeps costs extremely low, and allows us to provide public queues at low cost.

Developer background

This service provides docker compute functions as metaframes for the metapage.io platform. This allows sharing complex containerized workflows simply via URLs.

This service serves a web cliebt as an iframe (metaframe), that allows users to configure running a specific docker container (a job) on a specific queue. The iframed browser window sends that job configuration to the server, the job is added to the queue, then workers pick up the job, run it, and send the results back.

To run those docker containers, users can either rent compute from the metapage platform, or run worker(s) themselves, either on their own personal laptops/desktops, or on their own cluster infrastructure. Docker images can be used directly, or a git repo can be given, and the docker image built directly.

This repo contains all the infrastructure for the queues, workers, and examples of cloud providers managing the horizintal scaling worker fleets.

High level commands:

• develop: just dev
• bump or set a new version and publish artifacts: just deploy
• test the entire project: just test

Finer commands are just in subdirectories.

Overview

Quick links:

• production api: https://container.mtfm.io/

flowchart LR
    classDef greenOutline fill:#fff,stroke:##20a100,stroke-width:2px;
    classDef whalestuff fill:#32AEE4,stroke:#32AEE4,stroke-width:2px;
    subgraph workers [kubernetes scaling group]
        w2("worker 2"):::whalestuff
        w3("worker 3"):::whalestuff
    end

    subgraph u1 [User laptop]
        b1("browser client 1"):::greenOutline
        w1("worker 1"):::whalestuff
    end

    subgraph api [public api]
        q1["queue 1"]
        q2["queue 2"]
    end
    
    s3[(s3)]

    b1 --> |put blobs| s3
    b1 --> |put job| q1
    q1 --> |take job| w1
    q1 --> |take job| w2
    q2 --> |take job| w3
  
    w1 --- |get/put blobs| s3
	  w2 --- |get/put blobs| s3
    w3 --- |get/put blobs| s3

Loading

Local development

Host requirements:

• just: https://just.systems/man/en/chapter_1.html
• docker: https://docs.docker.com/engine/install/
• deno: https://docs.deno.com/runtime/manual/getting_started/installation
• mkcert: https://github.com/FiloSottile/mkcert
  • ❗ 👉 Make sure you run mkcert -install ❗

Run the local stack:

just dev

Go to this Test metapage to interact with a running simulation.

You might need to wait a bit to refresh the browser, it incorrectly returns a 200 when there are no browser assets (yet). (This looks like a bug with the hono webserver).

You can edit browser code, worker code, api code, and CLI and everything automatically updates.

Tests

• just test: runs the entire test suite, creating a new local stack
  • runs on every push to non-main branches
• just app/test/watch: (requires a running local stack) runs functional tests, currently only permissions
  • see just api/test for more test related commands

Start each service separately

You can develop workers locally, pointing to prod or local API

  just app/worker/dev
  # or
  just worker dev

  just app/worker/prod
  # or
  just worker prod

You can develop the browser locally, pointing to prod or local API

  just app/browser/dev
  # or
  just browser dev

  just app/browser/prod
  # or
  just browser prod

Submit jobs via the CLI

To the local stack:

cd app/cli
deno run --unsafely-ignore-certificate-errors --location https://worker-metaframe.localhost --allow-all src/cli.ts job add local1 --file ../../README.md -c 'sh -c "cat /inputs/README.md > /outputs/readme-copied.md"' --wait

To the production stack:

cd app/cli
deno run --allow-all src/cli.ts job add public1 --file ../../README.md -c 'sh -c "cat /inputs/README.md > /outputs/readme-copied.md"' --wait

The CLI tool has yet to be versioned and binaries built #21

Local development: cloud compute providers

E.g. kubernetes, nomad.

1. Run the local stack with just dev
2. Workers in the local worker cluster need to be able to reach (on the host): https://worker-metaframe.localhost
   1. Point the workers to a queue
3. Go to this metapage
   1. You should see the docker runner at the bottom, change the slider to create compute jobs

Deployment (automation)

(public) api:

• push to main:
  • The api server is deployed to deno.deploy
    • The browser is built as part of the api

worker:

• git semver tag:
  • the worker image is built, published
    • TODO: #2
  • worker host requirements:
    • just docker
  • our cloud worker providers update the worker version and redeploy
    • TODO: #3

💖🙏🤝 Acknowledgements

• 2025: support for this work was funded by Astera to which I am grateful. Due to this support, I aim to keep this project open and supported for as long as it is useful.
• 2018: the seed for this work was a previous project (https://github.com/dionjwa/docker-cloud-compute) supported by Autodesk Life Sciences (which was an amazing ambitious reach of innovation)