IPD - Image Push and Deploy

ipd - Receive app images and switch execution to new container.

Concept

IPD is a small self-hosted deployment service for containerized applications. A CI pipeline (or a developer) uploads a Docker image and a deployment package over HTTP(S), and IPD runs the release on the target server — no SSH access, no agent on build machines and no Docker registry required.

The typical scenario:

CI builds a Docker image, saves it to a tar archive and packs it together with the deployment package (for example a docker-compose.yml and a control.sh script) into a single upload.
The pipeline POSTs the upload to /upload with an auth token and then polls /status/$project/$deployment until the deployment reports ok.
On the server IPD unpacks the upload, loads the image with docker load and hands control to the project's control.sh, which switches the running stack to the new version.

This fits best when you run a handful of services on your own hosts and do not want a full Kubernetes/registry-based delivery stack: the server may even be air-gapped from external registries, the only requirement is that the CI can reach IPD over HTTPS.

Deployment logic is intentionally kept out of IPD itself: each project provides its own control.sh with three actions (check, prepare, deploy), so a release can be anything docker-compose up can express — while IPD handles the delivery parts common to all projects: authenticated uploads, queueing, timeouts, idempotency, status reporting, recovery after restarts and monitoring hooks. A working example of control.sh and the upload command line lives in the example/ directory.

Deploy flow

Run control.sh check. Return 1 if success, any other means error.
If check was erroneous run control.sh prepare.
Run control.sh deploy. Return 1 if success, any other means error.

Standard way to deploy project: upload package and wait until its processed. On destination server project folder should be accessible to write for daemon user.

Queue semantics

A newer upload of the same project supersedes an awaiting one: the old deployment is discarded (its files are removed) and later reports changed status.
The queue is served LIFO by design: a newer package is a newer version, so it is considered more important than older ones.
A deployment of a project that is already deploying waits in the queue and is started as soon as the project is free; it is never silently dropped.

Endpoints

Destination path prefixed with API_ROOT.

Endpoint	Method	Description	Variables
/	GET	Welcome message
/list	GET	List projects
/upload	POST	Upload deployment package	project - project name image - image file package - package file
/info	GET	Project information
/stat	GET	IPD statistics
/status/$project	GET	Deployment status	$project - project name
/status/$project/$deployment	GET	Deployment status	$project - project name $deployment - deployment ID, returned in upload
/status/$project/$deployment	DELETE	Cancel a queued deployment	$project - project name $deployment - deployment ID, returned in upload
/health	GET	Liveness check, no auth required
/metrics	GET	Prometheus metrics, ADMIN role

Idempotent uploads

Pass an Idempotency-Key header on upload to protect against retries (after a network timeout, for example). Repeating an upload with the same key for the same project returns the uuid of the already accepted deployment instead of creating a new one:

curl -H "Authorization: $TOKEN" -H "Idempotency-Key: build-123" ...

Cancelling a deployment

A deployment waiting in the queue can be removed without uploading a new package (running deployments can not be cancelled):

curl -X DELETE -H "Authorization: $TOKEN" \
    https://$REMOTE_HOST/deploy/status/$PROJECT/$DEPLOYMENT

Returns ok on success, active with code 409 if the deployment is already running, no with code 404 if it does not exist.

Response formats

All responses are plain text unless stated otherwise.

/upload returns the deployment uuid on success (err with HTTP code on failure)
/status/... returns one of: await, active, ok, changed, cancelled, no or a numeric code of the failed deployment script
/list returns project names, one per line
/info/$project returns a JSON object, for example {"start": 1690000000, "start_count": 3, "processed": 123456, "version": "1.2.3", "state": 4}, or no for an unknown project
/stat returns a JSON object like {"projects": 2, "queue": 0, "uploads": 5}

HTTP error codes: 400 — bad request (invalid project name, missing or wrong files, failed storage); 401 — missing or invalid token; 403 — token has no required role; 404 — cancelled deployment does not exist; 409 — deployment is already running and can not be cancelled; 413 — upload exceeds MAX_UPLOAD_SIZE; 507 — not enough disk space on the upload directory.

Environment variables for control.sh

Name	Value
PATH	Inherits from parent process
DEPLOY	UUID of deployment
PROJECT	project name passed on upload

Installation

Requirements: Python 3.9 or newer, Docker and docker-compose on the host.

Add user
adduser ipd
Add user to docker group
addgroup ipd docker
Create upload folder
mkdir -p /srv/upload && chown ipd:ipd /srv/upload
Fill with required values and copy config
cp .env.example /etc/default/ipd
Install virtualenv (Debian distro and derivatives)
sudo apt install -y python3-venv
Setup virtualenv
sudo python3 -m venv /opt/ipd
Create user file list
touch /opt/ipd/users.txt && chown ipd:ipd /opt/ipd/users.txt
Activate virtualenv
source /opt/ipd/bin/activate
Setup
pip install ipd-1.1.0-py3-none-any.whl
Add the first ADMIN user (prints a token, shown once)
sudo -u ipd /opt/ipd/bin/ipd user add ADMIN admin -f /opt/ipd/users.txt
Copy config to systemd services ipd.service to /etc/systemd/system/ipd.service
Reload systemd config
systemctl daemon-reload
Install service
systemctl enable ipd.service
Start service
systemctl start ipd.service
Verify the service is running
systemctl status ipd.service and
curl http://127.0.0.1:9955/deploy/health returns ok
Add to nginx config

server {
...

    location /deploy {
        client_max_body_size 0;
        proxy_pass http://localhost:9955;
        proxy_http_version 1.1;
        proxy_redirect                      off;
        proxy_set_header Host               $host;
        proxy_set_header X-Real-IP          $remote_addr;
        proxy_set_header X-Forwarded-For    $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto  $scheme;
        proxy_read_timeout 1m;
        proxy_connect_timeout 1m;
    }
}

The location /deploy must match API_ROOT (default /deploy), and proxy_pass must point to WEBADDRESS:WEBPORT (default 127.0.0.1:9955).

Your site is now ready to respond on /deploy

Logs

The service logs to stderr which is collected by systemd journal. No log file is created on disk.

View logs
journalctl -u ipd
Follow logs
journalctl -u ipd -f

Health and monitoring

Liveness check (no auth required)
GET /deploy/health returns ok
Metrics (ADMIN role) in Prometheus text format
GET /deploy/metrics

The systemd unit uses Type=notify and WatchdogSec=90: the service reports readiness to systemd and pings the watchdog, so a hung process is restarted automatically. Orphaned upload files of deployments interrupted by a restart are removed on the next start.

Systemd sandbox and Docker

The unit runs the service inside a systemd sandbox, but Docker deployments keep working because:

Docker is accessed through /var/run/docker.sock (AF_UNIX is kept in RestrictAddressFamilies), so the ipd user only needs membership in the docker group — no capabilities are required. Containers are started by the system dockerd, which is not affected by the sandbox.
ProtectSystem=strict is relaxed with ReadWritePaths covering UPLOAD_DIR and the project directories written by control.sh (the example script deploys to /srv, so the unit sets ReadWritePaths=/srv /srv/upload). If your project files live elsewhere, add their parent directory to ReadWritePaths.

Keep in mind that if the watchdog kills a hung service, the running control.sh is killed with it (default KillMode=control-group); leftover files are cleaned up by the recovery pass on next start.

Managing users

Use the built-in CLI instead of editing the auth file by hand:

Add a user and get a token
ipd user add USER alice -f /opt/ipd/users.txt
Add a user with a known token
ipd user add ADMIN bot -f /opt/ipd/users.txt --token MyToken
List users
ipd user list -f /opt/ipd/users.txt
Issue a new token for a user
ipd user passwd alice -f /opt/ipd/users.txt
Remove a user
ipd user del alice -f /opt/ipd/users.txt

The token is shown once on add (only its SHA-256 hash is stored).

Upgrade

Run under root (the virtualenv in /opt/ipd is owned by root).

Activate virtualenv
source /opt/ipd/bin/activate
Stop service
systemctl stop ipd.service — the service waits for running deployments to finish (up to SHUTDOWN_TIMEOUT seconds)
Remove old package
pip uninstall ipd
Install new package
pip install ipd-1.1.0-py3-none-any.whl
Start service
systemctl start ipd.service
Verify
systemctl status ipd.service and recent logs
journalctl -u ipd -n 50

Auth file structure

One user record per line.

ROLE:username:sha256

Supported roles: ADMIN, USER. Empty lines and lines starting with # are ignored.

Example to get hash of password:
echo -n "mypassword" | sha256sum | cut -f 1 -d " "

Environment variables

Must be set through /etc/default/ipd (see EnvironmentFile in ipd.service) or environment variables.

Name	Default	Description
WEBADDRESS	127.0.0.1	Address to listen on
WEBPORT	9955	Port to listen on
UPLOAD_DIR	.	Upload directory
AUTH_DB	/opt/ipd/users.txt	User password file
API_ROOT	/deploy	API root
MAX_UPLOAD_SIZE	10737418240	Max size of upload request body, bytes (10 GiB)
CHECK_TIMEOUT	1800	`control.sh check` timeout, seconds
PREPARE_TIMEOUT	1800	`control.sh prepare` timeout, seconds
DEPLOY_TIMEOUT	1800	`control.sh deploy` timeout, seconds
SHUTDOWN_TIMEOUT	60	Max time to wait for running deployments on shutdown

Prepare dev environment

Setup virtualenv
python3 -m venv venv
Activate virtualenv
source ./venv/bin/activate
Install app with dev dependencies
pip install -e .[dev]
Install pre-commit hooks
pre-commit install
Run linters manually (black, mypy)
pre-commit run --all-files
Run tests
pytest tests/

Return codes for deploy status

Request deploy status by retrieving url https://$REMOTE_HOST/deploy/status/$PROJECT/$PROJECT_DEPLOYMENT.

In return you retrieve states:

Code	Description
await	Awaiting in queue
active	Deployment enrolling
ok	Deployment successful
no	No such project processed
changed	Another deployment added while processing
cancelled	Deployment was cancelled from the queue
another code	Code number returned by deployment script

Name		Name	Last commit message	Last commit date
example		example
ipd		ipd
tests		tests
.env.example		.env.example
.gitignore		.gitignore
.gitlab-ci.yml		.gitlab-ci.yml
.pre-commit-config.yaml		.pre-commit-config.yaml
README.md		README.md
ipd.service		ipd.service
package.json		package.json
pyproject.toml		pyproject.toml

Navigation Menu

Repository files navigation

IPD - Image Push and Deploy

Concept

Deploy flow

Queue semantics

Endpoints

Idempotent uploads

Cancelling a deployment

Response formats

Environment variables for control.sh

Installation

Logs

Health and monitoring

Systemd sandbox and Docker

Managing users

Upgrade

Auth file structure

Environment variables

Prepare dev environment

Return codes for deploy status

About

Releases

Packages

Used by

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

IPD - Image Push and Deploy

Concept

Deploy flow

Queue semantics

Endpoints

Idempotent uploads

Cancelling a deployment

Response formats

Environment variables for control.sh

Installation

Logs

Health and monitoring

Systemd sandbox and Docker

Managing users

Upgrade

Auth file structure

Environment variables

Prepare dev environment

Return codes for deploy status

About

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages