Skip to content

Latest commit

 

History

History
 
 

ops

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
docker
docker
 
 
image
image
 
 
infrastructure
infrastructure
 
 
README.adoc
README.adoc
 
 
backup.sh
backup.sh
 
 
nomad.sh
nomad.sh
 
 
restore.sh
restore.sh
 
 
run-cljdoc-api.sh
run-cljdoc-api.sh
 
 
stats.sh
stats.sh
 
 

README.adoc

cljdoc Operations

Table of Contents

The intention here is not just to make it easy to spin up an environment but also to document what is needed.

Prerequsites

AWS and DigitalOcean access tokens:

export TF_VAR_aws_access_key_id=foo
export TF_VAR_aws_secret_key=foo
export TF_VAR_do_token=digital-ocean-api-token

Terraform and Packer should be installed on your system. Nomad and Consul are optional and only needed if you want to interact with those systems through the CLI.

Server Images (Packer)

We use Packer to create machine images for the cljdoc server. All required files can be found in ./image/. The images are based on CentOS 7. On top of the base OS the following things are added:

  • Docker

  • Nomad

  • Consul

For details see configuration and Systemd service files in conf/. All code that is ran to prepare the image can be found in nomad.packer.json.

Creating an Image

This requires TF_VAR_do_token to contain a valid DigitalOcean token.

cd ops/image
make nomad-image-id

This will create an image and the image-id will be saved to nomad-image-id. This file is required to proceed with the Terraform steps below.

Infrastructure (Terraform)

We use Terraform to create various resources on AWS and DigitalOcean:

  • an S3 bucket for cljdoc releases (+ user & key)

  • an S3 bucket for cljdoc backups (+ user & key)

  • a DigitalOcean droplet to run cljdoc

  • a Route53 HostedZone for domain supplied in the configuration

  • a HostedZone RecordSet to point the domain to the droplet

Note
 The used domain can be customized in cljdoc.tfvars. The domain will not be bought and it is expected that the domain’s nameserver are set to the nameservers of the Route53 HostedZone, which are part of the outputs.

Common Commands

Note
 These commands require the environment variables mentioned in Prerequisites to be set. 
cd ops/infrastructure
terraform plan -var-file cljdoc.tfvars
terraform apply -var-file cljdoc.tfvars
terraform refresh -var-file cljdoc.tfvars

Retrieving outputs:

terraform output
terraform output -json
terraform output main_ip

Packaging (Docker)

cd ops/docker
make image

This will package the cljdoc application in a Docker container. A tag will be determined based on number of commits, branch and commit SHA. Images are published to Docker Hub during CI. See .circleci/config.yml.

Packaging (Uberjar)

TBD.

Orchestration (Nomad)

To deploy the cljdoc service to the provisioned infrastructure we use Nomad. While Nomad also provides a convenient CLI interface it has proven easier to generate Nomad job specs using Clojure and submit them to the Nomad server via HTTP.

The code to do this can be found in modules/deploy.

To run it the following commands can be used:

cd modules/deploy/
docker_tag=$(../../script/version.sh)
clojure -m cljdoc.deploy deploy --docker-tag $docker_tag

This will fail unless the cljdoc Docker repository has an image with the provided tag. The tag names are determined based on Git commit count, branch and HEAD and images are pushed to Docker Hub as part of CI.

Accessing Nomad

./ops/nomad.sh

The script above will launch an SSH process forwarding port 4646 and 8500 from the host specified by main_ip in the Terraform outputs.

With Nomad installed you can now run nomad comands like the following:

nomad status cljdoc
nomad alloc logs -f 683ade58
nomad deployment list

Backing Up Data

See backup.sh and restore.sh.

Restoring from backups stored on S3:

data_dir="/data/cljdoc/"
curl https://s3.amazonaws.com/cljdoc-backups/backup-2018-12-12.tar.gz -o backup.tar.gz
tar -xf backup.tar.gz -C "${data_dir}"
rm backup.tar.gz

SSL Certificates

Traefik generates SSL certificates on demand but seems to have minor bugs here and there. If we need to drop Traefik for some reason we could look into acme.sh: