Merge branch 'master' of github.com:humanmade/node-tachyon

CONTRIBUTING.md

@@ -0,0 +1,7 @@
+# Contributing
+
+## Release Process
+
+1. Create and push a new tag following the convention `vx.x.x`
+1. Build a new ZIP file by running `npm run build-docker && npm run build-node-modules && build-zip`
+1. Publish a new GitHub release, uploading `lambda.zip` as the built artifact to GitHub

README.md

@@ -22,61 +22,38 @@ Tachyon is built with some strong opinions and assumptions:
 - Expects original image files to be stored on Amazon S3.
 - Only supports simple image resizing, not a full image manipulation service.
 
-Tachyon works best with WordPress, coupled with [S3 Uploads](github.com/humanmade/s3-uploads) and the [Tachyon Plugin](https://github.com/humanmade/tachyon-plugin).
+Tachyon works best with WordPress, coupled with [S3 Uploads](https://github.com/humanmade/s3-uploads) and the [Tachyon Plugin](https://github.com/humanmade/tachyon-plugin).
 
-![](https://engineering.hmn.md/projects/tachyon/diagram.png)
+**[View Documentation →](docs/README.md)**
 
----
+![](docs/diagram.png)
 
-## Installing
+## Documentation
 
-Tachyon is simple to install, just use the `cloudformation-template.json` in this repository to create the whole stack using AWS CloudFormation. It will ask you for a few details along the way.
+**[View Documentation →](docs/README.md)**
 
-You'll need to upload the latest release to Amazon S3 and put the location in the to the CloudFormation stack configuration.
 
-## Using
-
-Tachyon provides a simple HTTP interface in the form of:
-
-`https://{tachyon-domain}/my/image/path/on/s3.png?w=100&h=80`
-
-It's really that simple!
+### Setup
 
-#### Args Reference
+Tachyon comes in two parts: the [server to serve images](docs/server.md), and the [plugin to use it](docs/plugin.md). To use Tachyon, you need to run at least one server, as well as the plugin on all sites you want to use it.
 
-| URL Arg | Type | Description |
-|---|----|---|
-|`w`|Number|Max width of the image.|
-|`h`|Number|Max height of the image.|
-|`quality`|Number, 0-100|Image quality.|
-|`resize`|String, "w,h"|A comma separated string of the target width and height in pixels. Crops the image.|
-|`crop_strategy`|String, "smart", "entropy", "attention"|There are 3 automatic cropping strategies for use with `resize`: <ul><li>`attention`: good results, ~70% slower</li><li>`entropy`: mediocre results, ~30% slower</li><li>`smart`: best results, ~50% slower</li>|
-|`gravity`|String|Alternative to `crop_strategy`. Crops are made from the center of the image by default, passing one of "north", "northeast", "east", "southeast", "south", "southwest", "west", "northwest" or "center" will crop from that edge.|
-|`fit`|String, "w,h"|A comma separated string of the target maximum width and height. Does not crop the image.|
-|`crop`|Boolean\|String, "x,y,w,h"|Crop an image by percentages x-offset, y-offset, width and height (x,y,w,h). Percentages are used so that you don’t need to recalculate the cropping when transforming the image in other ways such as resizing it. You can crop by pixel values too by appending `px` to the values. `crop=160px,160px,788px,788px` takes a 788 by 788 pixel square starting at 160 by 160.|
-|`zoom`|Number|Zooms the image by the specified amount for high DPI displays. `zoom=2` produces an image twice the size specified in `w`, `h`, `fit` or `resize`. The quality is automatically reduced to keep file sizes roughly equivalent to the non-zoomed image unless the `quality` argument is passed.|
-|`webp`|Boolean, 1|Force WebP format.|
-|`lb`|String, "w,h"|Add letterboxing effect to images, by scaling them to width, height while maintaining the aspect ratio and filling the rest with black or `background`.|
-|`background`|String|Add background color via name (red) or hex value (%23ff0000). Don't forget to escape # as `%23`.|
+The server is also available as a [Docker image](docs/docker.md), which can be used in production or to set up a local test environment.
 
-For more details checkout the [docs](https://engineering.hmn.md/projects/tachyon/).
 
-## Release Process
+## Using
 
-1. Create and push a new tag following the convention `vx.x.x`
-1. Build a new ZIP file by running `npm run build-docker && npm run build-node-modules && build-zip`
-1. Publish a new GitHub release, uploading `lambda.zip` as the built artifact to GitHub
+Tachyon provides a simple HTTP interface in the form of:
 
-## Update Process
+`https://{tachyon-domain}/my/image/path/on/s3.png?w=100&h=80`
 
-Updates to Tachyon can be a combination of the CloudFormation template and a change in the JavaScript code-base. In either case, the CloudFormation stack should be updated with the latest template and the latest build of the code base.
+It's really that simple!
 
-Download the latest version of Tachyon from the GitHub releases page, and upload it to your S3 bucket you used when creating the initial Tachyon stack. It's recommended to use the pattern `tachyon-vx.x.x.zip` in the S3 path name to not overwrite old versions which allows rollbacks.
+**[View Args Reference →](docs/using.md)**
 
-Update the CloudFormation Tachyon stack with the latest CloudFormation template from this repository, and specify the S3 path to the latest Tachyon version ZIP file.
 
 ## Credits
-Created by Human Made for high volume and large-scale sites, such as [Happytables](http://happytables.com/). We run Tachyon on sites with millions of monthly page views, and thousands of sites.
+
+Created by Human Made for high volume and large-scale sites. We run Tachyon on sites with millions of monthly page views, and thousands of sites.
 
 Written and maintained by [Joe Hoyle](https://github.com/joehoyle).
 

docs/README.md

@@ -0,0 +1,23 @@
+# Documentation
+
+Tachyon is a faster than light image resizing service that runs on AWS. Super simple to set up, highly available and very performant.
+
+
+## Setup
+
+Tachyon comes in two parts: the [server to serve images](server.md), and the [plugin to use it](plugin.md). To use Tachyon, you need to run at least one server, as well as the plugin on all sites you want to use it.
+
+The server is also available as a [Docker image](docker.md), which can be used in production or to set up a local test environment.
+
+
+## Documentation
+
+* [Design](design.md) - Motivation and Design
+* [Server Setup](server.md) (or, [Docker Setup](docker.md))
+	* [Installation on AWS](server.md#installation-on-aws)
+	* [Manual Installation on Lambda](server.md#manual-installation-on-lambda)
+	* [Manual Installation](server.md#manual-installation)
+	* [Update Process](server.md#update-process)
+* [Plugin Setup](plugin.md)
+* [Using Tachyon](using.md)
+* [Hints and Tips](tips.md)

docs/design.md

@@ -0,0 +1,21 @@
+# Design
+
+## Why Tachyon?
+
+Tachyon aims at doing one thing well: serving images from S3. It allows resizing those images, cropping, and changing the image quality, but is not intended to be an all-in-one image manipulation server. This keeps the code lightweight and fast.
+
+Tachyon is entirely self-hosted, and relies on AWS infrastructure. This allows you to manage scale as you need, as well as control the cache.
+
+
+## Assumptions
+
+Tachyon is built with some strong opinions and assumptions:
+
+- Runs on AWS (using CloudFront, Lambda and API Gateway.)
+- Expects original image files to be stored on Amazon S3.
+- Only supports simple image resizing, not a full image manipulation service.
+
+
+## Limitations
+
+Tachyon only supports serving from S3 buckets you own, on servers you're running. For other use cases, consider [Photon](https://developer.wordpress.com/docs/photon/) or [Imgix](https://imgix.com/).

docs/docker.md

@@ -0,0 +1,110 @@
+# Using Docker
+
+Tachyon Docker is a docker image that runs the app in a container.
+
+* [Download from GitHub →](https://github.com/humanmade/tachyon-docker)
+* [View on Docker Hub →](https://hub.docker.com/r/humanmade/tachyon/)
+
+## Installation
+
+You'll need a server with docker engine installed. Vagrant has a convenience provider built in,
+and you can also install the [Docker command line tools for OSX, Windows or Linux](https://www.docker.com/products/overview).
+
+You can pull the image from hub.docker.com using the following:
+
+```sh
+docker pull humanmade/tachyon
+```
+
+Or if you prefer to build locally from a clone of the Github repository use the following from
+the same folder as the `Dockerfile`:
+
+```sh
+docker build -t humanmade/tachyon
+```
+
+
+## Usage
+
+To run Tachyon as a service simply run the container passing a few environment variables to it.
+We recommend passing the `-d` flag to daemonize the container and keep it running.
+
+```sh
+docker run -d \
+  --name tachyon \
+  -e AWS_REGION=<region> \
+  -e AWS_S3_BUCKET=<bucket> \
+  humanmade/tachyon
+```
+
+By default the service will expose port `8080`. You can change this by passing the `-p` flag
+to [`docker run`](https://docs.docker.com/engine/reference/run/) and map it to another port.
+
+
+### Custom S3 endpoint
+
+You can specify a custom S3 endpoint if you're running a tool like [fakeS3][] by passing
+the `AWS_S3_ENDPOINT` environment variable to `docker run`.
+
+
+## Vagrant
+
+Tachyon Docker comes with a `Vagrantfile` you can use to spin up a [fakeS3][] server and Tachyon
+server locally.
+
+First install the vagrant hosts updater plugin if you don't have it:
+
+```sh
+vagrant plugin install vagrant-hostsupdater
+```
+
+From a clone of the Github repository run `vagrant up`. Once the machine is provisioned you'll have
+the following servers running:
+
+```txt
+http://s3.srv         # fakeS3 server
+http://tchyn.srv      # Tachyon server
+```
+
+You can configure the URLs and endpoints as well as add additional Tachyon servers by editing
+the `config.yml` file in the cloned repository.
+
+
+### Configuring WordPress
+
+To configure WordPress to use the VM add the following to `wp-config.php` once you've
+installed [S3 Uploads](https://github.com/humanmade/S3-Uploads).
+
+```php
+<?php
+define( 'S3_UPLOADS_BUCKET_URL', 'http://s3.srv/local' );
+define( 'S3_UPLOADS_BUCKET', 'local' );
+
+// These can be any non falsy value
+define( 'S3_UPLOADS_KEY', 'missing' );
+define( 'S3_UPLOADS_SECRET', 'missing' );
+define( 'S3_UPLOADS_REGION', 'eu-west-1' );
+```
+
+And add a small script to your mu-plugins folder:
+
+```php
+<?php
+add_filter( 's3_uploads_s3_client_params', function( $params ) {
+    $params['endpoint'] = 'http://s3.srv/';
+    $params['path_style'] = true;
+    //$params['debug'] = true; // Useful if things aren't working to double check IPs etc
+    return $params;
+} );
+```
+
+You can then install the [Tachyon plugin](plugin.md) and configure it:
+
+```php
+<?php
+// Tachyon URL
+define( 'TACHYON_URL', 'http://tchyn.srv/uploads' );
+```
+
+
+[fakeS3]: https://github.com/jubos/fake-s3

docs/plugin.md

@@ -0,0 +1,21 @@
+# Plugin Setup
+
+The Tachyon plugin is responsible for replacing WordPress' default thumbnail handling with dynamic Tachyon URLs.
+
+[Download from GitHub →](https://github.com/humanmade/tachyon-plugin)
+
+
+## Installation
+
+Install the Tachyon plugin as a regular plugin in your WordPress install (mu-plugins also supported).
+
+You also need to point the plugin to your [Tachyon server](server.md). Add the following to your `wp-config-local.php`:
+
+```php
+define( 'TACHYON_URL', 'http://localhost:8080/<bucket name>/uploads' );
+```
+
+
+## Credits
+
+The Tachyon plugin is based on the Photon plugin code by Automattic, part of [Jetpack](https://github.com/Automattic/jetpack/blob/master/class.photon.php). Used under the GPL.

docs/server.md

@@ -0,0 +1,114 @@
+# Server Setup
+
+The Tachyon server is responsible for generating and serving images.
+
+It can be installed [using AWS CloudFormation](#installation-on-aws), [manually on AWS Lambda](#manual-installation-on-lambda), or [manually using Node.js](#manual-installation). Alternatively, you can use [the Docker installation](docker.md).
+
+
+## Installation on AWS
+
+We highly recommend using Tachyon on [AWS Lambda](https://aws.amazon.com/lambda/details/) to offload image processing task in a serverless configuration. This ensures you don't need lots of hardware to handle thousands of image resize requests, and can scale essentially infinitely. One Tachyon stack is required per S3 bucket, so we recommend using a common region bucket for all sites, which then only requires a single Tachyon stack per region.
+
+[CloudFormation](https://aws.amazon.com/cloudformation/) is the easy way to provision and set up Lambda, and includes the configuration for [CloudFront](https://aws.amazon.com/cloudfront/) and [API Gateway](https://aws.amazon.com/api-gateway/) to make your Lambda function available.
+
+Setting up Tachyon using CloudFormation is a three-step process.
+
+
+### Step 1: Creating Prerequisites
+
+Before provisioning Tachyon itself, you need to first upload Tachyon to an S3 bucket. This needs to be accessible by Lambda, and you need to note the bucket and key (path) for use during CloudFormation provisioning.
+
+You also need to create an SSL certificate in [Amazon Certificate Manager](https://aws.amazon.com/certificate-manager/) for the domain(s) you're using. CloudFormation does not provision this, to allow for use of existing certificates.
+
+**Important Note:** Due to [AWS limitations](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-cloudfront-distributionconfig-viewercertificate.html#cfn-cloudfront-distributionconfig-viewercertificate-acmcertificatearn), this certificate **must be created in us-east-1**.
+
+
+### Step 2: Creating CloudFormation Stack
+
+The next step is creating the CloudFormation stack. This provisions all the necessary resources to run Tachyon, and configures them to all work well with each other.
+
+The CloudFormation template is [available in the GitHub repository](https://github.com/humanmade/tachyon/blob/master/cloudformation-template.json) and can be uploaded directly through the AWS Console, or uploaded to an S3 bucket.
+
+This needs to be configured with all the necessary details:
+
+* `NodeTachyonBucket`: The bucket you uploaded Tachyon to in Step 1. (e.g. `my-meta-bucket`)
+* `NodeTachyonLambdaPath`: The key (path) you uploaded Tachyon to in Step 1. (e.g. `lambda/node-tachyon/lambda-nodejs4.3.zip`)
+* `SSLACMCertificateARN`: ARN for SSL certificate to use from Amazon Certificate Manager.
+* `Domains`: Comma-separated list of domains to attach to this Tachyon instance. (e.g. `us-east-1.tchyn.io`)
+* `UploadsS3Bucket`: Bucket to serve uploads from. (e.g. `my-uploads-bucket`)
+
+Once you've configured the stack, wait for it to provision. This is usually quite slow, as CloudFront can take up to an hour to provision.
+
+
+### Step 3: Tweak API Gateway
+
+Currently, API Gateway does not support setting the method into binary mode (required for images) via CloudFormation. This means we need to tweak it after the stack has been created.
+
+Open the AWS Console, head to API Gateway, and select the newly created <kbd>Tachyon</kbd>, then the <kbd>Binary Support</kbd> item. Set this to `image/*`.
+
+Save, and once the changes have been applied, Tachyon should be fully configured for your domain.
+
+
+## Manual Installation on Lambda
+
+Tachyon can also be manually installed on Lambda, however you will need to handle triggering the Lambda function yourself. This can be done via EC2 instances or another reverse proxy as required.
+
+Simply upload Tachyon to S3 per Step 1 above, then create a new Lambda function with this path set.
+
+![](lambda-upload.png)
+
+Select Node 4.3 for the environment. Tachyon requires the S3 bucket and region to be configured as Environment Variables, with the keys `S3_BUCKET` and `S3_REGION` (these can be changed after creation if required).
+
+![](lambda-env.png)
+
+Configure the rest of your Lambda function as desired.
+
+
+## Manual Installation
+
+If desired, Tachyon can be run on your own servers using Node 4.3. This can also be used for local installations during development.
+
+libvips must be installed on the system as a requirement for sharp. Under Linux, this is built automatically when installing `node_modules`. If you're running on OSX, the easiest way to install libvips is to use homebrew:
+
+```
+brew install homebrew/science/vips --with-webp --with-graphicsmagick
+```
+
+Clone and initialise the repo, including node modules:
+
+```
+git clone [email protected]:humanmade/node-tachyon.git
+npm install
+```
+
+
+### Configuration
+
+Populate the `config.json` with the AWS region and bucket name you want to use, in the following format:
+
+```json
+{
+	"region": "eu-west-1",
+	"bucket": "hmn-uploads-eu"
+}
+```
+
+These can also be passed via the `AWS_REGION`, and `AWS_S3_BUCKET` environment variables if required.
+
+
+### Running the server
+
+```
+node server.js [port] [--debug]
+```
+
+With no options passed you should see the server running by default on `http://localhost:8080/` - this is your Tachyon URL, which you need to configure [when installing the plugin](plugin.md).
+
+
+## Update Process
+
+Updates to Tachyon can be a combination of the CloudFormation template and a change in the JavaScript code-base. In either case, the CloudFormation stack should be updated with the latest template and the latest build of the code base.
+
+Download the latest version of Tachyon from the GitHub releases page, and upload it to your S3 bucket you used when creating the initial Tachyon stack. It's recommended to use the pattern `tachyon-vx.x.x.zip` in the S3 path name to not overwrite old versions which allows rollbacks.
+
+Update the CloudFormation Tachyon stack with the latest CloudFormation template from this repository, and specify the S3 path to the latest Tachyon version ZIP file.

docs/tips.md

@@ -0,0 +1,7 @@
+# Hints and Tips
+
+## Regions
+
+When running Tachyon in production, we recommend running one Tachyon instance per region. This instance should connect to the S3 bucket for the region, which can then be shared across all stacks in that region.
+
+While S3 buckets can be accessed from any region, running Lambda from the same region as the bucket is recommended. This reduces latency and improves image serving speed.