Bulk API

Design notes for the Gitana Bulk API

Usage

var gitana = require("gitana");

  gitana.createTransaction()
    .for("branch://<platformId>/<repositoryId>/<branchId>") // using a reference string
    //.for(branch) // using an object
    .insert({
      "title": "My first article",
      "_type": "custom:article"
    })
    .insert({
      "title": "My second page",
      "_type": "custom:page"
    })
    .remove("GUID1")
    .remove({
      "_doc": "GUID2"
    })
    .read("GUID4") // could we provide a way to bulk read objects?
    .retryCount(3)
    .fail(function(result) {
      // called if the commit fails
    })
    .success(function(result) {
      // called if the commit succeeds
    })
    .commit();

As a first pass, the created transaction should store all JSON objects in-memory until commit() is called. When commit is called, the transaction can run through all objects and send single requests. For N number of objects, there could be N requests to add items to the transaction.

As a second pass, the created transaction could still store all JSON objects in-memory until commit() is called. However, when commit() is called, the objects could be grouped and sent in packages of 10, 100 or more depending on the total size of the JSON payload.

As a third pass, the created transaction could start chunking sends as things are being added. This would remove the requirement to hold things in memory. Local Storage could also be used as a way to queue things. This is not required for the moment, but an intended optimization down the road.

HTTP/REST

The REST API supports four methods for creating, canceling and committing a transaction as well as populating the transaction with objects to commit.

Create a Transaction

Creates a transaction. The only required parameter is a reference against which the transaction will be run. In the case of a transaction containing nodes, the transaction should be referenced to the branch. All references are written out using the reference syntax:

<type>://<platformId>//<datastoreId>[//objectId1][//objectId2]

The method is invoked like this:

POST /transactions?reference=branch://{platformId}/{repositoryId}/{branchId}

The response is:

{ "_doc": "", "container-reference": "" }

#### Add objects to a Transaction
Adds one or more objects to an existing transaction

POST /transactions/{transactionId}/add

The request should look like this:

{ "objects": [{ "header": { "type": "node", "operation": "write" (or "delete") }, "data": { ... data fields ... } }, ... more items ...] }

And the response looks like:

{ "results": [{ "_doc": "", "transactionId": "", "operation": "write" (or "delete"), "type": "node" }, ... more results ...] }

#### Rollback a Transaction
Deletes a transaction and removes any objects associated with it
```DELETE /bulk/transactions/{transactionId}```

#### Commit a Transaction
Commits all of the objects within a transaction to the database
```POST /bulk/transactions/{transactionId}/commit```
The response looks like this:

{ "transactionId": "", "startTime": , "endTime": , "totalCount": , "errorCount": , "successCount": , "results": { "": { "ok": true, "startTime": , "endTime": , "dataId": "", "error": { "message": "[error message]" } } } }