Skip to content

Commit 83dfe30

Browse files
kitarkitar
authored
Merge pull request #34 from kitar/batch-write-item
2 parents edb21cb + e29ad04 commit 83dfe30

File tree

7 files changed

+492
-7
lines changed

7 files changed

+492
-7
lines changed

‎README.md

+85-6
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,11 @@ You can find an example implementation in [kitar/simplechat](https://github.com/
5353
* [Using Global Secondary Indexes](#using-global-secondary-indexes)
5454
+ [index()](#index)
5555
* [Atomic Counter](#atomic-counter)
56+
* [Batch Operations](#batch-operations)
57+
+ [batchGetItem()](#batchgetitem)
58+
+ [batchPutItem()](#batchputitem)
59+
+ [batchDeleteItem()](#batchdeleteitem)
60+
+ [batchWriteItem()](#batchwriteitem)
5661
* [DynamoDB-specific operators for condition() and filter()](#dynamodb-specific-operators-for-condition-and-filter)
5762
+ [Comparators](#comparators)
5863
+ [functions](#functions)
@@ -197,7 +202,6 @@ class User extends Model implements AuthenticatableContract
197202
}
198203
```
199204

200-
> **Note**
201205
> Note that this model is implementing `Illuminate\Contracts\Auth\Authenticatable` and using `Illuminate\Auth\Authenticatable`. This is **optional**, but if we use them, we can use this model with authentication as well. For authentication, please refer to [Authentication section](#authentication-with-model)) for more details.
202206
203207
### Basic Usage
@@ -234,7 +238,6 @@ public static function scan($exclusiveStartKey = null, $sort = 'asc', $limit = 5
234238
}
235239
```
236240

237-
> **Note**
238241
> DynamoDB can only handle result set up to 1MB per call, so we have to paginate if there are more results. see [Paginating the Results](#paginating-the-results) for more details.
239242
240243
#### Retrieving a model
@@ -474,7 +477,6 @@ $response = DB::table('ProductCatalog')
474477
->getItem(['Id' => 101]);
475478
```
476479

477-
> **Note**
478480
> Instead of marshaling manually, pass a plain array. `Kitar\Dynamodb\Query\Grammar` will automatically marshal them before querying.
479481
480482
#### putItem()
@@ -547,7 +549,6 @@ DB::table('ProductCatalog')
547549
]);
548550
```
549551

550-
> **Note**
551552
> Note that we specify `attribute_not_exists` for the operator of condition. This is DynamoDB-specific operator which called `function`. See [DynamoDB-specific operators for condition() and filter()](#dynamodb-specific-operators-for-condition-and-filter) for more details.
552553
553554
OR statements
@@ -625,7 +626,6 @@ $response = DB::table('Thread')
625626
->query();
626627
```
627628

628-
> **Note**
629629
> Note that DynamoDB's `ScanIndexForward` is a feature for `query`. It will not work with `scan`.
630630
631631
### Working with Scans
@@ -755,6 +755,86 @@ DB::('Thread')->key([
755755
]);
756756
```
757757

758+
### Batch Operations
759+
760+
Batch operations can get, put or delete multiple items with a single call. There are some DynamoDB limitations (such as items count, payload size, etc), so please check the documentation in advance. ([BatchGetItem](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchGetItem.html), [BatchWriteItem](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchWriteItem.html))
761+
762+
#### batchGetItem()
763+
764+
```php
765+
DB::table('Thread')
766+
->batchGetItem([
767+
[
768+
'ForumName' => 'Amazon DynamoDB',
769+
'Subject' => 'DynamoDB Thread 1'
770+
],
771+
[
772+
'ForumName' => 'Amazon DynamoDB',
773+
'Subject' => 'DynamoDB Thread 2'
774+
]
775+
]);
776+
```
777+
778+
#### batchPutItem()
779+
780+
```php
781+
DB::table('Thread')
782+
->batchPutItem([
783+
[
784+
'ForumName' => 'Amazon DynamoDB',
785+
'Subject' => 'DynamoDB Thread 3'
786+
],
787+
[
788+
'ForumName' => 'Amazon DynamoDB',
789+
'Subject' => 'DynamoDB Thread 4'
790+
]
791+
]);
792+
```
793+
794+
> This is a handy method to batch-put items using `batchWriteItem`
795+
796+
#### batchDeleteItem()
797+
798+
```php
799+
DB::table('Thread')
800+
->batchDeleteItem([
801+
[
802+
'ForumName' => 'Amazon DynamoDB',
803+
'Subject' => 'DynamoDB Thread 1'
804+
],
805+
[
806+
'ForumName' => 'Amazon DynamoDB',
807+
'Subject' => 'DynamoDB Thread 2'
808+
]
809+
]);
810+
```
811+
812+
> This is a handy method to batch-delete items using `batchWriteItem`
813+
814+
#### batchWriteItem()
815+
816+
```php
817+
DB::table('Thread')
818+
->batchWriteItem([
819+
[
820+
'PutRequest' => [
821+
'Item' => [
822+
'ForumName' => 'Amazon DynamoDB',
823+
'Subject' => 'DynamoDB Thread 3'
824+
]
825+
]
826+
],
827+
[
828+
'DeleteRequest' => [
829+
'Key' => [
830+
'ForumName' => 'Amazon DynamoDB',
831+
'Subject' => 'DynamoDB Thread 1'
832+
]
833+
]
834+
]
835+
]);
836+
```
837+
758838
### DynamoDB-specific operators for condition() and filter()
759839

760840
For `condition` and `filter` clauses, we can use DynamoDB's comparators and functions.
@@ -779,7 +859,6 @@ filter($key, 'begins_with', $value);
779859
filter($key, 'contains', $value);
780860
```
781861

782-
> **Note**
783862
> `size` function is not supported at this time.
784863
785864
## Testing

‎src/Kitar/Dynamodb/Model/Model.php

+4
Original file line numberDiff line numberDiff line change
@@ -354,6 +354,10 @@ public function __call($method, $parameters)
354354
"putItem",
355355
"deleteItem",
356356
"updateItem",
357+
"batchGetItem",
358+
"batchPutItem",
359+
"batchDeleteItem",
360+
"batchWriteItem",
357361
"scan",
358362
"filter",
359363
"filterIn",

‎src/Kitar/Dynamodb/Query/Builder.php

+58
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,20 @@ class Builder extends BaseBuilder
4141
'remove' => []
4242
];
4343

44+
/**
45+
* Keys array for BatchGetItem
46+
* https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchGetItem.html
47+
* @var array
48+
*/
49+
public $batch_get_keys = [];
50+
51+
/**
52+
* RequestItems array for BatchWriteItem
53+
* https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchWriteItem.html
54+
* @var array
55+
*/
56+
public $batch_write_request_items = [];
57+
4458
/**
4559
* ScanIndexForward option.
4660
*/
@@ -312,6 +326,48 @@ public function updateItem($item)
312326
return $this->process('updateItem', 'processSingleItem');
313327
}
314328

329+
public function batchGetItem($keys)
330+
{
331+
$this->batch_get_keys = $keys;
332+
333+
return $this->process('batchGetItem', 'processBatchGetItems');
334+
}
335+
336+
public function batchPutItem($items)
337+
{
338+
$this->batch_write_request_items = collect($items)->map(function ($item) {
339+
return [
340+
'PutRequest' => [
341+
'Item' => $item,
342+
],
343+
];
344+
})->toArray();
345+
346+
return $this->batchWriteItem();
347+
}
348+
349+
public function batchDeleteItem($keys)
350+
{
351+
$this->batch_write_request_items = collect($keys)->map(function ($key) {
352+
return [
353+
'DeleteRequest' => [
354+
'Key' => $key,
355+
],
356+
];
357+
})->toArray();
358+
359+
return $this->batchWriteItem();
360+
}
361+
362+
public function batchWriteItem($request_items = [])
363+
{
364+
if (! empty($request_items)) {
365+
$this->batch_write_request_items = $request_items;
366+
}
367+
368+
return $this->process('batchWriteItem', null);
369+
}
370+
315371
/**
316372
* @inheritdoc
317373
*/
@@ -539,6 +595,8 @@ protected function process($query_method, $processor_method)
539595
$this->grammar->compileKey($this->key),
540596
$this->grammar->compileItem($this->item),
541597
$this->grammar->compileUpdates($this->updates),
598+
$this->grammar->compileBatchGetRequestItems($this->from, $this->batch_get_keys),
599+
$this->grammar->compileBatchWriteRequestItems($this->from, $this->batch_write_request_items),
542600
$this->grammar->compileDynamodbLimit($this->limit),
543601
$this->grammar->compileScanIndexForward($this->scan_index_forward),
544602
$this->grammar->compileExclusiveStartKey($this->exclusive_start_key),

‎src/Kitar/Dynamodb/Query/Grammar.php

+48
Original file line numberDiff line numberDiff line change
@@ -135,6 +135,54 @@ public function compileUpdates($updates)
135135
];
136136
}
137137

138+
public function compileBatchGetRequestItems($table_name, $keys)
139+
{
140+
if (empty($keys)) {
141+
return [];
142+
}
143+
144+
$marshaler = $this->marshaler;
145+
$marshaled_items = collect($keys)->map(function ($key) use ($marshaler) {
146+
return $marshaler->marshalItem($key);
147+
})->toArray();
148+
149+
$table_name = $this->tablePrefix . $table_name;
150+
151+
return [
152+
'RequestItems' => [
153+
$table_name => [
154+
'Keys' => $marshaled_items,
155+
],
156+
]
157+
];
158+
}
159+
160+
public function compileBatchWriteRequestItems($table_name, $request_items)
161+
{
162+
if (empty($request_items)) {
163+
return [];
164+
}
165+
166+
$marshaler = $this->marshaler;
167+
$marshaled_items = collect($request_items)->map(function ($request_item) use ($marshaler) {
168+
return collect($request_item)->map(function ($request_body) use ($marshaler) {
169+
$marshaled = [];
170+
foreach ($request_body as $key => $body) {
171+
$marshaled[$key] = $marshaler->marshalItem($body);
172+
}
173+
return $marshaled;
174+
});
175+
})->toArray();
176+
177+
$table_name = $this->tablePrefix . $table_name;
178+
179+
return [
180+
'RequestItems' => [
181+
$table_name => $marshaled_items,
182+
],
183+
];
184+
}
185+
138186
/**
139187
* Compile the Limit attribute.
140188
*

‎src/Kitar/Dynamodb/Query/Processor.php

+33
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,14 @@ protected function unmarshal(Result $res)
3333
$responseArray['Attributes'] = $this->marshaler->unmarshalItem($responseArray['Attributes']);
3434
}
3535

36+
if (! empty($responseArray['Responses'])) {
37+
foreach ($responseArray['Responses'] as &$items) {
38+
foreach ($items as &$item) {
39+
$item = $this->marshaler->unmarshalItem($item);
40+
}
41+
}
42+
}
43+
3644
return $responseArray;
3745
}
3846

@@ -78,4 +86,29 @@ public function processMultipleItems(Result $awsResponse, $modelClass = null)
7886
return $item;
7987
});
8088
}
89+
90+
public function processBatchGetItems(Result $awsResponse, $modelClass = null)
91+
{
92+
$response = $this->unmarshal($awsResponse);
93+
94+
if (empty($modelClass)) {
95+
return $response;
96+
}
97+
98+
$items = collect();
99+
100+
foreach ($response['Responses'] as $_ => $table_items) {
101+
foreach ($table_items as $item) {
102+
$item = (new $modelClass)->newFromBuilder($item);
103+
$items->push($item);
104+
}
105+
}
106+
107+
unset($response['Responses']);
108+
109+
return $items->map(function ($item) use ($response) {
110+
$item->setMeta($response);
111+
return $item;
112+
});
113+
}
81114
}

0 commit comments

Comments
 (0)