feat(file): add presigned s3 driver

OpenverseHQ · Feb 1, 2024 · 6d4463d · 6d4463d
1 parent 1f479ba
commit 6d4463d
Show file tree

Hide file tree

Showing 26 changed files with 921 additions and 188 deletions.
diff --git a/docs/file-uploading.md b/docs/file-uploading.md
@@ -4,32 +4,33 @@
 
 ## Table of Contents <!-- omit in toc -->
 
-- [General info](#general-info)
 - [Drivers support](#drivers-support)
-- [Uploading and attach file flow](#uploading-and-attach-file-flow)
-  - [An example of uploading an avatar to a user profile](#an-example-of-uploading-an-avatar-to-a-user-profile)
+- [Uploading and attach file flow for `local` driver](#uploading-and-attach-file-flow-for-local-driver)
+  - [An example of uploading an avatar to a user profile (local)](#an-example-of-uploading-an-avatar-to-a-user-profile-local)
   - [Video example](#video-example)
+- [Uploading and attach file flow for `s3` driver](#uploading-and-attach-file-flow-for-s3-driver)
+  - [Configuration for `s3` driver](#configuration-for-s3-driver)
+  - [An example of uploading an avatar to a user profile (S3)](#an-example-of-uploading-an-avatar-to-a-user-profile-s3)
+- [Uploading and attach file flow for `s3-presigned` driver](#uploading-and-attach-file-flow-for-s3-presigned-driver)
+  - [Configuration for `s3-presigned` driver](#configuration-for-s3-presigned-driver)
+  - [An example of uploading an avatar to a user profile (S3 Presigned URL)](#an-example-of-uploading-an-avatar-to-a-user-profile-s3-presigned-url)
 - [How to delete files?](#how-to-delete-files)
 
 ---
 
-## General info
-
-`MulterModule` from `@nestjs/platform-express` is used to upload files. General principles you can read in [official documentation](https://docs.nestjs.com/techniques/file-upload).
-
----
-
 ## Drivers support
 
-Out of box boilerplate support two drivers: `local` and `s3`. You can set it in `.env` file, variable `FILE_DRIVER`. If you want use other service for storing files, you can extend it.
+Out-of-box boilerplate supports the following drivers: `local`, `s3`, and `s3-presigned`. You can set it in the `.env` file, variable `FILE_DRIVER`. If you want to use another service for storing files, you can extend it.
+
+> For production we recommend using the "s3-presigned" driver to offload your server.
 
 ---
 
-## Uploading and attach file flow
+## Uploading and attach file flow for `local` driver
 
-Endpoint `/api/v1/files/upload` is used for uploading files, which return `File` entity with `id` and `path`. After receiving `File` entity you can attach this to another entity.
+Endpoint `/api/v1/files/upload` is used for uploading files, which returns `File` entity with `id` and `path`. After receiving `File` entity you can attach this to another entity.
 
-### An example of uploading an avatar to a user profile
+### An example of uploading an avatar to a user profile (local)
 
 ```mermaid
 sequenceDiagram
@@ -46,6 +47,131 @@ sequenceDiagram
 
 <https://user-images.githubusercontent.com/6001723/224558636-d22480e4-f70a-4789-b6fc-6ea343685dc7.mp4>
 
+## Uploading and attach file flow for `s3` driver
+
+Endpoint `/api/v1/files/upload` is used for uploading files, which returns `File` entity with `id` and `path`. After receiving `File` entity you can attach this to another entity.
+
+### Configuration for `s3` driver
+
+1. Open https://s3.console.aws.amazon.com/s3/buckets
+1. Click "Create bucket"
+1. Create bucket (for example, `your-unique-bucket-name`)
+1. Open your bucket
+1. Click "Permissions" tab
+1. Find "Cross-origin resource sharing (CORS)" section
+1. Click "Edit"
+1. Paste the following configuration
+
+    ```json
+    [
+      {
+        "AllowedHeaders": ["*"],
+        "AllowedMethods": ["GET"],
+        "AllowedOrigins": ["*"],
+        "ExposeHeaders": []
+      }
+    ]
+    ```
+
+1. Click "Save changes"
+1. Update `.env` file with the following variables:
+
+    ```dotenv
+    FILE_DRIVER=s3
+    ACCESS_KEY_ID=YOUR_ACCESS_KEY_ID
+    SECRET_ACCESS_KEY=YOUR_SECRET_ACCESS_KEY
+    AWS_S3_REGION=YOUR_AWS_S3_REGION
+    AWS_DEFAULT_S3_BUCKET=YOUR_AWS_DEFAULT_S3_BUCKET
+    ```
+
+### An example of uploading an avatar to a user profile (S3)
+
+```mermaid
+sequenceDiagram
+    participant A as Fronted App
+    participant B as Backend App
+    participant C as AWS S3
+
+    A->>B: Upload file via POST /api/v1/files/upload
+    B->>C: Upload file to S3
+    B->>A: Receive File entity with "id" and "path" properties
+    note left of A: Attach File entity to User entity
+    A->>B: Update user via PATCH /api/v1/auth/me
+```
+
+## Uploading and attach file flow for `s3-presigned` driver
+
+Endpoint `/api/v1/files/upload` is used for uploading files. In this case `/api/v1/files/upload` receives only `fileName` property (without binary file), and returns the `presigned URL` and `File` entity with `id` and `path`. After receiving the `presigned URL` and `File` entity you need to upload your file to the `presigned URL` and after that attach `File` to another entity.
+
+### Configuration for `s3-presigned` driver
+
+1. Open https://s3.console.aws.amazon.com/s3/buckets
+1. Click "Create bucket"
+1. Create bucket (for example, `your-unique-bucket-name`)
+1. Open your bucket
+1. Click "Permissions" tab
+1. Find "Cross-origin resource sharing (CORS)" section
+1. Click "Edit"
+1. Paste the following configuration
+
+    ```json
+    [
+      {
+        "AllowedHeaders": ["*"],
+        "AllowedMethods": ["GET", "PUT"],
+        "AllowedOrigins": ["*"],
+        "ExposeHeaders": []
+      }
+    ]
+    ```
+
+   For production we recommend to use more strict configuration:
+
+   ```json
+   [
+     {
+       "AllowedHeaders": ["*"],
+       "AllowedMethods": ["PUT"],
+       "AllowedOrigins": ["https://your-domain.com"],
+       "ExposeHeaders": []
+     },
+      {
+        "AllowedHeaders": ["*"],
+        "AllowedMethods": ["GET"],
+        "AllowedOrigins": ["*"],
+        "ExposeHeaders": []
+      }
+   ]
+   ```
+
+1. Click "Save changes"
+1. Update `.env` file with the following variables:
+
+    ```dotenv
+    FILE_DRIVER=s3-presigned
+    ACCESS_KEY_ID=YOUR_ACCESS_KEY_ID
+    SECRET_ACCESS_KEY=YOUR_SECRET_ACCESS_KEY
+    AWS_S3_REGION=YOUR_AWS_S3_REGION
+    AWS_DEFAULT_S3_BUCKET=YOUR_AWS_DEFAULT_S3_BUCKET
+    ```
+
+### An example of uploading an avatar to a user profile (S3 Presigned URL)
+
+```mermaid
+sequenceDiagram
+    participant C as AWS S3
+    participant A as Fronted App
+    
+    participant B as Backend App
+
+    A->>B: Send file name (not binary file) via POST /api/v1/files/upload
+    note right of B: Generate presigned URL
+    B->>A: Receive presigned URL and File entity with "id" and "path" properties
+    A->>C: Upload file to S3 via presigned URL
+    note right of A: Attach File entity to User entity
+    A->>B: Update user via PATCH /api/v1/auth/me
+```
+
 ## How to delete files?
 
 We prefer not to delete files, as this may have negative experience during restoring data. Also for this reason we also use [Soft-Delete](https://orkhan.gitbook.io/typeorm/docs/delete-query-builder#soft-delete) approach in database. However, if you need to delete files you can create your own handler, cronjob, etc.

diff --git a/env-example-document b/env-example-document
@@ -14,7 +14,7 @@ DATABASE_PASSWORD=secret
 DATABASE_NAME=api
 DATABASE_URL=mongodb://mongo:27017
 
-# Support "local", "s3"
+# Support "local", "s3", "s3-presigned"
 FILE_DRIVER=local
 ACCESS_KEY_ID=
 SECRET_ACCESS_KEY=

diff --git a/env-example-relational b/env-example-relational
@@ -22,7 +22,7 @@ DATABASE_KEY=
 DATABASE_CERT=
 DATABASE_URL=
 
-# Support "local", "s3"
+# Support "local", "s3", "s3-presigned"
 FILE_DRIVER=local
 ACCESS_KEY_ID=
 SECRET_ACCESS_KEY=

diff --git a/package-lock.json b/package-lock.json
diff --git a/package.json b/package.json
@@ -35,6 +35,7 @@
   },
   "dependencies": {
     "@aws-sdk/client-s3": "3.501.0",
+    "@aws-sdk/s3-request-presigner": "^3.498.0",
     "@nestjs/common": "10.3.1",
     "@nestjs/config": "3.1.1",
     "@nestjs/core": "10.3.1",
@@ -101,9 +102,9 @@
     "is-ci": "3.0.1",
     "jest": "29.7.0",
     "prettier": "3.1.1",
-    "supertest": "6.3.4",
     "prompts": "^2.4.2",
     "ts-jest": "29.1.2",
+    "supertest": "6.3.4",
     "ts-loader": "9.5.1",
     "ts-node": "10.9.2",
     "tsconfig-paths": "4.2.0",

diff --git a/src/files/config/file-config.type.ts b/src/files/config/file-config.type.ts
@@ -1,5 +1,11 @@
+export enum FileDriver {
+  LOCAL = 'local',
+  S3 = 's3',
+  S3_PRESIGNED = 's3-presigned',
+}
+
 export type FileConfig = {
-  driver: string;
+  driver: FileDriver;
   accessKeyId?: string;
   secretAccessKey?: string;
   awsDefaultS3Bucket?: string;