Limit concurrency of remote file requests (#50)

* Limit concurrency of remote file requests

Use bluebird’s Promise.map() with concurrency option to limit HTTP requests when creating remote nodes for file attachments in order to avoid being rate-limited by Airtable’s HTTP servers.

* Require bluebird for map and allow concurrency to be set as option

* Document concurrency option for Attachment fileNode downloads

* Update README.md

Tighten language in documentation

* Tighten language in comment on concurrency default

Author: jonesbp

README.md

@@ -108,6 +108,8 @@ For an example of a markdown-and-airtable-driven site using `gatsby-transformer-
 
 If you are using the `Attachment` type field in Airtable, you may specify a column name with `fileNode` and the plugin will bring in these files. Using this method, it will create "nodes" for each of the files and expose this to all of the transformer plugins. A good use case for this would be attaching images in Airtable, and being able to make these available for use with the `sharp` plugins and `gatsby-image`. Specifying a `fileNode` does require a peer dependency of `gatsby-source-filesystem` otherwise it will fall back as a non-mapped field. The locally available files and any ecosystem connections will be available on the node as `localFiles`.
 
+When using the Attachment type field, this plugin governs requests to download the associated files from Airtable to 5 concurrent requests to prevent excessive requests on Airtable's servers - which can result in refused / hanging connections. You can adjust this limit with the concurrency option in your gatsby-config.js file. Set the option with an integer value for your desired limit on attempted concurrent requests. A value of 0 will allow requests to be made without any limit.
+
 ### The power of views
 
 Within Airtable, every table can have one or more named Views. These Views are a convenient way to pre-filter and sort your data before querying it in Gatsby. If you do not specify a view in your table object, raw data will be returned in no particular order.

gatsby-node.js

@@ -1,10 +1,11 @@
 const Airtable = require("airtable");
 const crypto = require(`crypto`);
 const { createRemoteFileNode } = require(`gatsby-source-filesystem`);
+const { map } = require('bluebird');
 
 exports.sourceNodes = async (
   { actions, createNodeId, store, cache },
-  { apiKey, tables }
+  { apiKey, tables, concurrency }
 ) => {
   // tables contain baseId, tableName, tableView, queryName, mapping, tableLinks
   const { createNode, setPluginStatus } = actions;
@@ -31,6 +32,14 @@ exports.sourceNodes = async (
     return;
   }
 
+  if (concurrency === undefined) {
+    // Airtable hasn't documented what the rate limit against their attachment servers is. 
+    // They do document that API calls are limited to 5 requests/sec, so the default limit of 5 concurrent 
+    // requests for remote files has been selected in that spirit. A higher value can be set as a plugin 
+    // option in gatsby-config.js
+    concurrency = 5;
+  }
+
   console.time(`\nfetch all Airtable rows from ${tables.length} tables`);
 
   let queue = [];
@@ -120,7 +129,10 @@ exports.sourceNodes = async (
     }
   });
 
-  let childNodes = allRows.map(async row => {
+  // Use the map function for arrays of promises imported from Bluebird.
+  // Using the concurrency option protects against being blocked from Airtable's
+  // file attachment servers for large numbers of requests.
+  return map(allRows, async row => {
     // don't love mutating the row here, but
     // not ready to refactor yet to clean this up
     // (happy to take a PR!)
@@ -153,19 +165,11 @@ exports.sourceNodes = async (
     };
 
     createNode(node);
-    return processedData.childNodes;
-  });
 
-  let flattenedChildNodes = await Promise.all(childNodes).then(nodes =>
-    nodes.reduce(
-      (accumulator, currentValue) => accumulator.concat(currentValue),
-      []
-    )
-  );
-
-  return Promise.all(flattenedChildNodes).then(nodes => {
-    nodes.forEach(node => createNode(node));
-  });
+    await Promise.all(processedData.childNodes).then(nodes => {
+      nodes.forEach(node => createNode(node));
+    });
+  }, { concurrency: concurrency });
 };
 
 const processData = async (

package.json

@@ -19,7 +19,8 @@
     "airtable"
   ],
   "dependencies": {
-    "airtable": "^0.5.6"
+    "airtable": "^0.5.6",
+    "bluebird": "^3.5.4"
   },
   "peerDependencies": {
     "gatsby-source-filesystem": ">=2.0.0-rc.0"