Merge develop into master (#255)

* feat: Allow Python runtime to build without requiring a manifest (#243)

* Allow Python to continue build without requirements.txt

* Allow missing requirements.txt file for Python builds

* Ruby optional Gemfile and test

* Style changes

* Add unit tests and additional comments

* Fix assertLogs() not compatible with Python2.7

* Fix assertion string

* Remove unused exception

* Integ. test make sure no artifacts dir has no additional files after build

* Kick off build

* Check for manifest and skip package actions if not found

* Revert Ruby changes until further discussion is had. Make Python workflow more readable.

* Remove unused import

* Whitespace fix

* feat: Allow Ruby runtime to build without requiring a manifest (#245)

* Allow Python to continue build without requirements.txt

* Allow missing requirements.txt file for Python builds

* Ruby optional Gemfile and test

* Style changes

* Add unit tests and additional comments

* Fix assertLogs() not compatible with Python2.7

* Fix assertion string

* Remove unused exception

* Integ. test make sure no artifacts dir has no additional files after build

* Kick off build

* Check for manifest and skip package actions if not found

* Revert Ruby changes until further discussion is had. Make Python workflow more readable.

* Remove unused import

* Whitespace fix

* Readability changes for Ruby workflow

* Remove magic number. Add link to Bundler error codes

* Moved var declaration

* docs: Guidance on integrating with Lambda Builders (#242)

* fix: README - showcase Makefile support (#247)

* Update VS2017 to VS2019 (#244)

* fix: Pip not resolving local packages (#250)

* Fix local packages not being built

* Add int. test to catch future local dependency issues

* Specify test requirements path from cwd

* Removed redundant/superset pattern

* Document the pip regex pattern change

* Updated integ to match use case

* Tests to check backward comp.

* chore: bump version to 1.5.0 (#254)

Co-authored-by: Daniel Mil <[email protected]>
Co-authored-by: Giorgio Azzinnaro <[email protected]>
Co-authored-by: Sriram Madapusi Vasudevan <[email protected]>

Author: 4 people

.appveyor.yml

@@ -1,12 +1,12 @@
 version: 1.0.{build}
 image:
-  - Visual Studio 2017
+  - Visual Studio 2019
   - Ubuntu
 
 environment:
   GOVERSION: 1.11
   GRADLE_OPTS: -Dorg.gradle.daemon=false
-  nodejs_version: "8.10.0"
+  nodejs_version: "10.10.0"
 
   matrix:
 
@@ -49,7 +49,7 @@ for:
 
     install:
     # To run Nodejs workflow integ tests
-    - ps: Install-Product node 8.10
+    - ps: Install-Product node $env:nodejs_version
 
     - "set PATH=%PYTHON%;%PYTHON%\\Scripts;%PYTHON%\\bin;%PATH%"
     - "%PYTHON%\\python.exe -m pip install -r requirements/dev.txt"

DESIGN.md

@@ -8,7 +8,7 @@
 
 
 Lambda Builders is a separate project that contains scripts to build Lambda functions, given a source location. It was
-built as part of SAM CLI `sam build` command.Read https://github.com/awslabs/aws-sam-cli/pull/743 for design document 
+built as part of SAM CLI `sam build` command. Read https://github.com/awslabs/aws-sam-cli/pull/743 for design document
 explaining how Lambda Builders work in context of SAM CLI. 
 
 This project has certain attributes that make it unique:
@@ -60,22 +60,36 @@ to make use of the default implementation. It helps reduce the variance in behav
 customers with a standard expectation. 
 
 #### Command Line Interface (Internal)
-This library provides a wrapper CLI interface for convenience. This interface is not supported at the moment. So we 
-don't provide any guarantees of back compatibility. 
+This library provides a wrapper CLI interface for convenience. This interface **is not supported** at the moment. So we
+don't provide any guarantees of back compatibility.
 
 It is a very thin wrapper over the library. It is meant to integrate
 with tools written in other programming languages that can't import Python libraries directly. The CLI provides
-a JSON-RPC interface over stdin/stdout to invoke the builder and get response.
+[a JSON-RPC 2.0 interface](https://www.jsonrpc.org/specification)
+over stdin/stdout to invoke the builder and get a response.
 
-**Request Format**
+The CLI should be installed and available on the path:
+
+```shell
+pip install aws-lambda-builders
+```
+
+Each execution of `aws-lambda-builders` handles one JSON-RPC request.
+Provide the whole body of the request via stdin, terminated by `EOF`.
+
+Currently, the only exposed method is `LambdaBuilder.build`.
+It closely maps to the
+[Python method `LambdaBuilder.build` in `aws_lambda_builders/builder.py`](aws_lambda_builders/builder.py).
+
+#### Request Format
 
 ```json
 {
   "jsonrpc": "2.0",
   "method": "LambdaBuilder.build",
   "id": 1,
   "params": {
-    "__protocol_version": "0.1",  // Expected version of RPC protocol 
+    "__protocol_version": "0.3",  // expected version of RPC protocol - from aws_lambda_builders/__main__.py
     "capability": {
       "language": "<programming language>",
       "dependency_manager": "<programming language framework>",
@@ -86,13 +100,13 @@ a JSON-RPC interface over stdin/stdout to invoke the builder and get response.
     "scratch_dir": "/path/to/tmp",
     "manifest_path": "/path/to/manifest.json",
     "runtime": "Function's runtime ex: nodejs8.10",
-    "optimizations": {},  // not supported 
-    "options": {}  // not supported
+    "optimizations": {},  // not supported
+    "options": {}  // depending on the workflow
   }
 }
 ```
 
-**Successful Response Format**
+#### Successful Response Format
 
 ```json
 {
@@ -102,7 +116,7 @@ a JSON-RPC interface over stdin/stdout to invoke the builder and get response.
 }
 ```
 
-**Error Response Format**
+#### Error Response Format
 
 ```json
 {
@@ -116,7 +130,7 @@ a JSON-RPC interface over stdin/stdout to invoke the builder and get response.
 }
 ```
 
-**Error Codes**:
+#### Error Codes
 
 Error codes returned by the application are similar to HTTP Status Codes.
 
@@ -125,6 +139,19 @@ Error codes returned by the application are similar to HTTP Status Codes.
 - 505 - RPC Protocol unsupported
 - -32601 - Method unsupported (standard JSON-RPC protocol error code)
 
+#### Params
+
+##### `capability`
+The 3-tuple `capability` is used to identify different workflows.
+As of today, `application_framework` is unused and may be ignored.
+
+##### `options`
+The parameter `options` should be configured depending on the selected workflow/capability.
+
+For more detail around the capabilities and options,
+check out the corresponding _design document_ and `workflow.py` for
+[the workflows you're interested in](aws_lambda_builders/workflows).
+
 ### Project Meta
 #### Directory Structure
 This project's directories are laid as follows:
@@ -173,6 +200,3 @@ And essentially drop into the builders package (or maybe we can have a notion of
 - **builder**: The entire project is called builder, because it can build Lambda functions
 - **workflows**: Building for each language+framework combination is defined using a workflow. 
 - **actions**: A workflow is implemented as a chain of actions.
-
-
-

README.md

@@ -16,6 +16,18 @@ Lambda Builders currently contains the following workflows
 * Go with Dep
 * Go with Mod
 
+In Addition to above workflows, AWS Lambda Builders also supports *Custom Workflows* through a Makefile.
+
 Lambda Builders is the brains behind the `sam build` command from [AWS SAM CLI](https://github.com/awslabs/aws-sam-cli)
 
+### Integrating with Lambda Builders
+
+Lambda Builders is a Python library.
+It additionally exposes a JSON-RPC 2.0 interface to use from other languages.
+
+If you intend to integrate with Lambda Builders,
+check out [this section of the DESIGN DOCUMENT](DESIGN.md#builders-library).
+
+### Contributing
+
 If you are a developer and interested in contributing, read the [DESIGN DOCUMENT](./DESIGN.md) to understand how this works.

aws_lambda_builders/__init__.py

@@ -1,5 +1,5 @@
 """
 AWS Lambda Builder Library
 """
-__version__ = "1.4.0"
+__version__ = "1.5.0"
 RPC_PROTOCOL_VERSION = "0.3"

aws_lambda_builders/workflows/python_pip/packager.py

@@ -35,6 +35,11 @@ class InvalidSourceDistributionNameError(PackagerError):
 
 
 class RequirementsFileNotFoundError(PackagerError):
+    """
+    Exceptions is no longer raised.
+    Keeping it here because this exception is 'public' and could still be used by a customer.
+    """
+
     def __init__(self, requirements_path):
         super(RequirementsFileNotFoundError, self).__init__("Requirements file not found: %s" % requirements_path)
 
@@ -131,9 +136,6 @@ def build_dependencies(self, artifacts_dir_path, scratch_dir_path, requirements_
         # by finding/creating a virtualenv of the correct version and when
         # pip is called set the appropriate env vars.
 
-        if not self.osutils.file_exists(requirements_path):
-            raise RequirementsFileNotFoundError(requirements_path)
-
         self._dependency_builder.build_site_packages(requirements_path, artifacts_dir_path, scratch_dir_path)
 
 
@@ -594,7 +596,10 @@ def main(self, args, env_vars=None, shim=None):
 class PipRunner(object):
     """Wrapper around pip calls used by chalice."""
 
-    _LINK_IS_DIR_PATTERN = "Processing (.+?)\n  Link is a directory, ignoring download_dir"
+    # Update regex pattern to correspond with the updated output from pip
+    # Specific commit:
+    # https://github.com/pypa/pip/commit/b28e2c4928cc62d90b738a4613886fb1e2ad6a81#diff-5225c8e359020adb25dfc8c7a505950fd649c6c5775789c6f6517f7913f94542L529
+    _LINK_IS_DIR_PATTERNS = ["Processing (.+?)\n"]
 
     def __init__(self, python_exe, pip, osutils=None):
         if osutils is None:
@@ -642,10 +647,16 @@ def download_all_dependencies(self, requirements_filename, directory):
                 package_name = match.group(1)
                 raise NoSuchPackageError(str(package_name))
             raise PackageDownloadError(error)
+
+        # Extract local packages from pip output.
+        # Iterate over possible pip outputs depending on pip version.
         stdout = out.decode()
-        matches = re.finditer(self._LINK_IS_DIR_PATTERN, stdout)
-        for match in matches:
-            wheel_package_path = str(match.group(1))
+        wheel_package_paths = set()
+        for pattern in self._LINK_IS_DIR_PATTERNS:
+            for match in re.finditer(pattern, stdout):
+                wheel_package_paths.add(str(match.group(1)))
+
+        for wheel_package_path in wheel_package_paths:
             # Looks odd we do not check on the error status of building the
             # wheel here. We can assume this is a valid package path since
             # we already passed the pip download stage. This stage would have

aws_lambda_builders/workflows/python_pip/workflow.py

@@ -1,11 +1,16 @@
 """
 Python PIP Workflow
 """
+import logging
+
 from aws_lambda_builders.workflow import BaseWorkflow, Capability
 from aws_lambda_builders.actions import CopySourceAction
 from aws_lambda_builders.workflows.python_pip.validator import PythonRuntimeValidator
 
 from .actions import PythonPipBuildAction
+from .utils import OSUtils
+
+LOG = logging.getLogger(__name__)
 
 
 class PythonPipWorkflow(BaseWorkflow):
@@ -59,16 +64,26 @@ class PythonPipWorkflow(BaseWorkflow):
         ".idea",
     )
 
-    def __init__(self, source_dir, artifacts_dir, scratch_dir, manifest_path, runtime=None, **kwargs):
+    def __init__(self, source_dir, artifacts_dir, scratch_dir, manifest_path, runtime=None, osutils=None, **kwargs):
 
         super(PythonPipWorkflow, self).__init__(
             source_dir, artifacts_dir, scratch_dir, manifest_path, runtime=runtime, **kwargs
         )
 
-        self.actions = [
-            PythonPipBuildAction(artifacts_dir, scratch_dir, manifest_path, runtime, binaries=self.binaries),
-            CopySourceAction(source_dir, artifacts_dir, excludes=self.EXCLUDED_FILES),
-        ]
+        if osutils is None:
+            osutils = OSUtils()
+
+        if osutils.file_exists(manifest_path):
+            # If a requirements.txt exists, run pip builder before copy action.
+            self.actions = [
+                PythonPipBuildAction(artifacts_dir, scratch_dir, manifest_path, runtime, binaries=self.binaries),
+                CopySourceAction(source_dir, artifacts_dir, excludes=self.EXCLUDED_FILES),
+            ]
+        else:
+            LOG.warning("requirements.txt file not found. Continuing the build without dependencies.")
+            self.actions = [
+                CopySourceAction(source_dir, artifacts_dir, excludes=self.EXCLUDED_FILES),
+            ]
 
     def get_validators(self):
         return [PythonRuntimeValidator(runtime=self.runtime)]

aws_lambda_builders/workflows/ruby_bundler/bundler.py

@@ -6,6 +6,12 @@
 
 LOG = logging.getLogger(__name__)
 
+"""
+Bundler error codes can be found here:
+https://github.com/rubygems/bundler/blob/3f0638c6c8d340c2f2405ecb84eb3b39c433e36e/lib/bundler/errors.rb#L36
+"""
+GEMFILE_NOT_FOUND = 10
+
 
 class BundlerExecutionError(Exception):
     """
@@ -51,7 +57,15 @@ def run(self, args, cwd=None):
         out, _ = p.communicate()
 
         if p.returncode != 0:
-            # Bundler has relevant information in stdout, not stderr.
-            raise BundlerExecutionError(message=out.decode("utf8").strip())
+            if p.returncode == GEMFILE_NOT_FOUND:
+                LOG.warning("Gemfile not found. Continuing the build without dependencies.")
+
+                # Clean up '.bundle' dir that gets generated before the build fails
+                check_dir = self.osutils.get_bundle_dir(cwd)
+                if self.osutils.directory_exists(check_dir):
+                    self.osutils.remove_directory(check_dir)
+            else:
+                # Bundler has relevant information in stdout, not stderr.
+                raise BundlerExecutionError(message=out.decode("utf8").strip())
 
         return out.decode("utf8").strip()

aws_lambda_builders/workflows/ruby_bundler/utils.py

@@ -6,6 +6,7 @@
 import platform
 import tarfile
 import subprocess
+import shutil
 
 
 class OSUtils(object):
@@ -38,3 +39,12 @@ def abspath(self, path):
 
     def is_windows(self):
         return platform.system().lower() == "windows"
+
+    def directory_exists(self, dirpath):
+        return os.path.exists(dirpath) and os.path.isdir(dirpath)
+
+    def remove_directory(self, dirpath):
+        shutil.rmtree(dirpath)
+
+    def get_bundle_dir(self, cwd):
+        return os.path.join(cwd, ".bundle")

tests/functional/workflows/ruby_bundler/test_ruby_utils.py

@@ -48,3 +48,28 @@ def test_popen_can_accept_cwd(self):
         out, err = p.communicate()
         self.assertEqual(p.returncode, 0)
         self.assertEqual(out.decode("utf8").strip(), os.path.abspath(testdata_dir))
+
+    def test_returns_true_if_directory_exists(self):
+        testdata_dir = os.path.dirname(__file__)
+        out = self.osutils.directory_exists(testdata_dir)
+        self.assertTrue(out)
+
+    def test_returns_false_if_directory_not_found(self):
+        testdata_dir = os.path.join(os.path.dirname(__file__), "test")
+        out = self.osutils.directory_exists(testdata_dir)
+        self.assertFalse(out)
+
+    def test_returns_bundle_directory(self):
+        testdata_dir = os.path.dirname(__file__)
+        out = self.osutils.get_bundle_dir(testdata_dir)
+        self.assertEqual(out, os.path.join(os.path.dirname(__file__), ".bundle"))
+
+    def test_removes_directory_if_exists(self):
+        test_dir = tempfile.mkdtemp()
+        bundle_dir = os.path.join(test_dir, ".bundle")
+        expected_files = set(os.listdir(test_dir))
+        os.mkdir(bundle_dir)
+        self.osutils.remove_directory(bundle_dir)
+        actual_files = set(os.listdir(test_dir))
+        shutil.rmtree(test_dir)
+        self.assertEqual(actual_files, expected_files)