Include files and directories while preserving its permissions.

[b04zdotcom]

This is a Feature Proposal

Description

Including a static file or directory and preserving its original file permissions. This would be particularly useful for including binaries in the final package.

Currently the only way that I am aware of to do this, is to run a shell script that copies the files into the .webpack folder before the bundling process. But maybe this functionality can be included in this plugin.

[HyperBrain]

The feature can be part of the 3.1.0 feature release.

[HyperBrain]

I've set the official release date for 3.0.0 to September 9th. We can take care of this right afterwards. See #213 .

[pierreis]

Is there any solution as of today to include static files while waiting for the official support? That seems to be a blocking issue for me.

[HyperBrain]

@pierreis If you do not need exact permission replication but only want to add arbitrary files you can either use the webpack file-loader or the webpack CopyFiles plugin.

file-loader is quite good, as you e.g. can use it to include files that you acquire in your code with require(). It will include the files as is, renames them by default with a hash and changes the requires accordingly. The naming is configurable.

The CopyFiles just adds files to the package/bundle.

[HyperBrain]

I reiterated over this again. If we'd add a includeBinary option to the config, that will only package the binary files or folders in the packaging step. But they would not be available in the compiled output of invoke local which might break the execution locally.
Having that in mind, I'd tend to write a CopyBinary webpack plugin, that is executed at compile time and leaves the binary files in the webpack output folder. Imo that approach is more consistent and puts the copy operation at the place it should be. Effort-wise it shouldn't make any difference.
Any thoughts?

[HyperBrain]

@serverless-heaven/serverless-webpack-contributors Any ideas out there?

[franciscocpg]

@HyperBrain
The #205 (comment) approach looks fine to me.
But I didn't understand why you named it includeBinary and CopyBinary, after all we're talking about any static file, right? Not specific to binaries files.
Or did I miss something?

[HyperBrain]

For files that do not need a replication of the permissions (e.g. executable persmission) in the target the existing webpack plugins like file-loader or the CopyFile plugins already work fine and lead to the expected results.
But this is not true, if you need to copy "raw" files and must persist the file permissions. Either of the plugins will not transfer the executable bit. To distinguish the needed functionality from the already existing I just added "binary" to express the difference. If there's a better way to name it - appreciated 😄

BTW: I'm not aware that you could tweak the webpack copy files plugin to copy permission accordingly, or is there a way to do that?

[franciscocpg]

For files that do not need a replication of the permissions (e.g. executable persmission) in the target the existing webpack plugins like file-loader or the CopyFile plugins already work fine and lead to the expected results.

So that's what I was missing 😃

I don't know any of these two plugins, but to be honest I always tend to try to contribute with existent projects (those plugins two plugins in this case) before creating a new one.
Eg: doesn't it make sense to have a keepFilePermission (or something like that) option in file-loader or CopyFile plugin?

[HyperBrain]

Yes, you're absolutely right. I'd go for the CopyFiles plugin as the loader is meant to be used for explicitly require()'d files like JSON and the like.
Having an option there would solve the problem and save efforts :)

[franciscocpg]

@HyperBrain
What's the project link for CopyFiles plugin?

[HyperBrain]

https://github.com/kevlened/copy-webpack-plugin

[franciscocpg]

That's the project I was taking a look.
Two things then:

1. Someone has already proposed a PR for that feat: add support for copying file permissions (options.copyPermissions) webpack-contrib/copy-webpack-plugin#119.
2. Unfortunately it looks like it's not being maintained anymore. The last merged PR (and commit) was on 2016, Nov 13.

[HyperBrain]

Ok. That's bad. (1) could be at least a workaround short-term if the PR behaves as expected.

But mid-term that's not really a solution. If it is not maintained anymore - as it is a really useful plugin - it would be a solution to ask the current owner and try integrate it into serverless-heaven to be able to maintain it further (if the owner would agree and if there is anyone willing to drive the project as maintainer).

If that's not feasible we need a different solution for our problem 🤔

[HyperBrain]

BTW: They already have a transfer branch with an open PR to change the license.

[franciscocpg]

@HyperBrain
Also see this one
webpack-contrib/copy-webpack-plugin#131

Guys from https://github.com/webpack-contrib/ wants to maintain it

[HyperBrain]

Good find 👍 . Maybe the best way would be to continue with the current working workarounds (see above) and wait for the transition to be finished.

[franciscocpg]

I think so.
And let's pray for the transition finish someday... 🙏

[HyperBrain]

Closing this task now - we'll wait for copy-webpack-plugin to be transferred and the proper PR being merged.
@franciscocpg Thank you very much for the help and the fresh ideas 🥇

If anyone wants to continue the discussion here feel free to do so. That the issue is closed has no impact on that.

[b04zdotcom]

It seems like the wait could be very long.. Is it an option to just fork copy-webpack-plugin and apply the changes as serverless-heaven, or is that not done?

[HyperBrain]

I'm personally against forking it into serverless-heaven because it is in transition and might lead to differences and incompatibilities in case the transfer goes on (which implies additional work that cannot really be estimated upfront). I tend to only have repositories in here that are maintained and where it is guaranteed that they are at a usable state - people will depend on it and might not be able to switch back to the transferred repository if the merged fixes differ over time there and here.

But what I see as an option is, that someone temporarily forks it to his personal GitHub account, merges necessary stuff and we tell people to use that in the meantime.

What do you think?

[b04zdotcom]

That makes sense. I'll fork it and will apply the changes. I will comment here when it's done.

[b04zdotcom]

There is still a lot to do, but this is the forked plugin: webpack-plugin-copy
The permissions copy seems to work, but I still need to write the tests.

Any pull requests are welcome. I'll do my best to actively maintain the package.

[tommedema]

I have an issue where the resulting binary is packaged into every single function, whereas I only want it to packaged into the function where I use it. Unfortunately there is no way to import the binary, so I guess webpack does not know which function needs it. In my case, it should only be in the requestCertificate function:

But it is also present in create, get, list, etc. functions.

This is my webpack.config.js:

const slsw = require('serverless-webpack')
const nodeExternals = require('webpack-node-externals')
var WebpackPluginCopy = require('webpack-plugin-copy')

module.exports = {
  entry: slsw.lib.entries,
  target: 'node',
  stats: 'errors-only',
  externals: [nodeExternals()],
  module: {
    rules: [
      {
        test: /\.js$/,
        loader: 'babel-loader',
        include: __dirname,
        exclude: /node_modules/
      }
    ]
  },
  plugins: [
    new WebpackPluginCopy([{
      copyPermissions: true, // necessary for AWS lambda
      from: 'scripts/certbot-auto'
    }])
  ]
}

And in my serverless.yml I have defined:

package:
  individually: true

Any way to specify for which functions it should be packaged? Preferably I would use file-loader, so that I can import 'certbot-auto' in the function and webpack would know where it is needed. But file-loader does not copy permissions.

[HyperBrain]

@tommedema Yes, you're right. The copy plugin will stupidly copy anything to each compile artifact because it just bypasses webpack's dependency detection. As it is just a single file the file-loader looks like the right direction - if it was configurable to copy the permissions.
Maybe this functionality could be added to the file loader (if it fits into their concept).

Otherwise as a last resort the plugin could be extended to support per function includes (like the module forced include we currently have per service). But I'm not really sure if that works.

Another crazy idea would be that you use the file-loader to get the content of the script into the process, write it to /tmp which is available on Lambda, set the permissions and execute it from there?

[tommedema]

@HyperBrain best solution seems to be to fork file-loader. Until then I will resort to disabling individual packaging...

[fantapop]

As a work around for this issue I was able to use the on-build-webpack plugin to manually fix the files after the webpack build. In my case I'm just adding executable permissions to everything passing through that has .exe for an extension.

const WebpackOnBuildPlugin = require('on-build-webpack');

...

module.exports = {

...

    plugins: [
        new WebpackOnBuildPlugin(fixExecutablePermissions),
    ],

...

function fixExecutablePermissions(stats) {
    for (const filename in stats.compilation.assets) {
        if (filename.endsWith('.exe')) {
            const basePath = stats.compilation.outputOptions.path;
            const fullPath = `${basePath}/${filename}`;
            fs.chmodSync(fullPath, '755');
        }
    }
}