Skip to content

Latest commit

 

History

History
323 lines (239 loc) · 27 KB

npm_import.md

File metadata and controls

323 lines (239 loc) · 27 KB

Repository rules to fetch third-party npm packages

Load these with,

load("@aspect_rules_js//npm:npm_import.bzl", "npm_translate_lock", "npm_import")

These use Bazel's downloader to fetch the packages. You can use this to redirect all fetches through a store like Artifactory.

See <https://blog.aspect.dev/configuring-bazels-downloader> for more info about how it works and how to configure it.

npm_translate_lock is the primary user-facing API. It uses the lockfile format from pnpm because it gives us reliable semantics for how to dynamically lay out node_modules trees on disk in bazel-out.

To create pnpm-lock.yaml, consider using pnpm import to preserve the versions pinned by your existing package-lock.json or yarn.lock file.

If you don't have an existing lock file, you can run npx pnpm install --lockfile-only.

Advanced users may want to directly fetch a package from npm rather than start from a lockfile. npm_import does this.

npm_import

npm_import(name, package, version, deps, extra_build_content, transitive_closure, root_package,
           link_workspace, link_packages, run_lifecycle_hooks, lifecycle_hooks_execution_requirements,
           lifecycle_hooks_env, lifecycle_hooks_no_sandbox, integrity, url, patch_args, patches,
           custom_postinstall, npm_auth, bins, kwargs)

Import a single npm package into Bazel.

Normally you'd want to use npm_translate_lock to import all your packages at once. It generates npm_import rules. You can create these manually if you want to have exact control.

Bazel will only fetch the given package from an external registry if the package is required for the user-requested targets to be build/tested.

This is a repository rule, which should be called from your WORKSPACE file or some .bzl file loaded from it. For example, with this code in WORKSPACE:

npm_import(
    name = "npm__at_types_node_15.12.2",
    package = "@types/node",
    version = "15.12.2",
    integrity = "sha512-zjQ69G564OCIWIOHSXyQEEDpdpGl+G348RAKY0XXy9Z5kU9Vzv1GMNnkar/ZJ8dzXB3COzD9Mo9NtRZ4xfgUww==",
)

> This is similar to Bazel rules in other ecosystems named "_import" like > apple_bundle_import, scala_import, java_import, and py_import. > go_repository is also a model for this rule.

The name of this repository should contain the version number, so that multiple versions of the same package don't collide. (Note that the npm ecosystem always supports multiple versions of a library depending on where it is required, unlike other languages like Go or Python.)

To consume the downloaded package in rules, it must be "linked" into the link package in the package's BUILD.bazel file:

load("@npm__at_types_node__15.12.2__links//:defs.bzl", npm_link_types_node = "npm_link_imported_package")

npm_link_types_node(name = "node_modules")

This links @types/node into the node_modules of this package with the target name :node_modules/@types/node.

A :node_modules/@types/node/dir filegroup target is also created that provides the the directory artifact of the npm package. This target can be used to create entry points for binary target or to access files within the npm package.

NB: You can choose any target name for the link target but we recommend using the node_modules/@scope/name and node_modules/name convention for readability.

When using npm_translate_lock, you can link all the npm dependencies in the lock file for a package:

load("@npm//:defs.bzl", "npm_link_all_packages")

npm_link_all_packages(name = "node_modules")

This creates :node_modules/name and :node_modules/@scope/name targets for all direct npm dependencies in the package. It also creates :node_modules/name/dir and :node_modules/@scope/name/dir filegroup targets that provide the the directory artifacts of their npm packages. These target can be used to create entry points for binary target or to access files within the npm package.

If you have a mix of npm_link_all_packages and npm_link_imported_package functions to call you can pass the npm_link_imported_package link functions to the imported_links attribute of npm_link_all_packages to link them all in one call. For example,

load("@npm//:defs.bzl", "npm_link_all_packages")
load("@npm__at_types_node__15.12.2__links//:defs.bzl", npm_link_types_node = "npm_link_imported_package")

npm_link_all_packages(
    name = "node_modules",
    imported_links = [
        npm_link_types_node,
    ]
)

This has the added benefit of adding the imported_links to the convienence :node_modules target which includes all direct dependencies in that package.

NB: You can pass an name to npm_link_all_packages and this will change the targets generated to "{name}/@scope/name" and "{name}/name". We recommend using "node_modules" as the convention for readability.

To change the proxy URL we use to fetch, configure the Bazel downloader:

  1. Make a file containing a rewrite rule like

    rewrite (registry.nodejs.org)/(.*) artifactory.build.internal.net/artifactory/$1/$2

  2. To understand the rewrites, see UrlRewriterConfig in Bazel sources.

  3. Point bazel to the config with a line in .bazelrc like common --experimental_downloader_config=.bazel_downloader_config

Read more about the downloader config: <https://blog.aspect.dev/configuring-bazels-downloader>

PARAMETERS

Name Description Default Value
name Name for this repository rule none
package Name of the npm package, such as acorn or @types/node none
version Version of the npm package, such as 8.4.0 none
deps A dict other npm packages this one depends on where the key is the package name and value is the version {}
extra_build_content Additional content to append on the generated BUILD file at the root of the created repository, either as a string or a list of lines similar to <https://github.com/bazelbuild/bazel-skylib/blob/main/docs/write_file_doc.md>. ""
transitive_closure A dict all npm packages this one depends on directly or transitively where the key is the package name and value is a list of version(s) depended on in the closure. {}
root_package The root package where the node_modules virtual store is linked to. Typically this is the package that the pnpm-lock.yaml file is located when using npm_translate_lock. ""
link_workspace The workspace name where links will be created for this package.

This is typically set in rule sets and libraries that are to be consumed as external repositories so links are created in the external repository and not the user workspace.

Can be left unspecified if the link workspace is the user workspace.		 ""
link_packages Dict of paths where links may be created at for this package to a list of link aliases to link as in each package. If aliases are an empty list this indicates to link as the package name.

Defaults to {} which indicates that links may be created in any package as specified by the direct attribute of the generated npm_link_package.		 {}
run_lifecycle_hooks If true, runs preinstall, install and postinstall lifecycle hooks declared in this package. False
lifecycle_hooks_execution_requirements Execution requirements when running the lifecycle hooks. For example:

lifecycle_hooks_execution_requirements: [ "requires-network" ]		 []
lifecycle_hooks_env Environment variables applied to the preinstall, install, and postinstall lifecycle hooks declared in this package. Lifecycle hooks are defined by providing an array of "key=value" entries. For example:

lifecycle_hooks_env: [ "PREBULT_BINARY=https://downloadurl"],		 []
lifecycle_hooks_no_sandbox If True, a "no-sandbox" execution requirement is added to the lifecycle hook if there is one.

Equivalent to adding "no-sandbox" to lifecycle_hooks_execution_requirements.

This defaults to True to limit the overhead of sandbox creation and copying the output TreeArtifact out of the sandbox.		 True
integrity Expected checksum of the file downloaded, in Subresource Integrity format. This must match the checksum of the file downloaded.

This is the same as appears in the pnpm-lock.yaml, yarn.lock or package-lock.json file.

It is a security risk to omit the checksum as remote files can change.

At best omitting this field will make your build non-hermetic.

It is optional to make development easier but should be set before shipping.		 ""
url Optional url for this package. If unset, a default npm registry url is generated from the package name and version. ""
patch_args Arguments to pass to the patch tool. -p1 will usually be needed for patches generated by git. ["-p0"]
patches Patch files to apply onto the downloaded npm package. []
custom_postinstall Custom string postinstall script to run on the installed npm package. Runs after any existing lifecycle hooks if run_lifecycle_hooks is True. ""
npm_auth Auth token to authenticate with npm. ""
bins Dictionary of node_modules/.bin binary files to create mapped to their node entry points.

This is typically derived from the "bin" attribute in the package.json file of the npm package being linked.

For example:

 bins = {     "foo": "./foo.js",     "bar": "./bar.js", }


In the future, this field may be automatically populated by npm_translate_lock from information in the pnpm lock file. That feature is currently blocked on pnpm/pnpm#5131.		 {}
kwargs Internal use only none

npm_translate_lock

npm_translate_lock(name, pnpm_lock, package_json, npm_package_lock, yarn_lock, npmrc, patches,
                   patch_args, custom_postinstalls, prod, public_hoist_packages, dev, no_optional,
                   lifecycle_hooks_exclude, run_lifecycle_hooks, lifecycle_hooks_envs,
                   lifecycle_hooks_execution_requirements, bins, lifecycle_hooks_no_sandbox,
                   verify_node_modules_ignored, warn_on_unqualified_tarball_url, link_workspace,
                   pnpm_version, kwargs)

Repository rule to generate npm_import rules from pnpm lock file or from a package.json and yarn/npm lock file.

The pnpm lockfile format includes all the information needed to define npm_import rules, including the integrity hash, as calculated by the package manager.

For more details see, https://github.com/pnpm/pnpm/blob/main/packages/lockfile-types/src/index.ts.

Instead of manually declaring the npm_imports, this helper generates an external repository containing a helper starlark module repositories.bzl, which supplies a loadable macro npm_repositories. This macro creates an npm_import for each package.

The generated repository also contains BUILD files declaring targets for the packages listed as dependencies or devDependencies in package.json, so you can declare dependencies on those packages without having to repeat version information.

Bazel will only fetch the packages which are required for the requested targets to be analyzed. Thus it is performant to convert a very large pnpm-lock.yaml file without concern for users needing to fetch many unnecessary packages.

Compatabilty with pnpm*

The node_modules tree laid out by rules_js should be bug-for-bug compatible with the node_modules tree that pnpm lays out with hoisting disabled (hoist=false set in your .npmrc).

We recommend adding hoist=false to your .npmrc:

echo "hoist=false" &gt;&gt; .npmrc

This will prevent pnpm from creating the hidden node_modules/.pnpm/node_modules folder with hoisted dependencies which allow for packages to depend on "phantom" undeclared dependencies. In most cases, if you hit import/require runtime failures in 3rd party npm packages when using rules_js, the failure will be reproducible with pnpm outside of Bazel when hoisting is disabled.

rules_js does not and will not support pnpm "phantom" hoisting which allows for packages to depend on undeclared dependencies. All dependencies between packages must be declared under rules_js in order to support lazy fetching and lazy linking of npm dependencies.

If a 3rd party npm package is relying on "phantom" dependencies to work, the recommended fix for rules_js is to use pnpm.packageExtensions in your package.json to add the missing dependencies or peerDependencies. For example, https://github.com/aspect-build/rules_js/blob/a8c192eed0e553acb7000beee00c60d60a32ed82/package.json#L12.

NB: We plan to add support for the .npmrc public-hoist-pattern setting to rules_js in a future release. For now, you can emulate public-hoist-pattern in rules_js using the public_hoist_packages attribute of npm_translate_lock.

Setup

In WORKSPACE, call the repository rule pointing to your pnpm-lock.yaml file:

load("@aspect_rules_js//npm:npm_import.bzl", "npm_translate_lock")

# Read the pnpm-lock.yaml file to automate creation of remaining npm_import rules
npm_translate_lock(
    # Creates a new repository named "@npm_deps"
    name = "npm_deps",
    pnpm_lock = "//:pnpm-lock.yaml",
    # Recommended attribute that also checks the .bazelignore file
    verify_node_modules_ignored = "//:.bazelignore",
)

Next, there are two choices, either load from the generated repo or check in the generated file. The tradeoffs are similar to this rules_python thread.

  1. Immediately load from the generated repositories.bzl file in WORKSPACE. This is similar to the pip_parse rule in rules_python for example. It has the advantage of also creating aliases for simpler dependencies that don't require spelling out the version of the packages. However it causes Bazel to eagerly evaluate the npm_translate_lock rule for every build, even if the user didn't ask for anything JavaScript-related.
# Following our example above, we named this "npm_deps"
load("@npm_deps//:repositories.bzl", "npm_repositories")

npm_repositories()
  1. Check in the repositories.bzl file to version control, and load that instead. This makes it easier to ship a ruleset that has its own npm dependencies, as users don't have to install those dependencies. It also avoids eager-evaluation of npm_translate_lock for builds that don't need it. This is similar to the update-repos approach from bazel-gazelle.

In a BUILD file, use a rule like write_source_files to copy the generated file to the repo and test that it stays updated:

write_source_files(
    name = "update_repos",
    files = {
        "repositories.bzl": "@npm_deps//:repositories.bzl",
    },
)

Then in WORKSPACE, load from that checked-in copy or instruct your users to do so.

This macro creates a "pnpm" repository. rules_js currently only uses this repository when npm_package_lock or yarn_lock are used rather than pnpm_lock. Set pnpm_version to None to inhibit this repository creation.

The user can create a "pnpm" repository before calling this in order to override.

PARAMETERS

Name Description Default Value
name The repository rule name none
pnpm_lock The pnpm-lock.yaml file.

Exactly one of [pnpm_lock, npm_package_lock, yarn_lock] should be set.		 None
package_json The package.json file. From this file and the corresponding package-lock.json/yarn.lock file (specified with the npm_package_lock/yarn_lock attributes), a pnpm-lock.yaml file will be generated using pnpm import.

Note that any changes to the package.json file will invalidate the npm_translate_lock repository rule, causing it to re-run on the next invocation of Bazel.

Mandatory when using npm_package_lock or yarn_lock, otherwise must be unset.		 None
npm_package_lock The package-lock.json file written by npm install.

When set, the package_json attribute must be set as well. Exactly one of [pnpm_lock, npm_package_lock, yarn_lock] should be set.		 None
yarn_lock The yarn.lock file written by yarn install.

When set, the package_json attribute must be set as well. Exactly one of [pnpm_lock, npm_package_lock, yarn_lock] should be set.		 None
npmrc Available to pnpm when running pnpm import when npm_package_lock or yarn_lock is set.

When set, this attribute will also be used to parse the npm auth token (if any) and use it in npm_import, providing authentication with private npm registries. In a future release, pnpm settings such as public-hoist-patterns will be used.		 None
patches A map of package names or package names with their version (e.g., "my-package" or "[email protected]") to a label list of patches to apply to the downloaded npm package. Paths in the patch file must start with extract_tmp/package where package is the top-level folder in the archive on npm. If the version is left out of the package name, the patch will be applied to every version of the npm package. {}
patch_args A map of package names or package names with their version (e.g., "my-package" or "[email protected]") to a label list arguments to pass to the patch tool. Defaults to -p0, but -p1 will usually be needed for patches generated by git. If patch args exists for a package as well as a package version, then the version-specific args will be appended to the args for the package. {}
custom_postinstalls A map of package names or package names with their version (e.g., "my-package" or "[email protected]") to a custom postinstall script to apply to the downloaded npm package after its lifecycle scripts runs. If the version is left out of the package name, the script will run on every version of the npm package. If a custom postinstall scripts exists for a package as well as for a specific version, the script for the versioned package will be appended with && to the non-versioned package script. {}
prod If true, only install dependencies. False
public_hoist_packages A map of package names or package names with their version (e.g., "my-package" or "[email protected]") to a list of Bazel packages in which to hoist the package to the top-level of the node_modules tree when linking.

This is similar to setting https://pnpm.io/npmrc#public-hoist-pattern in an .npmrc file outside of Bazel, however, wild-cards are not yet supported and npm_translate_lock will fail if there are multiple versions of a package that are to be hoisted.		 {}
dev If true, only install devDependencies False
no_optional If true, optionalDependencies are not installed False
lifecycle_hooks_exclude A list of package names or package names with their version (e.g., "my-package" or "[email protected]") to not run lifecycle hooks on []
run_lifecycle_hooks If true, runs preinstall, install and postinstall lifecycle hooks on npm packages if they exist True
lifecycle_hooks_envs Environment variables applied to the preinstall, install and postinstall lifecycle hooks on npm packages. The environment variables can be defined per package by package name or globally using "". Variables are declared as key/value pairs of the form "key=value".

For example:

 lifecycle_hooks_envs: {     "": ["GLOBAL_KEY1=value1", "GLOBAL_KEY2=value2"],     "@foo/bar": ["PREBULT_BINARY=http://downloadurl"], }
{}
lifecycle_hooks_execution_requirements Execution requirements applied to the preinstall, install and postinstall lifecycle hooks on npm packages.

The execution requirements can be defined per package by package name or globally using "".

For example:

 lifecycle_hooks_execution_requirements: {     "": ["requires-network"],     "@foo/bar": ["no-sandbox"], }
{}
bins Binary files to create in node_modules/.bin for packages in this lock file.

For a given package, this is typically derived from the "bin" attribute in the package.json file of that package.

For example:

 bins = {     "@foo/bar": {         "foo": "./foo.js",         "bar": "./bar.js"     }, }


In the future, this field may be automatically populated from information in the pnpm lock file. That feature is currently blocked on pnpm/pnpm#5131.		 {}
lifecycle_hooks_no_sandbox If True, a "no-sandbox" execution requirement is added to all lifecycle hooks.

Equivalent to adding "*": ["no-sandbox"] to lifecycle_hooks_execution_requirements.

This defaults to True to limit the overhead of sandbox creation and copying the output TreeArtifacts out of the sandbox.		 True
verify_node_modules_ignored node_modules folders in the source tree should be ignored by Bazel.

This points to a .bazelignore file to verify that all nested node_modules directories pnpm will create are listed.

See bazelbuild/bazel#8106		 None
warn_on_unqualified_tarball_url Deprecated. Will be removed in next major release. None
link_workspace The workspace name where links will be created for the packages in this lock file.

This is typically set in rule sets and libraries that vendor the starlark generated by npm_translate_lock so the link_workspace passed to npm_import is set correctly so that links are created in the external repository and not the user workspace.

Can be left unspecified if the link workspace is the user workspace.		 None
pnpm_version pnpm version to use when generating the @pnpm repository. Set to None to not create this repository. "7.9.1"
kwargs Internal use only none