Pip upgrade for direct url requirements

[bibsian]

The default pip version that the alpine docker ships with doesn't support direct URLs for dependency installation.

For example if we're trying to deploy a service with a private repo that has an install file like below, the deployment will error out with:
Direct url requirements in dependencies not supported

# Example private repo setup
setup(
    name="foo",
    packages=find_packages(exclude=["tests"])
    install_requires=(
        read_requirements("requirements.txt")+ # List of dependencies
        ["bar @ git+ssh://git@github.com/bar-project/bar#egg=bar"]
    )
)

I tried to get around this by editing lib/pip.js directly with the following changes:

function installRequirements(targetFolder, serverless, options) {
  ... # other code
  const pipCmd = [
    options.pythonBin,
    '-m',
    'pip',
    'install',
    ...options.pipCmdExtraArgs
  ];
  const pipUpgradeCmd = [
    options.pythonBin,
    '-m',
    'pip',
    'install',
    '--upgrade',
    'pip'
  ];
  const pipCmds = [pipUpgradeCmd,pipCmd];
  ... # other code
}

I can't say if this worked because I'm still debugging my deployment but any info on why this wouldn't work would help me.

[bibsian]

• Update: The deployment with the source code edit (as shown above) is working

And for clarification I tried the pipCmdExtraArgs including the following:

  pipCmdExtraArgs:
    - --upgrade
    - pip

  pipCmdExtraArgs:
    - --upgrade pip

  pipCmdExtraArgs:
    - ['--upgrade', 'pip']

  pipCmdExtraArgs: ['--upgrade', 'pip']

before editing the source code but none of those options worked.

[bibsian]

Also just wanted to note that if I try and upgrade pip through the requirements.txt file I still get the Direct url requirements in dependencies not supported error.

An example requirements.txt file is below where the git project has a direct URL install:

pip >= 19.1
git+ssh://git@github.com/bar-project/bar#egg=bar

[bsamuel-ui]

Options to address this:

• Check for a docker image with a newer pip
• Specifically add an action to update via get-pip.py

[miketheman]

Leaning towards looking at the docker image's version of pip.

[bibsian]

Out of curiosity is there any reason why there can't be a flag in the serverless.yml like:

pipDockerUpgrade:
  - true

Then it would just require a modification to installRequirements function like I did above (note, I don't know node.js and it took me a while to figure out that edit in the original post so not sure if it's that straight forward to integrate into tests and stuff).

[miketheman]

I tried this today:

$ docker run --rm -it lambci/lambda:build-python3.6 pip --version
pip 20.0.2 from /var/lang/lib/python3.6/site-packages/pip (python 3.6)
$ docker image inspect lambci/lambda:build-python3.6 | jq '.[].Created'
"2020-02-06T00:20:51.4461627Z"

Is it possible you're using older versions of the lambci/lambda:build-python3.6 image?

You can update it with docker pull lambci/lambda:build-python3.6 - in fact I did that right after I ran those tests and got a new image (same version of pip):

$ docker image inspect lambci/lambda:build-python3.6 | jq '.[].Created'
"2020-02-21T20:31:27.7162453Z"

[miketheman]

We could default to pulling the latest version of the Docker image every run - and then if someone wanted to "pin" to a specific image version/hash, they can in a custom Dockerfile like so:

FROM lambci/lambda:build-python3.6@sha256:5fba69e61f0a7503608fe92dd9978e22b0d465e9ae80bf489559ace0e0104672
...

[AndrewFarley]

@miketheman Users can already specify a custom image they want to run. If a user has locally cached an older version of the lambda ci build images, I don't think it falls under something we could/should handle.

Especially since currently we still don't have a way in the command-line to display when docker is pulling images, and it leaves users (including myself sometimes) caught off-guard when this plugin takes forever and you docker ps and see no docker images. Promoting/forcing to always use the remote latest version of a tag or something I think would be a bad approach.

I would just be in favor of closing this bug. Or, if the use-case of updating pip is valid enough, we can add a new option... updatePip into the configuration options for this plugin and inject the necessary commands to update pip before running anything.

Thoughts?

[miketheman]

If a user has locally cached an older version of the lambda ci build images, I don't think it falls under something we could/should handle.

I agree with this notion - but would it help the user experience if we checked and told them the base image "old/outdated"? Is that a burden for us to hold? I'm on the fence here.

re: updatePip - I'm leaning away from adding more configuration values when we have perfectly reasonable alternatives available, so I might also be fine with closing this enhancement request, and maybe adding some notes in the docs.

[AndrewFarley]

@miketheman A quick thought on it and a proposed flow and feature of this might be...

1. Upon invocation if using docker and using a non user-specified docker image (eg: an upstream lambci)
   1. Before running docker check the age of the image that is chosen
      1. If the image is non-existent, warn the user that the image will be downloaded (this is a hugely requested feature, although preferably would be a progress bar so the user can follow the progress. No one has figured this one out yet)
      2. If the image is existent but its age is > 3 months, then put up a warning about the image possibly being out of date, include in the warning a recommendation of how to solve the problem.
   2. If any of the above fails, throw a warning but do not fail, since this is merely informational. The next step will probably fail if this fails but in-case there is a failure in the logic here I'd rather fail open than fail closed.
2. Add a new config option into this plugin named dockerPull (or maybe dockerPullAlways) which is set to false by default. If this is set to true then before running docker run, it will docker pull the image to try to fetch the latest image for the tag.

That above sounds totally reasonable to me but maybe there's an edge case I'm not considering that might make for a worse user experience.