Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pin python-libarchive-c. Pin conda and conda-build for VS2019 conda-build step #5646

Merged
merged 2 commits into from
Sep 12, 2024

Conversation

atalman
Copy link
Contributor

@atalman atalman commented Sep 12, 2024

Fixes: pytorch/vision#8651

Trying to resolve this error during vision and audio windows conda builds: https://github.com/pytorch/audio/actions/runs/10829977489/job/30048718670

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "C:\Jenkins\Miniconda3\lib\site-packages\conda\exception_handler.py", line 18, in __call__
        return func(*args, **kwargs)
      File "C:\Jenkins\Miniconda3\lib\site-packages\conda\cli\main.py", line 84, in main_subshell
        exit_code = do_call(args, parser)
      File "C:\Jenkins\Miniconda3\lib\site-packages\conda\cli\conda_argparse.py", line 176, in do_call
        result = plugin_subcommand.action(getattr(args, "_args", args))
      File "C:\Jenkins\Miniconda3\lib\site-packages\conda_build\plugin.py", line 15, in build
If submitted, this report will be used by core maintainers to improve
future releases of conda.
        from .cli.main_build import execute
      File "C:\Jenkins\Miniconda3\lib\site-packages\conda_build\cli\main_build.py", line 19, in <module>
        from .. import api, build, source, utils
      File "C:\Jenkins\Miniconda3\lib\site-packages\conda_build\api.py", line 23, in <module>
        from .config import DEFAULT_PREFIX_LENGTH as _prefix_length
      File "C:\Jenkins\Miniconda3\lib\site-packages\conda_build\config.py", line 23, in <module>
        from .utils import (
      File "C:\Jenkins\Miniconda3\lib\site-packages\conda_build\utils.py", line 49, in <module>
        import libarchive
      File "C:\Jenkins\Miniconda3\lib\site-packages\libarchive\__init__.py", line 1, in <module>
        from .entry import ArchiveEntry
      File "C:\Jenkins\Miniconda3\lib\site-packages\libarchive\entry.py", line 6, in <module>
        from . import ffi
      File "C:\Jenkins\Miniconda3\lib\site-packages\libarchive\ffi.py", line 26, in <module>
        libarchive = ctypes.cdll.LoadLibrary(libarchive_path)
      File "C:\Jenkins\Miniconda3\lib\ctypes\__init__.py", line 452, in LoadLibrary
        return self._dlltype(name)
      File "C:\Jenkins\Miniconda3\lib\ctypes\__init__.py", line 364, in __init__
        if '/' in name or '\\' in name:
    TypeError: argument of type 'NoneType' is not iterable

`$ C:\Jenkins\Miniconda3\Scripts\conda-script.py build -c defaults -c nvidia -c pytorch-nightly --no-anaconda-upload --no-test --python 3.12 --output-folder distr/ packaging/torchaudio`

  environment variables:
                 CIO_TEST=<not set>
    CONDA_ALLOW_SOFTLINKS=false
      CONDA_BUILD_VARIANT=cuda
CONDA_CUDATOOLKIT_CONSTRAINT=- pytorch-cuda=12.1 # [not osx]
                CONDA_ENV=C:\actions-runner\_work\_temp/conda_environment_10829977489
  CONDA_PACKAGE_DIRECTORY=packaging/torchaudio
CONDA_PYTORCH_BUILD_CONSTRAINT=- pytorch==2.5.0.dev20240912
 CONDA_PYTORCH_CONSTRAINT=- pytorch==2.5.0.dev20240912
               CONDA_ROOT=C:\Jenkins\Miniconda3
                CONDA_RUN=conda run -p C:\actions-
                          runner\_work\_temp/conda_environment_10829977489
                CUDA_PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1
           CURL_CA_BUNDLE=<not set>
        GITHUB_EVENT_PATH=C:\actions-runner\_work\_temp\_github_workflow\event.json
              GITHUB_PATH=C:\actions-runner\_work\_temp\_runner_file_commands\add_path_eb2fa2af-
                          6e4a-4eab-96eb-e09885dfb14d
                 INFOPATH=C:\Program Files\Git\usr\local\info;C:\Program
                          Files\Git\usr\share\info;C:\Program Files\Git\usr\info;C:\Program
                          Files\Git\share\info
               LD_PRELOAD=<not set>
                  MANPATH=C:\Program Files\Git\usr\local\man;C:\Program
                          Files\Git\usr\share\man;C:\Program Files\Git\usr\man;C:\Program
                          Files\Git\share\man
            ORIGINAL_PATH=C:\Jenkins\Miniconda3\Scripts;C:\Windows\system32;C:\Windows;C:\Window
                          s\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\
                          System32\OpenSSH;C:\Program Files\Amazon\cfn-
                          bootstrap;C:\ProgramData\chocolatey\bin;C:\Program
                          Files\Amazon\AWSCLIV2;C:\Program Files\Git\cmd;C:\Program
                          Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Program Files
                          (x86)\Windows Kits\10\Windows Performance
                          Toolkit;C:\Users\runneruser\AppData\Local\Microsoft\WindowsApps
                     PATH=C;C:\Program Files\NVIDIA GPU Computing
                          Toolkit\CUDA\v12.1\bin;C:\Users\runneruser\bin;C:\Program
                          Files\Git\usr\local\bin;C:\Program Files\Git\usr\bin;C:\Program
                          Files\Git\usr\bin;C:\Program Files\Git\opt\bin;C:\Jenkins\Miniconda3\S
                          cripts;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Wind
                          ows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\Pro
                          gram Files\Amazon\cfn-
                          bootstrap;C:\ProgramData\chocolatey\bin;C:\Program
                          Files\Amazon\AWSCLIV2;C:\Program Files\Git\cmd;C:\Program
                          Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Program Files
                          (x86)\Windows Kits\10\Windows Performance Toolkit;C:\Users\runneruser\
                          AppData\Local\Microsoft\WindowsApps;C:\Program
                          Files\Git\usr\bin\vendor_perl;C:\Program Files\Git\usr\bin\core_perl
          PKG_CONFIG_PATH=C:\Program Files\Git\usr\lib\pkgconfig;C:\Program
                          Files\Git\usr\share\pkgconfig;C:\Program Files\Git\lib\pkgconfig
             PSMODULEPATH=C:\Windows\system32\WindowsPowerShell\v1.0\Modules;C:\Program
                          Files\WindowsPowerShell\Modules
           PYTHON_VERSION=3.12

Cause of the failure. New python-libarchive-c release:
https://anaconda.org/anaconda/python-libarchive-c/files?version=5.1

During setup binary builds we install following conda and conda-build:

    - conda-build=24.3.0
    - conda=24.4.0

However later we install these ones in different conda env:

    conda-24.7.1               |   py39haa95532_0         963 KB
    conda-build-24.7.1         |   py39haa95532_0         587 KB

This PR makes sure workflow is using same conda and conda-build

Copy link

vercel bot commented Sep 12, 2024

@atalman is attempting to deploy a commit to the Meta Open Source Team on Vercel.

A member of the Team first needs to authorize it.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Sep 12, 2024
@atalman atalman changed the title Pin conda and conda-build for VS2019 conda-build step Pin python-libarchive-c. Pin conda and conda-build for VS2019 conda-build step Sep 12, 2024
@atalman atalman merged commit 32033b0 into pytorch:main Sep 12, 2024
80 of 105 checks passed
@atalman atalman deleted the conda_build_fix branch September 12, 2024 18:04
@kit1980
Copy link
Contributor

kit1980 commented Sep 13, 2024

Cherry-picking this into release/2.5

kit1980 pushed a commit that referenced this pull request Sep 13, 2024
…uild step (#5646)

Fixes: pytorch/vision#8651

Trying to resolve this error during vision and audio windows conda
builds:
https://github.com/pytorch/audio/actions/runs/10829977489/job/30048718670

```
# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "C:\Jenkins\Miniconda3\lib\site-packages\conda\exception_handler.py", line 18, in __call__
        return func(*args, **kwargs)
      File "C:\Jenkins\Miniconda3\lib\site-packages\conda\cli\main.py", line 84, in main_subshell
        exit_code = do_call(args, parser)
      File "C:\Jenkins\Miniconda3\lib\site-packages\conda\cli\conda_argparse.py", line 176, in do_call
        result = plugin_subcommand.action(getattr(args, "_args", args))
      File "C:\Jenkins\Miniconda3\lib\site-packages\conda_build\plugin.py", line 15, in build
If submitted, this report will be used by core maintainers to improve
future releases of conda.
        from .cli.main_build import execute
      File "C:\Jenkins\Miniconda3\lib\site-packages\conda_build\cli\main_build.py", line 19, in <module>
        from .. import api, build, source, utils
      File "C:\Jenkins\Miniconda3\lib\site-packages\conda_build\api.py", line 23, in <module>
        from .config import DEFAULT_PREFIX_LENGTH as _prefix_length
      File "C:\Jenkins\Miniconda3\lib\site-packages\conda_build\config.py", line 23, in <module>
        from .utils import (
      File "C:\Jenkins\Miniconda3\lib\site-packages\conda_build\utils.py", line 49, in <module>
        import libarchive
      File "C:\Jenkins\Miniconda3\lib\site-packages\libarchive\__init__.py", line 1, in <module>
        from .entry import ArchiveEntry
      File "C:\Jenkins\Miniconda3\lib\site-packages\libarchive\entry.py", line 6, in <module>
        from . import ffi
      File "C:\Jenkins\Miniconda3\lib\site-packages\libarchive\ffi.py", line 26, in <module>
        libarchive = ctypes.cdll.LoadLibrary(libarchive_path)
      File "C:\Jenkins\Miniconda3\lib\ctypes\__init__.py", line 452, in LoadLibrary
        return self._dlltype(name)
      File "C:\Jenkins\Miniconda3\lib\ctypes\__init__.py", line 364, in __init__
        if '/' in name or '\\' in name:
    TypeError: argument of type 'NoneType' is not iterable

`$ C:\Jenkins\Miniconda3\Scripts\conda-script.py build -c defaults -c nvidia -c pytorch-nightly --no-anaconda-upload --no-test --python 3.12 --output-folder distr/ packaging/torchaudio`

  environment variables:
                 CIO_TEST=<not set>
    CONDA_ALLOW_SOFTLINKS=false
      CONDA_BUILD_VARIANT=cuda
CONDA_CUDATOOLKIT_CONSTRAINT=- pytorch-cuda=12.1 # [not osx]
                CONDA_ENV=C:\actions-runner\_work\_temp/conda_environment_10829977489
  CONDA_PACKAGE_DIRECTORY=packaging/torchaudio
CONDA_PYTORCH_BUILD_CONSTRAINT=- pytorch==2.5.0.dev20240912
 CONDA_PYTORCH_CONSTRAINT=- pytorch==2.5.0.dev20240912
               CONDA_ROOT=C:\Jenkins\Miniconda3
                CONDA_RUN=conda run -p C:\actions-
                          runner\_work\_temp/conda_environment_10829977489
                CUDA_PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1
           CURL_CA_BUNDLE=<not set>
        GITHUB_EVENT_PATH=C:\actions-runner\_work\_temp\_github_workflow\event.json
              GITHUB_PATH=C:\actions-runner\_work\_temp\_runner_file_commands\add_path_eb2fa2af-
                          6e4a-4eab-96eb-e09885dfb14d
                 INFOPATH=C:\Program Files\Git\usr\local\info;C:\Program
                          Files\Git\usr\share\info;C:\Program Files\Git\usr\info;C:\Program
                          Files\Git\share\info
               LD_PRELOAD=<not set>
                  MANPATH=C:\Program Files\Git\usr\local\man;C:\Program
                          Files\Git\usr\share\man;C:\Program Files\Git\usr\man;C:\Program
                          Files\Git\share\man
            ORIGINAL_PATH=C:\Jenkins\Miniconda3\Scripts;C:\Windows\system32;C:\Windows;C:\Window
                          s\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\
                          System32\OpenSSH;C:\Program Files\Amazon\cfn-
                          bootstrap;C:\ProgramData\chocolatey\bin;C:\Program
                          Files\Amazon\AWSCLIV2;C:\Program Files\Git\cmd;C:\Program
                          Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Program Files
                          (x86)\Windows Kits\10\Windows Performance
                          Toolkit;C:\Users\runneruser\AppData\Local\Microsoft\WindowsApps
                     PATH=C;C:\Program Files\NVIDIA GPU Computing
                          Toolkit\CUDA\v12.1\bin;C:\Users\runneruser\bin;C:\Program
                          Files\Git\usr\local\bin;C:\Program Files\Git\usr\bin;C:\Program
                          Files\Git\usr\bin;C:\Program Files\Git\opt\bin;C:\Jenkins\Miniconda3\S
                          cripts;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Wind
                          ows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\Pro
                          gram Files\Amazon\cfn-
                          bootstrap;C:\ProgramData\chocolatey\bin;C:\Program
                          Files\Amazon\AWSCLIV2;C:\Program Files\Git\cmd;C:\Program
                          Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Program Files
                          (x86)\Windows Kits\10\Windows Performance Toolkit;C:\Users\runneruser\
                          AppData\Local\Microsoft\WindowsApps;C:\Program
                          Files\Git\usr\bin\vendor_perl;C:\Program Files\Git\usr\bin\core_perl
          PKG_CONFIG_PATH=C:\Program Files\Git\usr\lib\pkgconfig;C:\Program
                          Files\Git\usr\share\pkgconfig;C:\Program Files\Git\lib\pkgconfig
             PSMODULEPATH=C:\Windows\system32\WindowsPowerShell\v1.0\Modules;C:\Program
                          Files\WindowsPowerShell\Modules
           PYTHON_VERSION=3.12
```

Cause of the failure. New python-libarchive-c release:
https://anaconda.org/anaconda/python-libarchive-c/files?version=5.1

During setup binary builds we install following conda and conda-build:
```
    - conda-build=24.3.0
    - conda=24.4.0
```

However later we install these ones in different conda env:
```
    conda-24.7.1               |   py39haa95532_0         963 KB
    conda-build-24.7.1         |   py39haa95532_0         587 KB
```

This PR makes sure workflow is using same conda and conda-build
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

All windows conda build jobs are failing
4 participants