Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Windows Arm64 - building from source #12326

Closed
doberst opened this issue Jan 22, 2025 · 15 comments
Closed

Windows Arm64 - building from source #12326

doberst opened this issue Jan 22, 2025 · 15 comments
Labels
waiting-on-reporter Issue is waiting on a reply from the reporter. It will be automatically cloesd if there is no reply.

Comments

@doberst
Copy link

doberst commented Jan 22, 2025

Hi Cryptography team!

I am trying to use the library on a Windows ARM64 machine, and looks like the only alternative is to build from source. I have set up the environment with the OpenSSL binaries for ARM64, but am having a lot of difficulties compiling the Rust source code, and the recipes provided in the documentation do not seem to be up-to-date or provide enough detail to get it working.

After a lot of tinkering, the Rust "cargo build" will find the OpenSSL binary, but then later in the process throws an error around not being able to find the path to "build_openssl.py" (from src\rust\cryptography-cffi\build.rs @ line 36). I have tried multiple obvious ways to ensure it finds the path, and not finding success with any. (Please note: I am not a Rust programmer - and am trying to use the cryptography library in a Python/C/C++ project.)

Welcome any feedback or advice. Generally, it would be great if you could support Windows ARM64, or alternatively, provide a more detailed recipe for how to build from source for this use case.

Thanks!

@alex
Copy link
Member

alex commented Jan 22, 2025

We'll ship Windows ARM64 wheels as soon as Github offers ARM64 Windows runners (#11199).

Have you tried using vcpkg to install OpenSSL?

@reaperhulk
Copy link
Member

vcpkg may make this simple, but if you have an installation of OpenSSL at a path like C:\openssl (with lib and include dirs under that) you can add OPENSSL_DIR=C:\openssl to your environment vars and the rust compilation will see it when invoked by the Python build machinery.

@alex alex added the waiting-on-reporter Issue is waiting on a reply from the reporter. It will be automatically cloesd if there is no reply. label Jan 25, 2025
@doberst
Copy link
Author

doberst commented Jan 27, 2025

@reaperhulk and @alex - thanks for your suggestions - I have not used vcpkg, but have installed OpenSSL and set the path as you describe but still getting errors in the Rust compiling process. I will explore vcpkg and other config options. Will close the loop when I get it resolved. Welcome and appreciate any other ideas. :)

Copy link

This issue has been waiting for a reporter response for 3 days. It will be auto-closed if no activity occurs in the next 5 days.

@github-actions github-actions bot added the Stale label Jan 31, 2025
@dlech
Copy link

dlech commented Feb 2, 2025

I had success with:

winget install ShiningLight.OpenSSL.Dev
$env:OPENSSL_DIR="C:\Program Files\OpenSSL-Win64-ARM"
$env:OPENSSL_LIB_DIR="C:\Program Files\OpenSSL-Win64-ARM\lib\VC\arm64\MT"
$env:OPENSSL_STATIC="1"

@doberst
Copy link
Author

doberst commented Feb 2, 2025

@dlech - thanks for this suggestion - I am going to try your approach this morning

@doberst
Copy link
Author

doberst commented Feb 2, 2025

@dlech - thanks, your suggestion is spot-on for getting OpenSSL installed on Win64-ARM. Perhaps I am missing something obvious (so apologies in advance) - but how are you building/compiling cryptography from source after these instructions. I have tried several paths, and getting errors in all of them. If you could complete the recipe with the next couple of lines to get cryptography built (e.g., rust pyd), I would really appreciate it! :)

@dlech
Copy link

dlech commented Feb 2, 2025

I just did poetry install on a project that has cryptography as a dependency. So basically, it should be the same as pip install cryptography.

For a different dependency (not sure if it is related to cryptography or not), I had to set $env:PYO3_USE_ABI3_FORWARD_COMPATIBILITY="1" to get it to build on Python 3.13. That might have been the rust-pyd that you are referring to, but I don't have the terminal output anymore to check.

@github-actions github-actions bot removed the Stale label Feb 3, 2025
@doberst
Copy link
Author

doberst commented Feb 3, 2025

Thanks to @dlech - I have a build of OPEN SSL for Windows ARM64 - and this allows the pip install of cryptography to progress without error - but after installing, the library does not seem to function, and throws errors upon trying to use in a project:

from cryptography.hazmat.bindings._rust import exceptions as rust_exceptions
ImportError: DLL load failed while importing _rust: The specified module could not be found.

My question remains the same: once configuring for the OPEN SSL lib, what is the process to build the Rust code from source for Windows ARM64 ? (e.g., cmake, cargo build, built in pip install, etc)

@dlech
Copy link

dlech commented Feb 3, 2025

You could try to figure out how to statically link OpenSSL instead of dynamically linking it, or you can copy the .dll to the directory where the python3.exe is so that Windows can find it.

@alex
Copy link
Member

alex commented Feb 3, 2025 via email

@dlech
Copy link

dlech commented Feb 3, 2025

Thanks, I edited my previous comment to include that. And changed MD to MT to select the correct folder for OPENSSL_LIB_DIR.

@doberst
Copy link
Author

doberst commented Feb 4, 2025

@alex and @dlech - appreciate all of your suggestions - some progress but still not working. I can successfully pip install cryptography with the following steps on Windows Arm64 with Python 3.12 (standard Lenovo Snapdragon laptop):

winget install ShiningLight.OpenSSL.Dev  # installs OpenSSL at the path below
$env:OPENSSL_DIR="C:\Program Files\OpenSSL-Win64-ARM"
$env:OPENSSL_LIB_DIR="C:\Program Files\OpenSSL-Win64-ARM\lib\VC\arm64\MT"   # note-change to MT, not MS (have tried MD as well)
$env:OPENSSL_STATIC="1"

# take steps to purge pip cache and force reinstall, e.g., --force-reinstall, pip cache purge, etc.

pip3 install cryptography --force-reinstall --verbose

Produces successful pip install, and in .venv/lib/site-packages/cryptography, there is _rust.pyd file installed with overall cryptography folder structure and files that appear to match the working version on amd_x86.

However, upon attempting to import cryptography into a project, e.g., from cryptography.fernet import Fernet, the _rust.pyd file can not be loaded, e.g.,

    from cryptography.hazmat.bindings._rust import exceptions as rust_exceptions
    ImportError: DLL load failed while importing _rust: The specified module could not be found.

I have tried pip install explicitly with the source distribution as well (e.g., pip3 install cryptography-44.0.0.tar.gz), and the pip build is successful, but still throws the import error when trying to load the _rust extension file.

(Without the $env variable settings above, the pip install process fails, as expected, with error of: Rust can not find the openssl library ...)

A couple of ideas:

  1. Is the version of OpenSSL correct, or mis-identified in the pip build script? Any other build configs or dependencies missing?
  2. Misconfigured Rust environment? (It is plain vanilla rust environment install - directly from the rust main download site, and in a couple of limited uses, does appear to work successfully, e.g., compiling Huggingface tokenizers which is rust source build.)

I am sharing below the --verbose pip install log readout if it is helpful- and it does seem to complete the pip install successfully - but throws the _rust import error when attempting to use at run-time:

(.venv) PS C:\Users\darre\PycharmProjects\py_cryptography_build_020225> pip3 install cryptography --verbose --force-reinstall
Using pip 23.2.1 from C:\Users\darre\PycharmProjects\py_cryptography_build_020225\.venv\Lib\site-packages\pip (python 3.12)
Collecting cryptography
  Using cached cryptography-44.0.0.tar.gz (710 kB)
  Running command pip subprocess to install build dependencies
  Collecting maturin<2,>=1
    Obtaining dependency information for maturin<2,>=1 from https://files.pythonhosted.org/packages/a4/f5/051413e04f6da25069db5e76759ecdb8cd2a8ab4a94045b5a3bf548c66fa/maturin-1.8.1-py3-none-win_arm64.whl.metadata
    Using cached maturin-1.8.1-py3-none-win_arm64.whl.metadata (16 kB)
  Collecting cffi>=1.12
    Using cached cffi-1.17.1-cp312-cp312-win_arm64.whl
  Collecting setuptools!=74.0.0,!=74.1.0,!=74.1.1,!=74.1.2
    Obtaining dependency information for setuptools!=74.0.0,!=74.1.0,!=74.1.1,!=74.1.2 from https://files.pythonhosted.org/packages/69/8a/b9dc7678803429e4a3bc9ba462fa3dd9066824d3c607490235c6a796be5a/setuptools-75.8.0-py3-none-any.whl.metadata
    Using cached setuptools-75.8.0-py3-none-any.whl.metadata (6.7 kB)
  Collecting pycparser (from cffi>=1.12)
    Obtaining dependency information for pycparser from https://files.pythonhosted.org/packages/13/a3/a812df4e2dd5696d1f351d58b8fe16a405b234ad2886a0dab9183fb78109/pycparser-2.22-py3-none-any.whl.metadata
    Using cached pycparser-2.22-py3-none-any.whl.metadata (943 bytes)
  Using cached maturin-1.8.1-py3-none-win_arm64.whl (6.6 MB)
  Using cached setuptools-75.8.0-py3-none-any.whl (1.2 MB)
  Using cached pycparser-2.22-py3-none-any.whl (117 kB)
  Installing collected packages: setuptools, pycparser, maturin, cffi
  Successfully installed cffi-1.17.1 maturin-1.8.1 pycparser-2.22 setuptools-75.8.0

  [notice] A new release of pip is available: 23.2.1 -> 25.0
  [notice] To update, run: python.exe -m pip install --upgrade pip
  Installing build dependencies ... done
  Running command Getting requirements to build wheel
  Getting requirements to build wheel ... done
  Running command Preparing metadata (pyproject.toml)
  📦 Including license file "C:\Users\darre\AppData\Local\Temp\pip-install-936lmegi\cryptography_34bfc17f54264587a67e6c19e5c0fffe\LICENSE"
  📦 Including license file "C:\Users\darre\AppData\Local\Temp\pip-install-936lmegi\cryptography_34bfc17f54264587a67e6c19e5c0fffe\LICENSE.APACHE"
  📦 Including license file "C:\Users\darre\AppData\Local\Temp\pip-install-936lmegi\cryptography_34bfc17f54264587a67e6c19e5c0fffe\LICENSE.BSD"
  ðŸ\x8d¹ Building a mixed python/rust project
  🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.7
  📡 Using build options features, locked from pyproject.toml
  cryptography-44.0.0.dist-info
  Checking for Rust toolchain....
  Running `maturin pep517 write-dist-info --metadata-directory C:\Users\darre\AppData\Local\Temp\pip-modern-metadata-ywfk1jvh --interpreter C:\Users\darre\PycharmProjects\py_cryptography_build_020225\.venv\Scripts\python.exe`
  Preparing metadata (pyproject.toml) ... done
Collecting cffi>=1.12 (from cryptography)
  Using cached cffi-1.17.1-cp312-cp312-win_arm64.whl
Collecting pycparser (from cffi>=1.12->cryptography)
  Obtaining dependency information for pycparser from https://files.pythonhosted.org/packages/13/a3/a812df4e2dd5696d1f351d58b8fe16a405b234ad2886a0dab9183fb78109/pycparser-2.22-py3-none-any.whl.metadata
  Using cached pycparser-2.22-py3-none-any.whl.metadata (943 bytes)
Using cached pycparser-2.22-py3-none-any.whl (117 kB)
Building wheels for collected packages: cryptography
  Running command Building wheel for cryptography (pyproject.toml)
  Running `maturin pep517 build-wheel -i C:\Users\darre\PycharmProjects\py_cryptography_build_020225\.venv\Scripts\python.exe --compatibility off`
  📦 Including license file "C:\Users\darre\AppData\Local\Temp\pip-install-936lmegi\cryptography_34bfc17f54264587a67e6c19e5c0fffe\LICENSE"
  📦 Including license file "C:\Users\darre\AppData\Local\Temp\pip-install-936lmegi\cryptography_34bfc17f54264587a67e6c19e5c0fffe\LICENSE.APACHE"
  📦 Including license file "C:\Users\darre\AppData\Local\Temp\pip-install-936lmegi\cryptography_34bfc17f54264587a67e6c19e5c0fffe\LICENSE.BSD"
  ðŸ\x8d¹ Building a mixed python/rust project
  🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.7
  📡 Using build options features, locked from pyproject.toml
     Compiling target-lexicon v0.12.16
     Compiling proc-macro2 v1.0.92
     Compiling unicode-ident v1.0.14
     Compiling shlex v1.3.0
     Compiling pkg-config v0.3.31
     Compiling once_cell v1.20.2
     Compiling vcpkg v0.2.15
     Compiling libc v0.2.166
     Compiling cfg-if v1.0.0
     Compiling autocfg v1.4.0
     Compiling foreign-types-shared v0.1.1
     Compiling cc v1.2.1
     Compiling heck v0.5.0
     Compiling openssl v0.10.68
     Compiling foreign-types v0.3.2
     Compiling bitflags v2.6.0
     Compiling itoa v1.0.14
     Compiling indoc v2.0.5
     Compiling unindent v0.2.3
     Compiling cryptography-key-parsing v0.1.0 (C:\Users\darre\AppData\Local\Temp\pip-install-936lmegi\cryptography_34bfc17f54264587a67e6c19e5c0fffe\src\rust\cryptography-key-parsing)
     Compiling cryptography-openssl v0.1.0 (C:\Users\darre\AppData\Local\Temp\pip-install-936lmegi\cryptography_34bfc17f54264587a67e6c19e5c0fffe\src\rust\cryptography-openssl)
     Compiling memoffset v0.9.1
     Compiling cryptography-rust v0.1.0 (C:\Users\darre\AppData\Local\Temp\pip-install-936lmegi\cryptography_34bfc17f54264587a67e6c19e5c0fffe\src\rust)
     Compiling base64 v0.22.1
     Compiling self_cell v1.0.4
     Compiling pem v3.0.4
     Compiling quote v1.0.37
     Compiling pyo3-build-config v0.23.2
     Compiling syn v2.0.89
     Compiling openssl-sys v0.9.104
     Compiling cryptography-cffi v0.1.0 (C:\Users\darre\AppData\Local\Temp\pip-install-936lmegi\cryptography_34bfc17f54264587a67e6c19e5c0fffe\src\rust\cryptography-cffi)
     Compiling pyo3-ffi v0.23.2
     Compiling pyo3-macros-backend v0.23.2
     Compiling pyo3 v0.23.2
     Compiling asn1_derive v0.20.0
     Compiling openssl-macros v0.1.1
     Compiling asn1 v0.20.0
     Compiling cryptography-x509 v0.1.0 (C:\Users\darre\AppData\Local\Temp\pip-install-936lmegi\cryptography_34bfc17f54264587a67e6c19e5c0fffe\src\rust\cryptography-x509)
     Compiling pyo3-macros v0.23.2
     Compiling cryptography-x509-verification v0.1.0 (C:\Users\darre\AppData\Local\Temp\pip-install-936lmegi\cryptography_34bfc17f54264587a67e6c19e5c0fffe\src\rust\cryptography-x509-verification)
     Compiling cryptography-keepalive v0.1.0 (C:\Users\darre\AppData\Local\Temp\pip-install-936lmegi\cryptography_34bfc17f54264587a67e6c19e5c0fffe\src\rust\cryptography-keepalive)
      Finished `release` profile [optimized] target(s) in 16.68s
  📦 Including files matching "CHANGELOG.rst"
  📦 Including files matching "CONTRIBUTING.rst"
  📦 Including files matching "LICENSE"
  📦 Including files matching "LICENSE.APACHE"
  📦 Including files matching "LICENSE.BSD"
  📦 Including files matching "docs/**/*"
  📦 Including files matching "src/_cffi_src/**/*.py"
  📦 Including files matching "src/_cffi_src/**/*.c"
  📦 Including files matching "src/_cffi_src/**/*.h"
  📦 Including files matching "**/Cargo.toml"
  📦 Including files matching "**/Cargo.lock"
  📦 Including files matching "src/rust/**/*.rs"
  📦 Including files matching "tests/**/*.py"
  📦 Built wheel for abi3 Python ≥ 3.7 to C:\Users\darre\AppData\Local\Temp\pip-install-936lmegi\cryptography_34bfc17f54264587a67e6c19e5c0fffe\target\wheels\cryptography-44.0.0-cp37-abi3-win_arm64.whl
  C:\Users\darre\AppData\Local\Temp\pip-install-936lmegi\cryptography_34bfc17f54264587a67e6c19e5c0fffe\target\wheels\cryptography-44.0.0-cp37-abi3-win_arm64.whl
  Building wheel for cryptography (pyproject.toml) ... done
  Created wheel for cryptography: filename=cryptography-44.0.0-cp37-abi3-win_arm64.whl size=1250186 sha256=99f5fbc1dbf4e58ef972ac1dee6dcdb040336d7a98b9b92c2c32af68729f6250
  Stored in directory: c:\users\darre\appdata\local\pip\cache\wheels\20\59\bd\9f36b59459e77e4416242fa5767dd4e285b8381e961ed7a45c
Successfully built cryptography
Installing collected packages: pycparser, cffi, cryptography
  Attempting uninstall: pycparser
    Found existing installation: pycparser 2.22
    Uninstalling pycparser-2.22:
      Removing file or directory c:\users\darre\pycharmprojects\py_cryptography_build_020225\.venv\lib\site-packages\pycparser-2.22.dist-info\ 
      Removing file or directory c:\users\darre\pycharmprojects\py_cryptography_build_020225\.venv\lib\site-packages\pycparser\
      Successfully uninstalled pycparser-2.22
Successfully installed cffi-1.17.1 cryptography-44.0.0 pycparser-2.22

Copy link

github-actions bot commented Feb 8, 2025

This issue has been waiting for a reporter response for 3 days. It will be auto-closed if no activity occurs in the next 5 days.

@github-actions github-actions bot added the Stale label Feb 8, 2025
@doberst
Copy link
Author

doberst commented Feb 8, 2025

@alex and @dlech - I have a workable resolution to the issue of getting cryptography running on Windows ARM64, and wanted to share the recipe and the issues b/c a more robust set of directions may be useful for others (and building off the steps and insights you had both shared)-

Net-net: the Rust compilation process, triggered inside the build of pyproject.toml, was not linking to two required OpenSSL libs (libcrypto-3-arm64.dll and libssl-3-arm64.dll). This could be a global Rust compiler setting, or perhaps other config in the build environment (I didn't get to root cause, as I had a workaround that suited my purpose).

Without linking to these two libs, the build process completes (without error), and pip install is then also possible without error - however, of course, without the libs, there is an import error on trying to use the library in a project and load the compiled _rust.pyd file.

My work-around is below:

# get OpenSSL dev install for Windows ARM64

winget install ShiningLight.OpenSSL.Dev
$env:OPENSSL_DIR="C:\Program Files\OpenSSL-Win64-ARM"
$env:OPENSSL_LIB_DIR = "C:\Program Files\OpenSSL-Win64-ARM\lib\VC\arm64\MT"

# SKIPPED setting static build env due to Rust linking issues

# build the project

git clone [email protected]:pyca/cryptography.git
cd cryptography
pip3 install build

# using build directly ensured that the Rust src code is compiled for aarch64 using pyproject.toml and creates wheel

python -m build

# after trying this recipe on both Win x86 and aarch64, I found that the static linking on the Rust compiler (out of the box) was not working and as a result, two key dll dependencies were not getting linked to the _rust.pyd file:   libcrypto-3-arm64.dll and libssl-3-arm64.dll - both of these dependencies are in the OpenSSL-Win64-ARM\bin folder - and even with PATH set correctly, they were not getting linked statically into the _rust.pyd file

# note: I could not find any warnings on the 2 missing DLLs in the compiler readout -> I used dumpbin /DEPENDENTS to confirm that the generated _rust.pyd had those two external dependencies.   In contrast, the pip install cryptography using 'standard default OpenSSL' on x86 is self-contained, and does not have those external dependencies.   The pip install _rust.pyd file is ~8 MB on Win x86, and the custom OpenSSL 'build' version without those two libs is ~3 MB.   

# patch the wheel after created with those two required DLLs, copied and pasted from the OpenSSL /bin folder.

shutil.copy(".\\OpenSSL-Win64-ARM\\bin\\libcrypto-3-arm64.dll", ".\\cryptography\\hazmat\\bindings\\libcrypto-3-arm64.dll")
shutil.copy(".\\OpenSSL-Win64-ARM\\bin\\libssl-3-arm64.dll", ".\\cryptography\\hazmat\\bindings\\libssl-3-arm64.dll")

# depending upon project structure, may need to add dll lookup path
os.add_dll_directory(os.path.abspath(".\\cryptography\\hazmat\\bindings\\")

# prior to importing the patched wheel into a project, need to set env variable (I found this issue on both Win x86 and ARM64 when building with custom OpenSSL libs):

os.environ["CRYPTOGRAPHY_OPENSSL_NO_LEGACY"] = "1"

There is probably an easy 1-liner (perhaps already in the setup files or documentation) to configure the Rust compiler so it finds those /bin folder automatically and turns off the blocking "no legacy" error -> and would be great to add in explicit setup directions.

Thanks for your help and support - happy to answer any questions/clarifications - or run any tests if that would be helpful!

@github-actions github-actions bot removed the Stale label Feb 9, 2025
@doberst doberst closed this as completed Feb 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
waiting-on-reporter Issue is waiting on a reply from the reporter. It will be automatically cloesd if there is no reply.
Development

No branches or pull requests

4 participants