Create a universal binary for macos. (#12)

* Build universal2 macOS binaries with PyInstaller

Replace separate macos-amd64 (macos-13) and macos-aarch64 builds with
a single universal2 build on macos-latest. The fat binary works natively
on both Intel and Apple Silicon Macs.

Upload the archive under both legacy names (macos-aarch64, macos-amd64)
for backwards compatibility.

Add verification step using lipo and arch -x86_64 to confirm both
architectures work.

* Install universal2 Python from python.org for macOS build

The setup-python action installs arm64-only Python on macOS ARM
runners, causing PyInstaller to fail with "not a fat binary" errors
when building universal2 binaries.

Install the official python.org universal2 framework build instead,
so all native extensions (bitarray, cryptography, etc.) are compiled
as fat binaries.

* Fix macOS Python setup: add python/pip symlinks

The python.org installer only provides python3/pip3 binaries.
Create symlinks so bare 'python' and 'pip' commands work.

Also fix comment about why we use python.org instead of setup-python.

* Force universal2 native extensions via ARCHFLAGS + no-binary

Pip downloads arm64-only wheels on ARM runners regardless of whether
Python itself is universal2. Use --no-binary :all: with ARCHFLAGS to
compile native extensions (bitarray, cffi, pyyaml, tibs) as fat
binaries for the universal2 PyInstaller build.

* Revert to setup-python for macOS

The python.org manual install is unnecessary — setup-python already
installs universal2 Python. The ARCHFLAGS + --no-binary fix handles
the actual problem (pip's wheel selection).

* Add hidden import for bitstring.bitstore_bitarray

bitstring 4.4.0 dynamically imports bitstore_bitarray, which
PyInstaller doesn't detect. Add --hidden-import for esptool and
espefuse builds.

* Use --collect-submodules bitstring instead of individual hidden imports

bitstring 4.4.0 has many dynamically imported submodules. Collect them
all rather than chasing individual ones.

* Only force-rebuild specific packages as universal2, not all

--no-binary :all: breaks cryptography (needs Rust toolchain).
Instead, install normally first, then force-reinstall only the
packages that don't publish universal2 wheels (bitarray, cffi,
pyyaml, tibs) from source with ARCHFLAGS.

* Add x86_64-apple-darwin Rust target for tibs cross-compilation

tibs uses maturin/Rust and needs the x86_64 target to build a
universal2 fat binary on an ARM runner.

* Fix cross-build: install Rust via rustup, collect bitstring submodules

tibs 0.5.7 requires Rust edition 2024, but the ubuntu 22.04 apt
package only provides Cargo 1.75.0. Install Rust via rustup to get
a recent enough version.

Also add --collect-submodules bitstring for the same PyInstaller
hidden import issue as the main build.

* Fix cross-build: remove comment inside line continuation

A shell comment inside a backslash-continued apt-get command broke
the line continuation, causing 'build-essential: command not found'.

* Add comment explaining --collect-submodules bitstring

Author: floitsch

.github/workflows/build.yml

@@ -76,7 +76,6 @@ jobs:
               python3 python3-pip python3-setuptools python3-wheel python3-dev \
               libffi-dev \
               libssl-dev \
-              rustc cargo \
               build-essential \
               pkg-config \
               git \
@@ -97,15 +96,19 @@ jobs:
             # Via https://github.com/rust-lang/cargo/issues/10781#issuecomment-1351670409
             export CARGO_NET_GIT_FETCH_WITH_CLI=true
             set -e
+            # Install Rust via rustup (apt version is too old for tibs edition2024).
+            curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
+            source "\$HOME/.cargo/env"
             export PATH=\$PATH:/home/builder/.local/bin
             echo \$HOME
             echo \$PATH
             cd /sources
             python -m pip install --upgrade pip
             pip install pyinstaller
             pip install --user -e .
-            pyinstaller --distpath ./${{ matrix.arch }} -F --icon=ci/espressif.ico --add-data="${{ env.STUBS_DIR }}1/*.json:${{ env.STUBS_DIR }}1/" --add-data="${{ env.STUBS_DIR }}2/*.json:${{ env.STUBS_DIR }}2/" esptool.py
-            pyinstaller --distpath ./${{ matrix.arch }} -F --icon=ci/espressif.ico --add-data="${{ env.EFUSE_DIR }}*.yaml:${{ env.EFUSE_DIR }}" espefuse.py
+            # bitstring uses dynamic imports that PyInstaller can't detect; collect all submodules explicitly.
+            pyinstaller --distpath ./${{ matrix.arch }} -F --icon=ci/espressif.ico --collect-submodules bitstring --add-data="${{ env.STUBS_DIR }}1/*.json:${{ env.STUBS_DIR }}1/" --add-data="${{ env.STUBS_DIR }}2/*.json:${{ env.STUBS_DIR }}2/" esptool.py
+            pyinstaller --distpath ./${{ matrix.arch }} -F --icon=ci/espressif.ico --collect-submodules bitstring --add-data="${{ env.EFUSE_DIR }}*.yaml:${{ env.EFUSE_DIR }}" espefuse.py
             pyinstaller --distpath ./${{ matrix.arch }} -F --icon=ci/espressif.ico espsecure.py
             pyinstaller --distpath ./${{ matrix.arch }} -F --icon=ci/espressif.ico esp_rfc2217_server.py
             ./${{ matrix.arch }}/esptool -h

.github/workflows/build_esptool.yml

@@ -23,12 +23,9 @@ jobs:
     runs-on: ${{ matrix.runner }}
     strategy:
       matrix:
-        platform: [macos-aarch64, macos-amd64, windows-amd64, linux-amd64]
+        platform: [macos-universal, windows-amd64, linux-amd64]
         include:
-          - platform: macos-amd64
-            separator: ":"
-            runner: macos-13
-          - platform: macos-aarch64
+          - platform: macos-universal
             separator: ":"
             runner: macos-latest
           - platform: windows-amd64
@@ -77,10 +74,21 @@ jobs:
       - name: Install dependencies
         # Using 6.11.1 as it apparently leads to fewer antivirus false positives.
         # See: https://github.com/espressif/python-binary-action/blob/8f20755b770c5f9d03e9ce3002af330828fe473f/action.yml#L35
+        shell: bash
         run: |
           python -m pip install --upgrade pip
           pip install pyinstaller==6.11.1
-          pip install --user -e .
+          if [[ "${{ runner.os }}" == "macOS" ]]; then
+            # Pip on ARM runners prefers arm64-only wheels over universal2,
+            # which PyInstaller rejects for universal2 builds. Rebuild
+            # native extensions from source as fat binaries.
+            rustup target add x86_64-apple-darwin
+            export ARCHFLAGS="-arch x86_64 -arch arm64"
+            pip install --user -e .
+            pip install --user --force-reinstall --no-binary bitarray,cffi,pyyaml,tibs bitarray cffi pyyaml tibs
+          else
+            pip install --user -e .
+          fi
 
       - name: Import signing keychain (macOS)
         if: (runner.os == 'macOS' && (github.event_name == 'release' || github.event.inputs.sign-macos == 'true'))
@@ -97,11 +105,17 @@ jobs:
           echo "PYINSTALLER_FLAGS=--codesign-identity ${{ vars.MACOS_TEAM_ID }}" >> $GITHUB_ENV
 
       - name: Build with PyInstaller
+        shell: bash
         run: |
-          pyinstaller ${{ env.PYINSTALLER_FLAGS }} --console --distpath ./${{ env.DISTPATH }} -F --icon=ci/espressif.ico --add-data="${{ env.STUBS_DIR }}1/*.json${{ matrix.separator }}${{ env.STUBS_DIR }}1/" --add-data="${{ env.STUBS_DIR }}2/*.json${{ matrix.separator }}${{ env.STUBS_DIR }}2/" --runtime-hook fix_encoding.py esptool.py
-          pyinstaller ${{ env.PYINSTALLER_FLAGS }} --console --distpath ./${{ env.DISTPATH }} -F --icon=ci/espressif.ico --add-data="${{ env.EFUSE_DIR }}*.yaml${{ matrix.separator }}${{ env.EFUSE_DIR }}" --runtime-hook fix_encoding.py espefuse.py
-          pyinstaller ${{ env.PYINSTALLER_FLAGS }} --console --distpath ./${{ env.DISTPATH }} -F --icon=ci/espressif.ico --runtime-hook fix_encoding.py espsecure.py
-          pyinstaller ${{ env.PYINSTALLER_FLAGS }} --console --distpath ./${{ env.DISTPATH }} -F --icon=ci/espressif.ico --runtime-hook fix_encoding.py esp_rfc2217_server.py
+          EXTRA_FLAGS=""
+          if [[ "${{ runner.os }}" == "macOS" ]]; then
+            EXTRA_FLAGS="--target-architecture universal2"
+          fi
+          # bitstring uses dynamic imports that PyInstaller can't detect; collect all submodules explicitly.
+          pyinstaller ${{ env.PYINSTALLER_FLAGS }} $EXTRA_FLAGS --console --distpath ./${{ env.DISTPATH }} -F --icon=ci/espressif.ico --collect-submodules bitstring --add-data="${{ env.STUBS_DIR }}1/*.json${{ matrix.separator }}${{ env.STUBS_DIR }}1/" --add-data="${{ env.STUBS_DIR }}2/*.json${{ matrix.separator }}${{ env.STUBS_DIR }}2/" --runtime-hook fix_encoding.py esptool.py
+          pyinstaller ${{ env.PYINSTALLER_FLAGS }} $EXTRA_FLAGS --console --distpath ./${{ env.DISTPATH }} -F --icon=ci/espressif.ico --collect-submodules bitstring --add-data="${{ env.EFUSE_DIR }}*.yaml${{ matrix.separator }}${{ env.EFUSE_DIR }}" --runtime-hook fix_encoding.py espefuse.py
+          pyinstaller ${{ env.PYINSTALLER_FLAGS }} $EXTRA_FLAGS --console --distpath ./${{ env.DISTPATH }} -F --icon=ci/espressif.ico --runtime-hook fix_encoding.py espsecure.py
+          pyinstaller ${{ env.PYINSTALLER_FLAGS }} $EXTRA_FLAGS --console --distpath ./${{ env.DISTPATH }} -F --icon=ci/espressif.ico --runtime-hook fix_encoding.py esp_rfc2217_server.py
 
       - name: Sign Windows binary
         if: (runner.os == 'Windows' && (github.event_name == 'release' || github.event.inputs.sign-windows == 'true'))
@@ -139,6 +153,15 @@ jobs:
           ./${{ env.DISTPATH }}/espsecure$EXTEN -h
           ./${{ env.DISTPATH }}/esp_rfc2217_server$EXTEN -h
 
+      - name: Verify universal2 binaries (macOS)
+        if: runner.os == 'macOS'
+        run: |
+          for bin in esptool espefuse espsecure esp_rfc2217_server; do
+            echo "Checking $bin..."
+            lipo -info ./${{ env.DISTPATH }}/$bin
+            arch -x86_64 ./${{ env.DISTPATH }}/$bin -h
+          done
+
       - name: Add license and readme
         shell: bash
         run: mv LICENSE README.md ./${{ env.DISTPATH }}
@@ -168,3 +191,27 @@ jobs:
           file: ${{ env.ARCHIVE_NAME }}
           tag: ${{ github.event.release.tag_name }}
           overwrite: true
+
+      - name: Upload macOS release aliases
+        if: github.event_name == 'release' && runner.os == 'macOS'
+        shell: bash
+        run: |
+          for alias in macos-aarch64 macos-amd64; do
+            cp "${{ env.ARCHIVE_NAME }}" "esptool-${alias}.tar.gz"
+          done
+      - name: Upload macOS release (macos-aarch64)
+        if: github.event_name == 'release' && runner.os == 'macOS'
+        uses: svenstaro/upload-release-action@v2
+        with:
+          repo_token: ${{ secrets.GITHUB_TOKEN }}
+          file: esptool-macos-aarch64.tar.gz
+          tag: ${{ github.event.release.tag_name }}
+          overwrite: true
+      - name: Upload macOS release (macos-amd64)
+        if: github.event_name == 'release' && runner.os == 'macOS'
+        uses: svenstaro/upload-release-action@v2
+        with:
+          repo_token: ${{ secrets.GITHUB_TOKEN }}
+          file: esptool-macos-amd64.tar.gz
+          tag: ${{ github.event.release.tag_name }}
+          overwrite: true