Solve "Error: Workflow file too large" problem on Windows

[traversaro]

Recently, on full rebuild the post-PR Windows builds started failing with:

Error
Workflow file too large

See for example https://github.com/traversaro/ros-humble/actions/runs/16347297455 .

Example workflows files:

• https://github.com/traversaro/ros-humble/blob/d285de092481f0b8af262c6be36b1c82628bd72d/.github/workflows/win.yml (~15k lines)
• https://github.com/traversaro/ros-humble/blob/a4c9ecc97e62e9d39ce859fdfaad3472fa873034/.github/workflows/linux.yml (~5k lines)

Example of the first job on Linux (~20 lines):

  stage_0_job_0:
    name: mutex ament-package ament-cmake-core
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
    needs: []
    steps:
    - name: Checkout code
      uses: actions/checkout@v4
    - name: Build ros2-distro-mutex ros-humble-ament-package ros-humble-ament-cmake-core
      env:
        ANACONDA_API_TOKEN: ${{ secrets.ANACONDA_API_TOKEN }}
        CURRENT_RECIPES: ros2-distro-mutex ros-humble-ament-package ros-humble-ament-cmake-core
        BUILD_TARGET: ''
      run: |
        export CI=azure
        export GIT_BRANCH=$BUILD_SOURCEBRANCHNAME
        export FEEDSTOCK_NAME=$(basename ${BUILD_REPOSITORY_NAME})
        .scripts/build_unix.sh --target $BUILD_TARGET

while on Windows (~94 lines)

  stage_0_job_0:
    name: mutex ament-package ament-cmake-core
    runs-on: windows-2022
    strategy:
      fail-fast: false
    needs: []
    env:
      CONDA_BLD_PATH: C:\\bld\\
    steps:
    - name: Checkout code
      uses: actions/checkout@v4
    - name: Setup pixi
      uses: prefix-dev/setup-pixi@v0.8.10
      with:
        pixi-version: v0.40.3
        cache: 'true'
    - uses: egor-tensin/cleanup-path@v4
      with:
        dirs: C:\Program Files\Git\usr\bin;C:\Program Files\Git\bin;C:\Program Files\Git\cmd;C:\Program
          Files\Git\mingw64\bin
    - shell: cmd
      run: |
        set "CI=true"

        :: 4 cores available on GHA: https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners
        :: CPU_COUNT is passed through conda build: https://github.com/conda/conda-build/pull/1149
        set CPU_COUNT=4

        set PYTHONUNBUFFERED=1

        call setup_x64

        :: Set the conda-build working directory to a smaller path
        if "%CONDA_BLD_PATH%" == "" (
            set "CONDA_BLD_PATH=C:\\bld\\"
        )

        :: On azure, there are libcrypto*.dll & libssl*.dll under
        :: C:\\Windows\\System32, which should not be there (no vendor dlls in windows folder).
        :: They would be found before the openssl libs of the conda environment, so we delete them.
        if defined CI (
            DEL C:\\Windows\\System32\\libcrypto-1_1-x64.dll || (Echo Ignoring failure to delete C:\\Windows\\System32\\libcrypto-1_1-x64.dll)
            DEL C:\\Windows\\System32\\libssl-1_1-x64.dll || (Echo Ignoring failure to delete C:\\Windows\\System32\\libssl-1_1-x64.dll)
        )

        :: Make paths like C:\\hostedtoolcache\\windows\\Ruby\\2.5.7\\x64\\bin garbage
        set "PATH=%PATH:ostedtoolcache=%"
      name: conda-forge build setup
    - shell: cmd
      run: |
        setlocal EnableExtensions EnableDelayedExpansion

        set CONDA_BLD_PATH=C:\bld
        echo "PATH is %PATH%"

        rmdir /Q/S C:\Strawberry\
        rmdir /Q/S "C:\Program Files (x86)\Windows Kits\10\Include\10.0.17763.0\"

        set "FEEDSTOCK_ROOT=%cd%"

        mkdir %CONDA_BLD_PATH%

        :: Enable long path names on Windows
        reg add HKLM\SYSTEM\CurrentControlSet\Control\FileSystem /v LongPathsEnabled /t REG_DWORD /d 1 /f

        for %%X in (%CURRENT_RECIPES%) do (
            echo "BUILDING RECIPE %%X"
            cd %FEEDSTOCK_ROOT%\recipes\%%X\
            pixi run -v rattler-build build --recipe %FEEDSTOCK_ROOT%\recipes\%%X\ ^
                -m %FEEDSTOCK_ROOT%\conda_build_config.yaml ^
                -c conda-forge -c robostack-staging ^
                --output-dir %CONDA_BLD_PATH%

            if errorlevel 1 exit 1
            rem -m %FEEDSTOCK_ROOT%\.ci_support\conda_forge_pinnings.yaml
        )

        :: Check if .conda files exist in the win-64 directory
        dir /b "%CONDA_BLD_PATH%\win-64\*.conda" >nul 2>&1
        if errorlevel 1 (
            :: No files found, display warning
            echo Warning: No .conda files found in %CONDA_BLD_PATH%\win-64
            echo This might be due to all the packages being skipped
        ) else (
            :: Files found, run the upload command
            pixi run upload "%CONDA_BLD_PATH%\win-64\*.conda" --force
            if errorlevel 1 exit 1
        )
      env:
        ANACONDA_API_TOKEN: ${{ secrets.ANACONDA_API_TOKEN }}
        CURRENT_RECIPES: ros2-distro-mutex ros-humble-ament-package ros-humble-ament-cmake-core
        PYTHONUNBUFFERED: 1
      name: Build ros2-distro-mutex ros-humble-ament-package ros-humble-ament-cmake-core

[traversaro]

A simple workaround for now is just to increase the number of packages compiled per stage, so we pay less times the "fixed" lines cost for each stage, as we decrease the number of total stages.

[traversaro]

Using a batch_size of 30, the size of the file is reduced to ~3k lines, see https://github.com/traversaro/ros-humble/blob/a4aa2c19f916bb66f1d4b89e211157646f2d5cd6/.github/workflows/win.yml .

[traversaro]

Using a batch_size of 30, the size of the file is reduced to ~3k lines, see https://github.com/traversaro/ros-humble/blob/a4aa2c19f916bb66f1d4b89e211157646f2d5cd6/.github/workflows/win.yml .

With batch_size 10, the file size is ~8k lines, that still works, so probably that is a good tradeoff.