Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Emscripten-wasm32 target #268

Open
wants to merge 108 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
108 commits
Select commit Hold shift + click to select a range
4324cf0
Add changes required for emscripten-wasm32 target
Tobias-Fischer Jan 12, 2025
0b2e99e
Fixup vinca branch
Tobias-Fischer Jan 14, 2025
e70d650
Update yamls with CEP0014
nmarticorena Jan 15, 2025
e787543
Add emscripten enviroment
nmarticorena Jan 15, 2025
76502a0
fix deprecated depends_on for depends-on
nmarticorena Jan 15, 2025
9869851
Update yamls
nmarticorena Jan 16, 2025
6f76936
Change chanels priority and use local vinca version
nmarticorena Jan 16, 2025
ebd12b3
Add emscripten-forge channel
nmarticorena Jan 17, 2025
d86532c
Some fixes for new emscripten
Tobias-Fischer Jan 17, 2025
fb960f9
Update patch
Tobias-Fischer Jan 17, 2025
a65d7ed
gmock fix
Tobias-Fischer Jan 17, 2025
02d6c8d
Some more packages
Tobias-Fischer Jan 17, 2025
a462988
Fix build of additional recipes
Tobias-Fischer Jan 18, 2025
ba0e48c
Sync spdlog version
Tobias-Fischer Jan 18, 2025
c316da8
Dirty hack to find correct Python_NumPy_INCLUDE_DIRS
Tobias-Fischer Jan 18, 2025
93e4311
We need custom osx-arm64 ros-humble for the generators
Tobias-Fischer Jan 18, 2025
b06e735
Add missing run dep
Tobias-Fischer Jan 18, 2025
d6ac51f
Remove outdated comment
Tobias-Fischer Jan 18, 2025
4a508ad
Successfully built all packages
Tobias-Fischer Jan 19, 2025
366994a
Cleanup
Tobias-Fischer Jan 20, 2025
e6323c7
Add required patches
Tobias-Fischer Jan 20, 2025
6465b62
Add missing build.sh
Tobias-Fischer Jan 28, 2025
2a5d879
add gha for emscripten-wasm32 target
nmarticorena Jan 28, 2025
d21bc95
Merge branch 'main' into emscripten
nmarticorena Jan 28, 2025
2ea93e4
fix typo with pixi version
nmarticorena Jan 28, 2025
0bd00b7
Add missing build.sh
Tobias-Fischer Jan 28, 2025
777bed8
Add emscripten build on gha
nmarticorena Jan 29, 2025
77784ae
Attempt to build emscripten skipping tests
nmarticorena Jan 30, 2025
eb2950d
Remove skip existing because we are not using that channel
nmarticorena Jan 30, 2025
671cedf
remove my channel and use changes now available on emscripten-forge
nmarticorena Feb 5, 2025
65d83b8
Nasty trick to pin down the latest build of the cross-python_emscript…
nmarticorena Feb 5, 2025
5baabc4
Remove personal channel
nmarticorena Feb 5, 2025
8bcc223
Needed to re add personal channel, but after I fully cleaned, this is…
nmarticorena Feb 5, 2025
48f52dd
attempt to build osrf-testing-tools-cpp
nmarticorena Feb 7, 2025
592da0a
Change version of rmw to use extra fixes from mbatc
nmarticorena Feb 20, 2025
86fe7b8
full rebuild
nmarticorena Feb 20, 2025
4660cc5
Add mbatc changes as a patch file
nmarticorena Feb 21, 2025
4df36d9
Fix aarch64
Tobias-Fischer Feb 23, 2025
e44bea3
add roslibJS wasm public
nmarticorena Feb 23, 2025
f62d4b6
Remove fastrtps from emscripten
Tobias-Fischer Feb 24, 2025
19460ba
Try fix: Failed to load dynlib /lib/libmemory_tools.so Error: WebAsse…
Tobias-Fischer Feb 24, 2025
b58fbb2
Remove unneeded part of patch
Tobias-Fischer Feb 24, 2025
64353f4
Use Tobi's fork of rmw_wasm
Tobias-Fischer Feb 24, 2025
22d6a30
Rebuild with -pthread in vinca
Tobias-Fischer Feb 25, 2025
e1de673
Use emscripten-async-fixes
Tobias-Fischer Feb 25, 2025
b6514be
Update build.sh
Tobias-Fischer Feb 25, 2025
cd19d28
Update build.sh
Tobias-Fischer Feb 25, 2025
ee91b42
Delete additional_recipes/ros-humble-wasm-cpp/fix-build.patch
Tobias-Fischer Feb 25, 2025
fcc98fb
Delete additional_recipes/ros-humble-rmw-wasm-cpp/fix-build.patch
Tobias-Fischer Feb 25, 2025
4552557
Update build.sh
Tobias-Fischer Feb 25, 2025
f0ea61e
Update build.sh
Tobias-Fischer Feb 25, 2025
a3810f0
[ci skip] -mbulk-memory -matomics
Tobias-Fischer Feb 25, 2025
f9cae19
-mbulk-memory -matomics
Tobias-Fischer Feb 25, 2025
9064495
[skip ci] go back to working version
Tobias-Fischer Feb 25, 2025
cb0b830
Go back to working version
Tobias-Fischer Feb 25, 2025
e720753
--batch_size 1 for test
Tobias-Fischer Feb 25, 2025
c134620
Trigger
Tobias-Fischer Feb 25, 2025
2107f6e
Update ros-humble-rclpy.patch
Tobias-Fischer Feb 26, 2025
c732873
Test fix duplicate Python
Tobias-Fischer Feb 26, 2025
f5ee8a5
Update ros-humble-rclpy.patch
Tobias-Fischer Feb 27, 2025
a93324b
Update ros-humble-rclpy.patch
Tobias-Fischer Feb 27, 2025
37a0fd7
Test - seems to work locally
Tobias-Fischer Feb 27, 2025
4d600e5
Remove NO_EXTRAS
nmarticorena Feb 28, 2025
64de39e
Recover NO_EXTRAS to try just with assert 2
nmarticorena Feb 28, 2025
494fb36
Bump version, running with assertions=1
nmarticorena Mar 2, 2025
f7da24e
Bump version, running with demangle support=1
nmarticorena Mar 2, 2025
4185bd7
Bump version, increase stack size by 10
nmarticorena Mar 3, 2025
8ba60bc
Bump version
nmarticorena Mar 3, 2025
a5b4e8e
Static linking changes by Tobi; also bump versions to 12
nmarticorena Mar 7, 2025
9b55112
Update fix-build.patch
Tobias-Fischer Mar 8, 2025
a28e279
Update build.sh
Tobias-Fischer Mar 8, 2025
4fe23af
Update build.sh
Tobias-Fischer Mar 8, 2025
4a1d47b
Update build.sh
Tobias-Fischer Mar 8, 2025
19b5ae9
Merge main
Tobias-Fischer Mar 8, 2025
740c112
Start reducing diff to main branch
Tobias-Fischer Mar 8, 2025
2af19ee
Remove outdated files
Tobias-Fischer Mar 8, 2025
ba9470a
Add additional run reqs
Tobias-Fischer Mar 11, 2025
fc20a37
Simplify dependencies
Tobias-Fischer Mar 11, 2025
b0adeb1
Remove unnecessary/unused patches
nmarticorena Mar 12, 2025
62f1a91
Clean up dynmsg
nmarticorena Mar 12, 2025
b78b6f0
remove emscripten env and replaced it by build-emscripten task
nmarticorena Mar 12, 2025
3ba9ee2
Clean the workflows to match as close the original
nmarticorena Mar 12, 2025
5b0d9cf
missing task
nmarticorena Mar 12, 2025
fbd6840
Remove personal channel used for testing
nmarticorena Mar 12, 2025
17606cb
remove duplicated vinca
nmarticorena Mar 12, 2025
fba1e07
Fixup spdlog version
Tobias-Fischer Mar 12, 2025
9fcd7be
Fixup dynmsg version to match upstream, use specific commit
Tobias-Fischer Mar 12, 2025
d643861
Remove deprecated emscripten env
Tobias-Fischer Mar 12, 2025
397fdf7
rename dynmsg->ros-humble-dynmsg
Tobias-Fischer Mar 12, 2025
805cc34
Remove auto-generated message
Tobias-Fischer Mar 12, 2025
820aa9a
fix issue where --target was deleted
nmarticorena Mar 13, 2025
2398c77
Rename ros-humble-examples-rclcpp-minimal-publisher.patch to ros-humb…
Tobias-Fischer Mar 13, 2025
df250cd
Rename ros-humble-examples-rclcpp-minimal-subscriber.patch to ros-hum…
Tobias-Fischer Mar 13, 2025
b7d762f
Rename ros-humble-rclcpp-components.patch to ros-humble-rclcpp-compon…
Tobias-Fischer Mar 13, 2025
00da082
Rename ros-humble-rclpy.patch to ros-humble-rclpy.emscripten.patch
Tobias-Fischer Mar 13, 2025
6f22d14
Update fix-build.patch
Tobias-Fischer Mar 13, 2025
cb9f5b3
Fix malformed patch
nmarticorena Mar 13, 2025
c1010a8
All libs are needed
Tobias-Fischer Mar 13, 2025
0501eef
Fix closing )
nmarticorena Mar 13, 2025
2051609
New vinca with emscripten support
Tobias-Fischer Mar 13, 2025
e64b276
Fix up
Tobias-Fischer Mar 13, 2025
1388783
Update main.yml
Tobias-Fischer Mar 13, 2025
20ad6f1
Do not specify pixi version
Tobias-Fischer Mar 13, 2025
be87da9
Fix dynmsg version
Tobias-Fischer Mar 13, 2025
d68926e
Fix version, only build for emscripten
Tobias-Fischer Mar 13, 2025
977fed3
Only build for wasm
Tobias-Fischer Mar 13, 2025
3d7af61
Only build for wasm
Tobias-Fischer Mar 13, 2025
d42616e
Only build for wasm
Tobias-Fischer Mar 13, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 28 additions & 1 deletion .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@ jobs:
- name: Install vinca
run: |
pip install git+https://github.com/RoboStack/vinca.git

- name: Generate recipes for linux-64
run: |
git clean -fdx
Expand Down Expand Up @@ -148,6 +147,34 @@ jobs:
env:
INPUT_GITHUB_TOKEN: ${{ secrets.GHA_PAT }}

# Build emscripten-wasm32
- name: Reset repo
run: |
git reset --hard origin/main
- name: Generate recipes for emscripten-wasm32
run: |
git clean -fdx
cp vinca_emscripten32.yaml vinca.yaml
vinca --multiple --platform emscripten-wasm32
sh -c 'find additional_recipes/* -maxdepth 0 -type d -exec ln -s ../{} recipes/ \;'

- name: Generate azure pipelines for emscripten-wasm32
run: |
vinca-gha --platform emscripten-wasm32 --trigger-branch buildbranch_emscripten -d ./recipes --additional-recipes --batch_size 1
- name: Commit files for emscripten-wasm32
run: |
if [[ -f "emscripten_wasm32.yml" ]]; then
mv emscripten_wasm32.yml .github/workflows/
git config --local user.email "[email protected]"
git config --local user.name "GitHub Action"
git add . -f
git commit -m "Add build files `date '+%Y-%m-%d-%H%M'`" -a
remote_repo="https://${GITHUB_ACTOR}:${INPUT_GITHUB_TOKEN}@github.com/${GITHUB_REPOSITORY}.git"
git push "${remote_repo}" HEAD:buildbranch_emscripten --follow-tags --force
fi
env:
INPUT_GITHUB_TOKEN: ${{ secrets.GHA_PAT }}

on:
push:
branches:
Expand Down
18 changes: 18 additions & 0 deletions .github/workflows/testpr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@ jobs:
platform: osx-arm64
- os: windows-2019
platform: win-64
- os: ubuntu-latest
platform: emscripten-wasm32

runs-on: ${{ matrix.os }}

Expand Down Expand Up @@ -64,6 +66,8 @@ jobs:
set +e
git diff --exit-code --name-only origin/main -- vinca_linux_64.yaml > /dev/null
echo "::set-output name=LINUX_YAML_CHANGED::${?}"
git diff --exit-code --name-only origin/main -- vinca_emscripten32.yaml > /dev/null
echo "::set-output name=EMSCRIPTEN32_YAML_CHANGED::${?}"
git diff --exit-code --name-only origin/main -- vinca_linux_aarch64.yaml > /dev/null
echo "::set-output name=LINUX_AARCH_YAML_CHANGED::${?}"
git diff --exit-code --name-only origin/main -- vinca_osx.yaml > /dev/null
Expand All @@ -80,6 +84,14 @@ jobs:
mkdir -p recipes
$HOME/.pixi/bin/pixi run -e beta -v vinca --platform linux-64 -m
ls -la recipes
- name: Generate recipes for emscripten-wasm32
shell: bash -l {0}
if: steps.filecheck.outputs.EMSCRIPTEN32_YAML_CHANGED == 1 && matrix.platform == 'emscripten-wasm32'
run: |
cp vinca_emscripten32.yaml vinca.yaml
mkdir -p recipes
$HOME/.pixi/bin/pixi run -e beta -v vinca --platform emscripten-wasm32 -m -n
ls -la recipes
- name: Generate recipes for linux-aarch64
shell: bash -l {0}
if: steps.filecheck.outputs.LINUX_AARCH_YAML_CHANGED == 1 && matrix.platform == 'linux-aarch64'
Expand Down Expand Up @@ -129,6 +141,12 @@ jobs:
run: |
env -i $HOME/.pixi/bin/pixi run -e beta rattler-build build --recipe-dir additional_recipes --target-platform linux-64 -m ./conda_build_config.yaml -c robostack-staging -c conda-forge --skip-existing
env -i $HOME/.pixi/bin/pixi run -e beta rattler-build build --recipe-dir recipes --target-platform linux-64 -m ./conda_build_config.yaml -c robostack-staging -c conda-forge --skip-existing
- name: Build recipes for emscripten-wasm32
shell: bash -l {0}
if: steps.newrecipecheck.outputs.RECIPE_CREATED == 1 && steps.filecheck.outputs.EMSCRIPTEN32_YAML_CHANGED == 1 && matrix.platform == 'emscripten-wasm32'
run: |
env -i $HOME/.pixi/bin/pixi run -e beta sh -c 'find additional_recipes/* -maxdepth 0 -type d -exec ln -s ../{} recipes/ \;'
env -i $HOME/.pixi/bin/pixi run -e beta rattler-build build --recipe-dir ./recipes --target-platform emscripten-wasm32 -m ./conda_build_config.yaml -c https://repo.mamba.pm/emscripten-forge/ -c robostack-staging -c conda-forge --skip-existing --test skip
- name: Build recipes for linux-aarch64
shell: bash -l {0}
if: steps.newrecipecheck.outputs.RECIPE_CREATED == 1 && steps.filecheck.outputs.LINUX_AARCH_YAML_CHANGED == 1 && matrix.platform == 'linux-aarch64'
Expand Down
28 changes: 0 additions & 28 deletions .scripts/build_linux.sh

This file was deleted.

25 changes: 0 additions & 25 deletions .scripts/build_osx.sh

This file was deleted.

25 changes: 0 additions & 25 deletions .scripts/build_osx_arm64.sh

This file was deleted.

50 changes: 50 additions & 0 deletions .scripts/build_unix.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
#!/usr/bin/env bash

# Default values
target=""

# Parse arguments
while [[ "$#" -gt 0 ]]; do
case $1 in
--target) target="$2"; shift ;; # Capture value for --target
*) echo "Unknown parameter passed: $1"; exit 1 ;; # Handle invalid options
esac
shift
done

set -xeuo pipefail
export PYTHONUNBUFFERED=1

export FEEDSTOCK_ROOT=`pwd`
export "CONDA_BLD_PATH=$HOME/conda-bld/"

curl -fsSL https://pixi.sh/install.sh | bash
export PATH="$HOME/.pixi/bin:$PATH"

if [[ "$target" == *"osx"* ]]; then
echo "osx"
export PATH=$(echo $PATH | tr ":" "\n" | grep -v 'homebrew' | xargs | tr ' ' ':')
fi

if [[ "$target" == "emscripten-wasm32" ]]; then
extra_channel="-c https://repo.mamba.pm/emscripten-forge"
cross_compile="--target-platform emscripten-wasm32 --test skip"

else
extra_channel=""
cross_compile=""
fi

for recipe in ${CURRENT_RECIPES[@]}; do
pixi run -v rattler-build build \
--recipe ${FEEDSTOCK_ROOT}/recipes/${recipe} \
-m ${FEEDSTOCK_ROOT}/conda_build_config.yaml \
-c robostack-staging -c conda-forge \
${extra_channel} \
--output-dir $CONDA_BLD_PATH \
${cross_compile}
# -m ${FEEDSTOCK_ROOT}/.ci_support/conda_forge_pinnings.yaml \

done

pixi run upload ${CONDA_BLD_PATH}/${target}*/*.conda --force
Loading
Loading