-
Notifications
You must be signed in to change notification settings - Fork 3
emsdk: new recipe #4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
perseoGI
wants to merge
36
commits into
conan-io:main
Choose a base branch
from
perseoGI:pgi/new/emsdk
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
36 commits
Select commit
Hold shift + click to select a range
695ba52
emsdk: added new recipe
perseoGI ac1c0d2
Merge branch 'main' into pgi/new/emsdk
czoido 8f27571
wip
czoido 12820bd
wip
czoido 1768ae6
wip
czoido 8bfa242
wip
czoido 240b479
wip
czoido e01773a
wip
czoido 4c44611
wip
perseoGI e62adda
Removed requirements
perseoGI 334f376
try to fix output
czoido 48823f6
Updated profiles, readme and ci
perseoGI 06b9bd6
Updated CI, build only changed recipes
perseoGI 43ab52f
Updated readme
perseoGI ecb363a
Moved general config to base CI profile
perseoGI fa69614
Fix error
perseoGI d08d241
Fix error
perseoGI 44f11d9
Added user_toolchain example on native profile
perseoGI c080b28
Restore build_folder_vars
perseoGI 81f7789
Added threads comment
perseoGI 6ff3792
Removed unneded min node version
perseoGI c5746ce
Update requirements.txt
czoido 3c475c9
Removed asmjs profile
perseoGI 9736f29
Applied some review suggestions
perseoGI 9ec67da
Removed unneded builddirs environment as already included by Emscript…
perseoGI 073931e
Removed auto install conf in profile
perseoGI 64ef64c
Restored original test_package and created test_example
perseoGI b1fe0cb
Fix ci
perseoGI 6e335fb
Testing publish profiles workflow
perseoGI cbc1e53
WIP
perseoGI a5ed161
WIP
perseoGI c6aae81
Publish profiles
perseoGI dfe8b47
Publish profiles
perseoGI 8dc22f5
Publish profiles
perseoGI 92be15d
Publish profiles
perseoGI 516d065
Removed support for multi versions in CI
perseoGI File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
include(default) | ||
|
||
[conf] | ||
# Verbosity | ||
tools.build:verbosity=verbose | ||
tools.compilation:verbosity=verbose |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,76 @@ | ||
#!/usr/bin/env python3 | ||
import os | ||
import json | ||
from pathlib import Path | ||
import subprocess | ||
import yaml | ||
from collections import defaultdict | ||
|
||
BASE_PATH = Path(__file__).parent | ||
|
||
|
||
def get_updated_recipes(): | ||
updated_paths = os.getenv("BUILD_RECIPES", "").split(" ") | ||
recipe_files = defaultdict(set) | ||
for path in updated_paths: | ||
parts = Path(path).parts | ||
if len(parts) >= 2 and parts[0] == "recipes": | ||
recipe_name = parts[1] | ||
file = "/".join(parts[2:]) | ||
recipe_files[recipe_name].add(file) | ||
return set(recipe_files.keys()) | ||
|
||
|
||
def load_versions(recipe_dir: Path) -> dict: | ||
cfg = recipe_dir / "config.yml" | ||
if not cfg.is_file(): | ||
return {} | ||
return yaml.safe_load(cfg.read_text() or "").get("versions", {}) | ||
|
||
|
||
for recipe_name in get_updated_recipes(): | ||
recipe_dir = Path("recipes") / recipe_name | ||
versions = load_versions(recipe_dir) | ||
|
||
for version, info in versions.items(): | ||
recipe_path = recipe_dir / info.get("folder", ".") | ||
|
||
inspect_json = subprocess.check_output( | ||
["conan", "inspect", str(recipe_path / "conanfile.py"), "--format=json"], | ||
text=True, | ||
) | ||
recipe_name = json.loads(inspect_json)["name"] | ||
|
||
cmd = [ | ||
"conan", | ||
"create", | ||
str(recipe_path), | ||
"--version", | ||
version, | ||
"--build", | ||
"missing", | ||
"-pr:a", | ||
str(BASE_PATH / ".ci_base_profile"), | ||
] | ||
|
||
with open(BASE_PATH / "profiles.json", "r") as f: | ||
profile_list = json.load(f).get(recipe_name) | ||
|
||
print(f"::group::{recipe_name}/{version}", flush=True) | ||
|
||
try: | ||
print("executing:", " ".join(cmd), flush=True) | ||
subprocess.run(cmd, check=True) | ||
for profile in profile_list.get("profiles", []): | ||
example_cmd = [ | ||
"conan", | ||
"test", | ||
str(recipe_path / "test_example"), | ||
f"{recipe_name}/{version}", | ||
"--profile", | ||
profile, | ||
] | ||
print("executing:", " ".join(example_cmd), flush=True) | ||
subprocess.run(example_cmd, check=True) | ||
finally: | ||
print("::endgroup::", flush=True) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
{ | ||
"emsdk": { | ||
"profiles": ["emsdk/wasm32"] | ||
} | ||
} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,51 @@ | ||
name: Test all recipes | ||
|
||
on: | ||
push: | ||
pull_request: | ||
branches: | ||
- main | ||
workflow_dispatch: | ||
|
||
jobs: | ||
create: | ||
name: Create and test Conan recipes | ||
runs-on: ubuntu-latest | ||
steps: | ||
- name: Checkout sources | ||
uses: actions/checkout@v4 | ||
|
||
- name: Set up Python | ||
uses: actions/setup-python@v5 | ||
with: | ||
python-version: "3.11" | ||
|
||
- name: Install requirements | ||
run: | | ||
python -m pip install --upgrade pip | ||
pip install -r requirements.txt | ||
|
||
- name: Cache local Conan repo | ||
uses: actions/cache@v3 | ||
with: | ||
path: ~/.conan2/p | ||
key: ${{ runner.os }}-conan2-${{ hashFiles('recipes/**') }} | ||
restore-keys: | | ||
${{ runner.os }}-conan2- | ||
|
||
- name: Get changed recipes | ||
id: changed-recipes | ||
uses: tj-actions/changed-files@ed68ef82c095e0d48ec87eccea555d944a631a4c # v46 | ||
with: | ||
files: | | ||
recipes/** | ||
files_ignore: recipes/*.{md,txt} | ||
|
||
- name: Build and test recipes | ||
if: steps.changed-recipes.outputs.any_changed == 'true' | ||
env: | ||
BUILD_RECIPES: ${{ steps.changed-recipes.outputs.all_changed_files }} | ||
run: | | ||
conan profile detect | ||
conan config install conan_config | ||
python .github/scripts/build_recipes.py |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,41 @@ | ||
name: Manually publish profile releases | ||
|
||
on: | ||
workflow_dispatch: | ||
inputs: | ||
recipe: | ||
description: 'Recipe to publish profiles for' | ||
required: true | ||
version: | ||
description: 'Version of the packaged profiles' | ||
required: true | ||
|
||
jobs: | ||
create: | ||
name: Create a profile release | ||
runs-on: ubuntu-latest | ||
steps: | ||
- name: Checkout sources | ||
uses: actions/checkout@v4 | ||
|
||
- name: Set variables | ||
id: vars | ||
run: | | ||
echo "RECIPE=${{ github.event.inputs.recipe }}" >> $GITHUB_ENV | ||
echo "VERSION=${{ github.event.inputs.version }}" >> $GITHUB_ENV | ||
echo "ZIP_NAME=${{ github.event.inputs.recipe }}-${{ github.event.inputs.version }}-profiles.zip" >> $GITHUB_ENV | ||
|
||
- name: Zip profile folder | ||
run: | | ||
mkdir -p dist | ||
cd conan_config | ||
zip -r "../dist/$ZIP_NAME" "profiles/$RECIPE" | ||
|
||
- name: Create GitHub release | ||
uses: softprops/action-gh-release@v2 | ||
with: | ||
tag_name: ${{ env.ZIP_NAME }} | ||
name: ${{ env.ZIP_NAME }} | ||
files: dist/${{ env.ZIP_NAME }} | ||
env: | ||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -5,7 +5,6 @@ utilities, for example the Emscripten SDK or other compilers. These recipes are | |
maintained independently of [Conan Center Index](https://github.com/conan-io/conan-center-index) | ||
with flexible maintenance scopes. | ||
|
||
--- | ||
|
||
## 🌟 Why a separate repo? | ||
|
||
|
@@ -18,10 +17,25 @@ tailoring recipes as needed. It also supports bleeding-edge or niche toolchains | |
that don't belong in the main index, offering maximum flexibility and | ||
control. | ||
|
||
--- | ||
## 🧰 Reference Profiles | ||
|
||
This repository includes several **pre-configured Conan profiles** designed to make toolchain setup effortless: | ||
|
||
To install the respective profiles: | ||
|
||
```sh | ||
$ conan config install https://github.com/conan-io/conan-toolchains.git -sf conan_config | ||
``` | ||
|
||
Profiles could be later listed: | ||
```sh | ||
$ conan profile list | ||
``` | ||
Toolchain profiles will be located under `<toolchain_name>/<profile>`. | ||
Profiles starting with dot (`.`) are considered *base* which are used for other profiles and should not be used directly. | ||
|
||
## 🚀 Getting started | ||
|
||
## 🚀 Getting started | ||
|
||
### Setup `conan-toolchains` as a [local recipe index](https://docs.conan.io/2/devops/devops_local_recipes_index.html#devops-local-recipes-index) repository | ||
|
||
|
@@ -38,21 +52,14 @@ This repository is still under active development, and no Conan remote with pre- | |
|
||
## Contributing | ||
|
||
We welcome and appreciate contributions to **conan-toolchains**! | ||
|
||
If you wish to contribute to **conan-toolchains**, follow these steps to clone the repository | ||
and install the required development dependencies. | ||
We recommend using [pre-commit](https://pre-commit.com/) to enforce code style and formatting. To | ||
activate the pre-commit hooks contributors could optionally run the following commands: | ||
|
||
``` | ||
git clone [email protected]:conan-io/conan-toolchains.git | ||
cd conan-toolchains | ||
# Recommended: Setup your virtual environment before installing dependencies | ||
# Setup your virtual environment before installing dependencies | ||
pip install pre-commit | ||
``` | ||
|
||
We use [pre-commit](https://pre-commit.com/) to enforce code style and formatting. To | ||
activate the pre-commit hooks: | ||
|
||
``` | ||
pre-commit install | ||
``` | ||
|
||
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,23 @@ | ||
[settings] | ||
build_type=Release | ||
compiler=emcc | ||
compiler.cppstd=17 | ||
compiler.libcxx=libc++ | ||
# Choose between both types of multithreading support (or none) | ||
# compiler.threads=<posix|wasm_workers> | ||
compiler.version=4.0.10 | ||
os=Emscripten | ||
|
||
[tool_requires] | ||
ninja/[*] | ||
emsdk/4.0.10 | ||
|
||
[conf] | ||
tools.build:exelinkflags+=['-sALLOW_MEMORY_GROWTH=1'] | ||
tools.build:sharedlinkflags+=['-sALLOW_MEMORY_GROWTH=1'] | ||
|
||
# Set Ninja as default generator as it will avoid Windows issues | ||
tools.cmake.cmaketoolchain:generator=Ninja | ||
|
||
# Distinguish between architectures | ||
tools.cmake.cmake_layout:build_folder_vars=['settings.build_type', 'settings.arch'] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,126 @@ | ||
# EMSDK - Emscripten Compiler Profiles for Conan | ||
|
||
This repository provides a collection of pre-built Conan profiles that can be | ||
used as a reference for **cross-compiling projects to WebAssembly (WASM)**. | ||
|
||
📘 For detailed information and usage examples, refer to the official [Conan | ||
documentation](https://docs.conan.io/2/examples/cross_build/emscripten.html#setting-up-conan-profile-for-webassembly-wasm). | ||
|
||
💡 Issues and suggestions are welcome via the `conan-toolchains` repository. | ||
Contributions are encouraged! | ||
|
||
|
||
## 🧭 Introduction | ||
|
||
As of **Conan 2.18**, the `emcc` compiler (from | ||
[Emscripten](https://emscripten.org/docs/)) is natively supported. This allows | ||
accurate modeling of the compiler’s built-in features. | ||
|
||
The current `emcc` compiler model includes the following settings: | ||
|
||
``` | ||
emcc: | ||
version: [ANY] | ||
libcxx: [null, libstdc++, libstdc++11, libc++] | ||
threads: [null, posix, wasm_workers] | ||
cppstd: [null, 98, gnu98, 11, gnu11, 14, gnu14, 17, gnu17, 20, gnu20, 23, gnu23, 26, gnu26] | ||
cstd: [null, 99, gnu99, 11, gnu11, 17, gnu17, 23, gnu23] | ||
``` | ||
|
||
> ℹ️ Note: `emcc` is a front-end for the `clang` compiler, so it shares many of the same settings and options. | ||
|
||
However, there are a few **important caveats** to consider when using Emscripten with Conan. | ||
|
||
|
||
## ⚠️ Caveats | ||
|
||
### 🔄 ABI Incompatibility | ||
|
||
There is **no ABI compatibility guarantee** between different `emcc` versions. | ||
Refer to the [Emscripten Changelog](https://github.com/emscripten-core/emscripten/blob/main/ChangeLog.md) for details. | ||
|
||
In Conan v2, the `compiler.version` setting mostly acts as **syntactic sugar**. While it doesn't influence compilation directly, it: | ||
|
||
- Enables Conan to apply version-specific flags, compile and link flags. | ||
- Ensures distinct **package IDs** for different compiler versions. | ||
|
||
To prevent ABI issues, make sure your **Conan profile's `compiler.version`** | ||
matches the **actual `emsdk` version** used. This ensures consistent builds and | ||
automatic recompilation when updating `emsdk`. | ||
|
||
|
||
|
||
### 🧵 Multithreading | ||
|
||
The `emcc` compiler can produce incompatible binaries depending on whether threading is enabled. | ||
|
||
Refer to the Emscripten documentation on [pthreads](https://emscripten.org/docs/porting/pthreads.html#compiling-with-pthreads-enabled) | ||
and [wasm workers](https://emscripten.org/docs/api_reference/wasm_workers.html). | ||
|
||
Emscripten supports two threading models: | ||
|
||
- **POSIX Threads (Pthreads)** | ||
- **Wasm Workers** | ||
|
||
These are **mutually incompatible**, and this distinction must be reflected in the Conan profile to avoid mixing binaries. | ||
|
||
Enable threading in your profile by setting: | ||
|
||
```ini | ||
compiler.threads=posix # For Pthreads | ||
compiler.threads=wasm_workers # For Wasm Workers | ||
``` | ||
|
||
Conan will automatically inject the necessary linker flags. | ||
|
||
## 📂 Profile Overview | ||
|
||
This repository includes the following profiles: | ||
- `wasm32`: WebAssembly 32-bit target (default). | ||
- `wasm64`: Experimental 64-bit WebAssembly target (for projects needing >4 GB dynamic memory). | ||
- `local`: Use your locally installed emsdk instead of the Conan-managed one. | ||
|
||
|
||
⚠️ WASM64 caveats: | ||
|
||
The latest node version `emsdk/4.0.10` installs is the `node/22.16.0`, which can not run directly wasm64 binaries. | ||
Also, it is not valid to compile wasm64 with `-sMIN_NODE_VERSION=221600`, see following error: | ||
``` | ||
em++: warning: MIN_NODE_VERSION=221600 is not compatible with MEMORY64 (230000 or above required) [-Wcompatibility] | ||
``` | ||
Thus, to be able to run an `wasm64` binary you will need to download at least `node/23` manually in your system. | ||
|
||
🛠️ For the local profile: | ||
- Ensure `emcc`, `em++`, `emar`, etc. are available in your `PATH`. | ||
- Check the `[buildenv]` section in the profile. | ||
- Optionally, provide your own `Emscripten.cmake` via the `user_toolchains` setting. | ||
- Define the arch setting: `wasm`, `wasm64`, or the mostly deprecated `asm.js`. | ||
|
||
|
||
## ▶️ Usage | ||
|
||
After installing the profiles (TBD), you can build your project like this: | ||
|
||
``` | ||
$ conan build <path> -pr emsdk/wasm32 | ||
``` | ||
|
||
### 🧠 Dynamic Memory Allocation | ||
|
||
By default, WebAssembly does not allow dynamic memory growth. To enable it, you must set the following linker flag: | ||
|
||
`-s ALLOW_MEMORY_GROWTH=1` | ||
|
||
Our base profiles enable this flag by default to simplify usage. | ||
|
||
If you want to disable memory growth, simply remove or comment out the relevant line in the profile. | ||
|
||
🔎 The Conan docs explain the dynamic memory limits for each architecture. | ||
These are also preconfigured in the `wasm32` and `wasm64` profiles and can be | ||
customized as needed, including the `INITIAL_MEMORY` setting. | ||
|
||
|
||
### 🙌 Contribute | ||
|
||
Feel free to open issues or pull requests in the `conan-toolchains` repository. | ||
Contributions to extend or improve these profiles are welcome! |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Each "toolchain" should have a readme with tailored instructions - as I would expect this grow in the future.
I would refer to them as "reference" profiles - the ones we test, leaving it up to users to customise before installing. e.g. the "native" one still requires manual configuration, so I'm not sure that's effortless, there's still some use intervention needed in some instances.
we may consider having the profile .zip files as releases - so that a user can do
conan config install https://github.com/xxxx/emsdk_profiles_1.0.zip
or something like that - so that they get only what they needThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In this line, another approach would be to have the profiles in a specific folder, under the
emsdk
folder. Let the users manually copy them, or if not, they can do aconan config install --source-folder
to specify the specific folder. But better that than a singleconan config install
that will bring different profiles foremcc
,cuda
, etc, etc, of different unrelated things that users will not want in generalThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That is another good approach, right now, what I've done in the CI is the ability to publish recipe specific profiles manually with a github action, check out my fork releases: https://github.com/perseoGI/conan-toolchains/releases
This can be installed with:
Allowing profile isolation with different recipes, and also, keeping a profile versioning.
Do you like this approach?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seem a bit overkill to me, in terms of extra effort to publish and release the profiles, version them, etc.
One of the reasons of using the
conan-toolchains
repo is that it is not necessary to provide the same backward compatibility and previous versions support, and it is possible a more "live at head" approach. So such a strong effort towards versioning profiles seems very overkill to me.