From c47dfceb1fa6a78d8e32331cec275961881d6de9 Mon Sep 17 00:00:00 2001 From: Adrian Lundell Date: Fri, 23 Oct 2026 09:18:18 +0200 Subject: [PATCH] Update arm .wiki information Remove out of date information, clarify, and remove some content that will be put in a new /arm skill Signed-off-by: Adrian Lundell Change-Id: Ic742036a8ea0364f63439cfb745919c37ae7eb99 --- .wiki/backends/arm/known-issues.md | 265 ------------------------ .wiki/backends/arm/overview.md | 320 +++++------------------------ .wiki/index.md | 9 +- .wiki/rules/arm-backend.md | 124 +++-------- 4 files changed, 87 insertions(+), 631 deletions(-) delete mode 100644 .wiki/backends/arm/known-issues.md diff --git a/.wiki/backends/arm/known-issues.md b/.wiki/backends/arm/known-issues.md deleted file mode 100644 index bcaa8b77f1b..00000000000 --- a/.wiki/backends/arm/known-issues.md +++ /dev/null @@ -1,265 +0,0 @@ ---- -title: Arm Backend Known Issues -category: DEBUGGING -backends: [Arm] -last_validated: 2026-04-15 -source_issues: [1004, 1110, 1161, 1163, 1230, 11913, 11999, 12237, 10899, 12270, 12959, 12991, 13022, 13399, 13557, 13842, 13901, 15805, 15870, 16090, 16225, 16374, 16426, 16541, 16629, 16739, 16779, 16784, 16864, 16899, 16902, 17241, 17397, 17437, 17489, 17667, 17668, 17753, 17902, 18306, 18319, 18491, 18500, 18873] ---- - -# Arm Backend Known Issues - -## Submodule / Setup Issues - -### git.mlplatform.org SSL and availability - -The Arm backend's submodule (`ethos-u-core-driver`) is hosted on `git.mlplatform.org` which has recurring issues (note: `serialization_lib` has been removed from the repo): - -- **SSL certificate verification failures** — `fatal: unable to access ... server certificate verification failed` -- **HTTP 500 errors** — server outages -- These failures block ALL submodule init, not just Arm submodules [Source: #1004, #1163] - -**Fix:** Remove the Arm submodule if not using the Arm backend: -```bash -git submodule deinit backends/arm/third-party/ethos-u-core-driver/ -``` -Or disable SSL verification (not recommended): `git config --global http.sslVerify "false"` [Source: #1004] - -### install_executorch.sh failures on macOS - -Build failures during pip wheel build on macOS may be caused by CMake version conflicts. Some users report that downgrading CMake to 3.25, re-running the install script (which then upgrades CMake), resolves the issue. This is likely a caching/state issue. [Source: #10151] - -**Best fix:** Use a clean environment and v0.6+. [Source: #10151] - -## Operator / Compilation Issues - -### Dynamic shapes not supported - -The Arm backend cannot handle models with dynamic shapes. `SymFloat` or `SymInt` objects in the graph cause assertion failures in `get_first_fake_tensor()`. - -``` -AssertionError: Found zuf38 in meta["val"] of _local_scalar_dense_2, expected to find FakeTensor -``` -or: -``` -TypeError: Expected a FakeTensor ... but got SymFloat -``` - -**Workaround:** Fix all input shapes at export time. For YOLO models, remove the dynamic anchor generation. [Source: #12237] - -### Attribute mutation during export - -Models that mutate attributes (like YOLO's `self.anchors`) fail with strict export: -``` -AssertionError: Mutating module attribute anchors during export. -``` - -**Fix:** Use `strict=False` in `torch.export.export_for_training()`. [Source: #12237] - -### NHWC memory format conversion - -TOSA requires channel-last (NHWC) format. The `Permute_Memory_Format_Pass` handles this, but was historically WIP with incomplete shape updates for neighbor operators. [Source: #1110] - -### Vela compiler internal errors - -Early versions had issues with Vela rejecting TOSA output: -- `AttributeError: 'ReshapeAttribute' object has no attribute 'NewshapeAsNumpy'` — case sensitivity bug in Vela -- Linear layers could fail until the TOSA-to-Vela mapping was revised [Source: #1161] - -### Missing quantized op kernels - -Running quantized models without delegation requires linking the quantized op library: -``` -RuntimeError: Missing out variants: {'quantized_decomposed::dequantize_per_tensor', ...} -``` - -**Fix:** Build and link `quantized_ops_lib`. Performance without NPU delegation will be poor. [Source: #1161] - -## Build Issues - -### c10/macros/cmake_macros.h not found - -When building backends as separate CMake projects (e.g., MediaTek LLaMA runner), you may see: -``` -fatal error: 'c10/macros/cmake_macros.h' file not found -``` - -**Fix:** Define `C10_USING_CUSTOM_GENERATED_MACROS` in the CMakeLists.txt. This is needed whenever a separate CMake project sets up ExecuTorch include paths directly rather than using the `executorch_core` target's public compile definitions. [Source: #11999] - -### Selective build for baremetal - -`libportable_kernels` for Arm baremetal may not include selective build by default. Use CMake flags to enable: -```bash --DEXECUTORCH_SELECT_OPS_FROM_MODEL=".pte" --DEXECUTORCH_DTYPE_SELECTIVE_BUILD=ON -``` -[Source: #11913] - -## Performance Profiling - -### Vela estimator vs FVP profiling - -The Vela compiler includes a performance estimator, but its estimates can differ significantly from actual FVP (Fixed Virtual Platform) profiling results. Always validate performance on FVP or real hardware. [Source: #18319] - -### Non-delegated performance - -Running quantized models on Cortex-M CPU without Ethos-U delegation has "tragic" performance (as noted by core team). Always use delegation for production workloads. [Source: #1161] - -## Preserved Ops API - -Cadence and Arm backends need `to_edge_with_preserved_ops` (experimental) to prevent decomposition of ops like `aten.rms_norm`. This API is being promoted to official status: -- `preserve_ops` will be added to `EdgeCompileConfig` -- View/mutation ops can be preserved if consumed by a delegate backend -- View/mutation ops should NOT be preserved if they remain in the portable graph [Source: #12306] - -## Quantizer Issues - -### Observer sharing bug at Conv-ReLU + residual junctions - -The Arm Ethos quantizer incorrectly shares observers across `add`, `permute`, `relu` at residual connections. This causes quantization errors in models with skip connections (e.g., ResNet, MobileNet). Root cause: `quantization_annotator.py` doesn't properly handle shared quantization specs at add nodes. [Source: #12959] - -### SharedQuantizationSpec infinite recursion - -Using `SharedQuantizationSpec` with certain topologies (e.g., `minimum → eq` chains) causes `RecursionError`. Fixed upstream in pytorch/ao#3011. [Source: #13842] - -### LeakyReLU fails with device mismatch - -ARM quantizers (VGF, Ethos-U) fail on `nn.LeakyReLU` because the `negative_slope` constant gets placed on wrong device. XNNPACK quantizer doesn't have this. Root cause: kwargs removal in `quantization_annotator.py`. [Source: #16541] - -### ReLU(inplace=True) with 16-bit activation - -`ReLU(inplace=True)` with `a16w8` quantization config fails at `to_edge_transform_and_lower` with `Expected tensor aten_convolution_default in aten.clamp`. Fixed on main branch. [Source: #16629] - -### FuseQuantizedActivationPass INT16 failure - -`FuseQuantizedActivationPass` does not handle INT16 symmetric quantization correctly in some cases. [Source: #17437] - -### aot_arm_compiler.py Conv2d quantization failure - -`aot_arm_compiler.py` may not quantize `Conv2d` for `cortex-m55+int8` target in certain configurations. [Source: #17902] - -### Name filter doesn't match nodes correctly - -`arm_quantizer.py`'s `module_name_filter` assumes names start with `"L['self']."`, which may not be present. Fixed on main. [Source: #15870] - -### GroupNorm decomposition failure - -`DecomposeGroupNormPass(ArmPass)` fails when running `prepare_pt2e` on models with `torch.nn.GroupNorm`. May be related to dynamic shape handling. [Source: #16090] - -## Vela Compiler Issues - -### Custom config file crashes with trailing spaces - -Custom `[System_Config.*]` sections crash Vela with `IndexError` if config lines have trailing spaces. Fixed in Vela 4.5.0. [Source: #15805] - -### `--optimise Size` produces incorrect results - -Vela with `--optimise Size` flag can produce different (wrong) results compared to default optimization. [Source: #16864] - -### reduce_mean not fully delegated - -Operator support checks for views/reshapes are overly pessimistic — they reject view nodes with axis-product > 65536 even when no transpose is needed. This prevents full delegation of reduce_mean to the NPU. [Source: #16779] - -### Vela internal errors on certain models - -Vela may crash internally on certain model structures. The Vela team is actively investigating. [Source: #13022] - -## Delegation Issues - -### conv→relu→permute→reshape(5D) crashes partitioner - -This specific graph pattern crashes during `to_edge_transform_and_lower` for Ethos-U. [Source: #16739] - -### PReLU unsupported on Ethos-U - -`torch.nn.PReLU` decomposes to `torch.where(x>0, x, weights*x)` which isn't supported by the Ethos-U backend. No workaround exists. [Source: #16902] - -### BatchNorm2d without preceding Conv not delegated - -Standalone `BatchNorm2d` (not fused with Conv) fails Ethos-U delegation, though it works in TFLite→Vela flow. Workaround: manually decompose to `mul + add`. [Source: #17241, #17397] - -### GRU / RNN layers not supported - -GRU decomposition fails during Ethos-U lowering. LSTM support via CMSIS-NN is planned but not yet implemented. [Source: #12270, #17753] - -### RewriteConvPass crashes on non-fuseable conv→relu branches - -**Symptom**: -``` -ValueError: RewriteConvPass: No output quantization parameter found in node tosa_conv2d_default -original_aten=aten.convolution.default -``` -Occurs during `to_edge_transform_and_lower` when a delegated `conv → relu/clamp` branch has an activation whose output quantization has `zero_point != qmin` (non-fuseable). [Source: #18491] - -**Root Cause**: `FoldAndAnnotateQParamsPass` places `output_qparams` on the downstream `clamp` node rather than the `conv` node in the non-fuseable case. `RewriteConvPass` unconditionally calls `get_output_qparams(conv)` which crashes because the conv doesn't own its output quantization. - -**Fix**: Fixed by PR #18778. The fix makes `RewriteConvPass` check for `output_qparams` on successor activation nodes when the conv itself has no output qparams. [Source: #18491] - -### Quantized sigmoid TABLE generation bug with qmin=-127 - -**Symptom**: Quantized `aten.sigmoid.default` produces incorrect outputs when lowered to TOSA TABLE with `qmin=-127, qmax=127, dtype=torch.int8`. The generated 256-entry LUT has duplicate entries and off-by-one shifts. [Source: #18873] - -**Root Cause**: `InsertTableOpsPass.generate_8bit_table_values()` uses `torch.linspace(start=-127, end=127, steps=256, dtype=torch.int8)` which cannot produce 256 distinct values in a 255-code range, causing code `0` to be duplicated. - -**Status**: Open issue. The fix should use the full int8 domain `[-128, 127]` as table input regardless of `qmin/qmax`, or use explicit integer range instead of `torch.linspace`. [Source: #18873] - -### ConvTranspose2d fallback failure - -`ConvTranspose2d` fails to fall back to CPU when it can't run on the NPU, producing "Non-passthrough operation could not run on NPU" error. [Source: #17668] - -### Ethos-U base_addr mismatch - -The Ethos-U backend may use `base_addr` values that don't match ExecuTorch's planned memory pool, causing output buffers to remain unchanged on real hardware despite reported successful execution. Works on FVP but fails on real MCUs. [Source: #16784] - -## Performance Issues - -### Softmax decomposition slow on NPU - -Softmax decomposition uses `aten::amax` which runs on the elementwise engine (not MACs). The Vela performance estimator is unreliable for cycle counts — always validate on FVP or real hardware. [Source: #18319] - -### LayerNorm quantization accuracy - -LayerNorm quantization is sensitive to epsilon values. For transformer models (DeiT-tiny, etc.), accuracy drops in TOSA quantized pipeline may be caused by epsilon sensitivity. Use `--stable_softmax` flag for numerically stable algorithm. [Source: #16426, #18306, #18316] - -### amax support added for U55 - -`amax` op support was added for Ethos-U55 (via Vela update). To use it, set `ArmPassPipelineConfig` in compile spec with `stable_softmax=True`. [Source: #17211] - -## Setup / Build Issues - -### Dependency conflicts in setup.sh - -`examples/arm/setup.sh` has known dependency conflicts between ethos-u-vela (flatbuffers==24.12.23) and tosa-tools (flatbuffers==23.5.26). These are known and the backend still works. [Source: #10899, #12991] - -### No module named 'tosa' after pip install - -`pip install executorch` does not install tosa dependencies. Run `examples/arm/setup.sh` after pip install. Future: `pip install executorch[ethos-u]`. [Source: #13901] - -### ARM GitLab access issues (resolved) - -`git.gitlab.arm.com` had recurring access issues. Resolved with improved IP access management. [Source: #13557] - -### Cross-compilation flatc issues - -Remove manual `FLATBUFFERS_FLATC_EXECUTABLE` args — newer ExecuTorch builds handle host flatc automatically. [Source: #10964] - -### strided_copy in output graph - -When sample inputs are transposed (e.g., NHWC numpy arrays), `aten.as_strided_copy` appears in the graph. This is inserted by `ExportedProgram.run_decompositions()` and is often a no-op that can be removed. [Source: #16374] - -## Runner Issues - -### Object lifetime bug in arm_executor_runner.cpp - -`BufferCleanup` used `free()` on memory from `ArmMemoryAllocator` (static pools). Hidden by FVP, crashes on real hardware. Fixed in PR #16339. [Source: #16225] - -### FVP log format issues - -ARM GNU compiler may not support C99 format specifiers (`%zd`) by default, causing garbled FVP output. Use `%ld` instead. [Source: #13038] - -### int8 I/O with ML Toolkit - -When using `QuantizeInputs`/`QuantizeOutputs` passes, the PTE expects int8 I/O. The ML Toolkit (MLEK) preprocessing may feed float data, causing type mismatches. [Source: #16899] - -### Cortex-M quantization operators incorrect results - -When using the Arm backend without Ethos-U delegation, Cortex-M quantization operators (`cortex_m_dequantize`, etc.) can produce incorrect results if calibration data is not representative. The default calibration in `aot_arm_compiler` uses `torch.randn(32, 2, 2)` which may not be appropriate. [Source: #13399] diff --git a/.wiki/backends/arm/overview.md b/.wiki/backends/arm/overview.md index bac3dcfd4b4..8a79e300460 100644 --- a/.wiki/backends/arm/overview.md +++ b/.wiki/backends/arm/overview.md @@ -1,277 +1,53 @@ --- -title: Arm Backend Overview +title: Arm Backend Reference Map category: BACKEND_CONSTRAINT backends: [Arm] -last_validated: 2026-04-05 -source_issues: [1004, 1110, 1161, 1163, 1230, 11913, 12237, 12306, 10899, 10964, 12270, 12447, 12627, 12959, 12991, 13022, 13399, 13557, 13842, 13901, 15805, 15870, 16090, 16225, 16244, 16426, 16541, 16628, 16629, 16739, 16779, 16784, 16864, 16899, 16902, 17157, 17211, 17241, 17397, 17437, 17489, 17651, 17653, 17667, 17668, 17753, 17902, 18306, 18319, 18320, 18491, 18500] +last_validated: 2026-10-23 +source_issues: [] --- -# Arm Backend (Ethos-U / Cortex-M) - -The Arm backend delegates model execution to Arm Ethos-U NPUs (U55, U85) for acceleration, with fallback to Cortex-M CPUs. It uses TOSA (Tensor Operator Set Architecture) as the intermediate representation and the Vela compiler to generate Ethos-U command streams. - -## Architecture - -The delegation flow: -``` -PyTorch model → torch.export → PT2E quantization → to_edge → -ArmPartitioner/TOSAPartitioner → TOSA IR → Vela compiler → .pte with Ethos-U command stream -``` - -Three output modes: -1. **Vela output** (default): Generates a binary with Ethos-U command stream for NPU execution -2. **TOSA output**: For reference checking and debugging (not runnable on hardware) -3. **No delegate**: Runs entirely on Cortex-M CPU via portable ExecuTorch kernels [Source: #1161] - -### TOSA Representation - -TOSA requires NHWC (channel-last) memory format. PyTorch uses NCHW (channel-first). The `Permute_Memory_Format_Pass` handles this conversion during lowering, though it was historically WIP with shape update issues. [Source: #1110] - -## Hardware Targets - -| Target | Description | Use Case | -|--------|-------------|----------| -| Ethos-U55 | Micro NPU for Cortex-M | Ultra-low-power edge inference | -| Ethos-U85 (MAC-256) | Higher-performance micro NPU | Corstone-320 based systems | -| Cortex-M55 | CPU with Helium (MVE) | Fallback for unsupported ops | - -## Setup - -### Environment setup - -```bash -cd examples/arm -./setup.sh -``` - -Note: The `--skip-fvp-setup` flag is ignored because `setup.sh` unconditionally calls `install_reference_model.sh` which requires FVP binaries. [Source: #12306 area] - -### Submodule issues - -The Arm backend depends on submodules hosted on `git.mlplatform.org` which has historically been unstable: -- SSL certificate verification failures -- HTTP 500 errors from the server - -**Workaround:** To disable the Ethos-U driver dependency, deinit the corresponding third-party submodule. See upstream docs for current command syntax. [Source: #1004, #1163] - -These submodules have been moved to backend-specific install scripts in later versions. [Source: #1004] - -## Export Example - -```python -from executorch.backends.arm.tosa.partitioner import TOSAPartitioner -from executorch.backends.arm.tosa.specification import TosaSpecification -from executorch.backends.arm.quantizer.arm_quantizer import ( - ArmQuantizer, get_symmetric_quantization_config, TOSAQuantizer -) - -# Quantize -quantizer = TOSAQuantizer(TosaSpecification.create_from_string("TOSA-0.80+BI+u55")) -quantizer.set_global(get_symmetric_quantization_config()) -prepared = prepare_pt2e(exported_model, quantizer) -# ... calibration ... -quantized = convert_pt2e(prepared) - -# Export with delegation -exported = torch.export.export_for_training(quantized, example_inputs) -edge = to_edge_transform_and_lower( - exported, - partitioner=[TOSAPartitioner(...)], - compile_config=EdgeCompileConfig(_check_ir_validity=False), -) -et_program = edge.to_executorch() -``` - -## Quantization - -- Use `TOSAQuantizer` with `get_symmetric_quantization_config()` for Arm targets -- Use `XNNPACKQuantizer` only if targeting XNNPACK fallback (different numerics) [Source: #1161] -- For fused quantized operators: use `quantization_tag` during annotation, or `SubgraphMatcher` for pattern matching Q/DQ nodes [Source: #1230] -- Reference integer decomposition available via `convert_pt2e` with `use_reference_representation=True` for TOSA numeric matching [Source: #1230] - -## Running on FVP - -Models are compiled to `.pte` files, then converted to C headers for embedding in the Cortex-M firmware: -```bash -python backends/cadence/utils/gen_header.py -``` - -The `method_allocator_pool` size in `runner.cpp` controls tensor arena space. Increase it for larger models: -```cpp -__attribute__((section(".sram.data"), aligned(16))) -uint8_t method_allocator_pool[136 * 1024U]; -``` -[Source: #1161] - -## Cross-Compilation for Cortex-M - -Cross-compiling ExecuTorch for Arm Cortex-M hardware (e.g., Raspberry Pi Pico 2) has known friction points: - -1. Documentation needs Cortex-M specific sections -2. The build for Pico 2 may not produce the final binary -3. Using ET as a third-party project in embedded CMake is difficult -4. `libportable_kernels` for Arm baremetal may not support selective build - -Use `-DEXECUTORCH_SELECT_OPS_FROM_MODEL=".pte"` and `-DEXECUTORCH_DTYPE_SELECTIVE_BUILD=ON` for smaller binaries. [Source: #11913] - -## Dynamic Shapes - -The Arm backend does **not** support models with dynamic shapes. SymFloat objects in the graph will cause errors like: -``` -TypeError: Expected a FakeTensor in meta["val"] of node _local_scalar_dense_2, but got SymFloat -``` - -**Workaround:** Fix input sizes at export time. YOLO models in particular require removing dynamic anchor computation. [Source: #12237] - -## YOLO on Ethos-U - -YOLO models require special handling: -- Use `strict=False` in `export_for_training` (attribute mutation not supported in strict mode) -- Fix input sizes to avoid dynamic shapes -- YOLOv12 example available at `examples/models/yolo12` -- Successfully tested on Ethos-U85 MAC-256 / Corstone-320 with fixed input sizes [Source: #12237] - -## INT16 Extended Profile - -The Arm backend supports INT16 quantization for the extended TOSA profile. Supported INT16 ops include: -- Linear (FCNode), Add, Mul, Sub -- Sigmoid, Tanh, Slice -- View/Transpose, Cat, Rescale -- Quantize/Dequantize nodes - -Use `get_symmetric_a16w8_quantization_config()` for 16-bit activations with 8-bit weights. [Source: #13840, #13635] - -**Known issue:** `FuseQuantizedActivationPass` fails for INT16 symmetric quantization in some cases. [Source: #17437] - -## CMSIS-NN Integration (Cortex-M) - -The `backends/cortex_m` module provides CMSIS-NN optimized operators for Cortex-M CPUs (M33, M55, M85): - -| Op | CMSIS-NN Status | -|---|---| -| quantized_add | Supported [#13506] | -| quantized_linear (per tensor/channel) | Supported [#13708] | -| quantized_conv2d | Supported [#13707] | -| quantized_avg_pool2d | Supported [#13709] | -| quantized_max_pool2d | Supported [#13710] | -| quantized_relu / hardtanh | Supported [#13711, #13712] | -| quantized_sub | In progress [#13706] | -| depthwise_conv2d | Supported [#16105] | -| transpose_conv2d | Supported [#16106] | -| max_pool | Supported [#16107] | -| batch_matmul | Supported [#16109] | -| SVDF | Supported [#16110] | -| Pad | Supported [#16111] | -| LSTM | Not yet [#16108] | - -### Benchmark: CMSIS-NN vs TFLite Micro (Alif E8 Cortex-M55) - -MobileNetV2 int8 benchmarks on Alif E8 HP Cortex-M55 (SRAM arenas, MRAM model): - -| Framework | Inference Time | Notes | -|---|---|---| -| TFLite Micro + CMSIS-NN | Baseline | Reference | -| ExecuTorch + CMSIS-NN | ~Comparable | Improving with each release | -| ExecuTorch (portable ops) | Much slower | Use CMSIS-NN for production | - -Key insight: Q/DQ nodes outside delegation add overhead. Use `QuantizeInputs`/`QuantizeOutputs` passes to keep I/O as int8 when possible. [Source: #17157, #17651, #16899] - -## Zephyr RTOS Support - -ExecuTorch can be built as a Zephyr external module. Documentation and sample apps are under `zephyr/samples/`. - -- Ethos-U delegation works via Zephyr -- Cortex-M only (no Ethos-U) also supported -- See `zephyr/README.md` for build instructions [Source: #13508, #17618, #17653] - -## Cortex-A Support - -ExecuTorch runs on Cortex-A CPUs (Android, Linux, macOS, iOS) using XNNPACK for acceleration. No special Arm-specific backend is needed for Cortex-A — use XNNPACK or Vulkan delegates. [Source: #12627] - -## GRU / RNN Support - -GRU layers are not directly supported by the Arm Ethos-U backend. The decomposition of `torch.nn.GRU` fails during lowering. Manual decomposition of GRU into its component ops may work but requires careful handling. [Source: #12270, #17753] - -## BatchNorm2d Without Preceding Conv - -Standalone `BatchNorm2d` (not immediately following a convolution) is not supported for Ethos-U delegation. In TFLite→Vela flow this works, but in ExecuTorch it fails. - -**Workaround:** Manually decompose the BatchNorm into equivalent operations (mul + add with running mean/var). [Source: #17241, #17397] - -## Softmax Performance on Ethos-U - -Softmax decomposition for Ethos-U uses `aten::amax` which runs on the elementwise engine (not MACs), causing poor performance. The Vela performance estimator is not accurate for cycle counts — always validate on FVP or real hardware. [Source: #18319] - -## LayerNorm and Quantization Accuracy - -LayerNorm quantization is sensitive to epsilon values. The default epsilon (1e-5) can cause accuracy issues in int8. For DeiT-tiny and similar transformer models, accuracy drops in the TOSA quantized pipeline may be caused by LayerNorm sensitivity. The `--stable_softmax` flag enables a numerically stable algorithm. [Source: #16426, #18306, #18316] - -## PReLU Not Supported on Ethos-U - -`torch.nn.PReLU` is not supported for Ethos-U delegation. The forward call decomposes to `torch.where(x>0, x, self.weights * x)` which isn't handled by the backend. [Source: #16902] - -## Object Lifetime Bug in ARM Runner - -`arm_executor_runner.cpp` had a major object lifetime error where `BufferCleanup` used `free()` on memory not allocated by `malloc()` (e.g., from `ArmMemoryAllocator` using static pools). This was hidden by ARM FVP but crashes on real hardware. Fixed in PR #16339. [Source: #16225] - -## Quantizer Issues - -### SharedQuantizationSpec recursion - -Using `SharedQuantizationSpec` in certain graph topologies (e.g., `minimum → eq` chains) causes infinite recursion. Fixed upstream in pytorch/ao#3011. [Source: #13842] - -### Conv-ReLU + Residual observer sharing bug - -The Arm Ethos quantizer incorrectly shares/derives observers across `add`, `permute`, and `relu` operations at Conv-ReLU + residual junctions. The root cause is in `quantization_annotator.py` where `add` node annotation doesn't properly handle shared quantization specs. [Source: #12959] - -### LeakyReLU device placement - -ARM quantizers fail on `nn.LeakyReLU` due to the `negative_slope` constant being placed on the wrong device. XNNPACK quantizer doesn't have this issue. [Source: #16541] - -### set_module_name doesn't apply to ops - -`VgfQuantizer.set_module_name()` targets submodules (`torch.nn.Module`), not raw operators. To apply module-level quantization config to an op like `add`, wrap it in a `torch.nn.Module` subclass. [Source: #16542] - -### Name filter doesn't match nodes - -`arm_quantizer.py` name filter assumes module names start with `"L['self']."` which may not be present, causing filters to miss target nodes. Fixed on main. [Source: #15870] - -## Vela Compiler Issues - -### Custom Vela config file - -Custom Vela configuration files may crash due to a parsing bug; fixed in Vela 4.5.0. [Source: #15805] - -### `--optimise Size` result mismatch - -Running Vela with `--optimise Size` can produce different (incorrect) results compared to the default optimization. [Source: #16864] - -### reduce_mean not fully delegated - -The operator support checks for views/reshapes are overly pessimistic — they assume transposes are always needed, rejecting view nodes with axis-product > 65536 even when no transpose is required. [Source: #16779] - -### Ethos-U crashes on conv→relu→permute→reshape(5D) - -Specific graph pattern `conv → relu → permute → reshape(5D)` crashes during partitioning for Ethos-U. [Source: #16739] - -## Setup Issues - -### Dependency conflicts in setup.sh - -`examples/arm/setup.sh` has known dependency conflicts: -- `ethos-u-vela` requires `flatbuffers==24.12.23` but `tosa-tools` requires `flatbuffers==23.5.26` -- `executorch` requires `numpy>=2.0` but `tosa-tools` requires older numpy - -These conflicts are known and the backend still works despite pip warnings. [Source: #10899, #12991] - -### tosa module import failure - -`from executorch.backends.arm.tosa.partitioner import TOSAPartitioner` may fail with `No module named 'tosa'` after `pip install executorch`. You must also run `examples/arm/setup.sh` to install tosa dependencies. A future `pip install executorch[ethos-u]` flow is planned. [Source: #13901] - -### ARM GitLab instability - -Arm-hosted GitLab (`git.gitlab.arm.com`) for tosa-serialization had recurring access issues causing CI failures. This has been resolved with improved IP access management. [Source: #13557] - -### Cross-compilation flatc issues - -When cross-compiling for ARM targets, `flatc` compilation may fail because it tries to build for the target instead of the host. On newer ExecuTorch versions, remove manual `FLATBUFFERS_FLATC_EXECUTABLE` and `EXECUTORCH_BUILD_FLATC` args — the build system handles host flatc compilation automatically. [Source: #10964] +# Arm Backend Reference Map + +This wiki keeps debugging lessons, not a second backend manual. Architecture, +setup commands, supported operators, and example code belong in the maintained +references below. Validation here means comparison with this checkout's code +and documentation, not execution on every target. + +## Current Entry Points + +| Topic | Reference | +|-------|-----------| +| Development decisions, partitioning, layout, and test selection | [Arm skill](../../../.claude/skills/arm/SKILL.md) | +| Ethos-U setup and first deployment | [Getting started](../../../docs/source/backends/arm-ethos-u/tutorials/ethos-u-getting-started.md) | +| Ethos-U configuration and runtime memory | [Backend overview](../../../docs/source/backends/arm-ethos-u/arm-ethos-u-overview.md) | +| VGF setup and deployment | [Getting started](../../../docs/source/backends/arm-vgf/tutorials/vgf-getting-started.md) and [backend overview](../../../docs/source/backends/arm-vgf/arm-vgf-overview.md) | +| Quantizer customization | [Quantizer tutorial](../../../examples/arm/quantizer_tutorial.ipynb) | +| Static and symbolic shapes | [Symbolic TOSA shape lowering](../../../backends/arm/SYMINT_ET_LOWERING.md) | +| Runnable community examples | [Arm examples](../../../examples/arm/README.md) | +| Cortex-M / CMSIS-NN development | [Cortex-M skill](../../../.claude/skills/cortex-m/SKILL.md) | + +## Distinctions That Matter When Debugging + +- **TOSA is an intermediate representation.** Ethos-U compiles it with Vela; + VGF uses Model Converter. A successful TOSA reference-model comparison does + not establish successful compilation or execution on either target. +- **Cortex-M is not another output mode of the Ethos-U partitioner.** Its + CMSIS-NN backend has its own quantizer and graph transformations. Nodes left + outside an Ethos-U delegate still require appropriate runtime kernels. +- **Shape support is target-specific.** TOSA symbolic-shape lowering exists; + the Ethos-U partitioner currently requires resolved tensor shapes, while VGF + does not impose that same check. The selected specification, extensions, and + operator checks still determine whether a particular graph is accepted. +- **Support is configuration-specific.** Use current partitioner checks and + generated support documents, then reproduce the specific case. + Tests describe covered configurations, not every possible model. + +## Operator Support References + +- [Ethos-U55 operator support](../../../docs/source/backends/arm-ethos-u/U55_op_support.md) +- [Ethos-U85 operator support](../../../docs/source/backends/arm-ethos-u/U85_op_support.md) +- [VGF operator support](../../../docs/source/backends/arm-vgf/VGF_op_support.md) +- [Support-check implementation](../../../backends/arm/operator_support/tosa_supported_operators.py) + +For a different release, consult that release's documentation and dependency +pins rather than mixing current instructions with its source tree. diff --git a/.wiki/index.md b/.wiki/index.md index 518bd324b2a..958b2ef1d8d 100644 --- a/.wiki/index.md +++ b/.wiki/index.md @@ -32,9 +32,12 @@ This knowledge base captures tribal knowledge — debugging steps, backend quirk ### CoreML - [Overview](backends/coreml/overview.md) — Apple hardware targets, MPS integration, iOS/macOS deployment, dim_order issues -### Arm (Ethos-U) -- [Overview](backends/arm/overview.md) — Ethos-U55/U85, TOSA, Vela compiler, Cortex-M deployment, FVP setup -- [Known Issues](backends/arm/known-issues.md) — Dynamic shapes, submodule issues, NHWC conversion, build problems +### Arm (TOSA, VGF, Ethos-U) +- [Reference Map](backends/arm/overview.md) — Maintained backend docs, examples, operator support, and development guidance +- [Debugging Guardrails](rules/arm-backend.md) — Target-specific shape support, delegation, quantization, and hardware diagnosis + +Arm entries were refreshed against this checkout on 2026-10-23; this does not +revalidate the rest of the April corpus. ### Cadence (Xtensa) - [Overview](backends/cadence/overview.md) — No-delegation architecture, preserved_ops pattern diff --git a/.wiki/rules/arm-backend.md b/.wiki/rules/arm-backend.md index 2b05048f4a1..d5ee0b67cf0 100644 --- a/.wiki/rules/arm-backend.md +++ b/.wiki/rules/arm-backend.md @@ -1,96 +1,38 @@ --- -title: Arm Backend Rules +title: Arm Backend Debugging Rules category: BACKEND_CONSTRAINT backends: [Arm] -last_validated: 2026-04-05 -source_issues: [1004, 1110, 1161, 1163, 1230, 11913, 12237, 12959, 13842, 13901, 16225, 16426, 16541, 16739, 16779, 16784, 16902, 17241, 17397, 17489, 17668, 18306, 18319] +last_validated: 2026-10-23 +source_issues: [] --- -# Arm Backend — Critical Tribal Knowledge - -1. **No dynamic shapes** — The Arm backend does NOT support models with dynamic shapes. - SymFloat/SymInt in the graph causes `TypeError: Expected FakeTensor ... got SymFloat`. - Fix all input dimensions at export time. [Source: #12237] - -2. **TOSA output is NOT runnable on hardware** — Setting `output_format=tosa` bypasses - Vela and produces a .pte that will fail at runtime with `Init failed for backend ArmBackend`. - You MUST use Vela output for Ethos-U execution. [Source: #1161] - -3. **Submodule init blocks on git.mlplatform.org** — Arm submodules hosted on - git.mlplatform.org have recurring SSL and availability issues. If not using Arm, - deinit these submodules before running `git submodule update --init`. [Source: #1004, #1163] - -4. **Use strict=False for complex models** — Models with attribute mutation (YOLO, etc.) - require `strict=False` in `torch.export.export_for_training()`. [Source: #12237] - -5. **Quantized ops need explicit linking** — Running quantized models without delegation - requires linking `quantized_ops_lib`. Error: `Missing out variants: - quantized_decomposed::dequantize_per_tensor`. [Source: #1161] - -6. **Non-delegated performance is poor** — Running quantized models on Cortex-M CPU - without Ethos-U delegation has very low performance. Always use delegation - for production. [Source: #1161] - -7. **NHWC conversion** — TOSA requires channel-last (NHWC). PyTorch is channel-first (NCHW). - The Permute_Memory_Format_Pass handles this but was historically incomplete. [Source: #1110] - -8. **Increase method_allocator_pool for larger models** — The tensor arena size in - `runner.cpp` defaults to 4KB. Increase it for real models: - `uint8_t method_allocator_pool[136 * 1024U];` [Source: #1161] - -9. **Selective build for baremetal** — Use CMake flags for smaller binaries: - `-DEXECUTORCH_SELECT_OPS_FROM_MODEL=".pte"` - `-DEXECUTORCH_DTYPE_SELECTIVE_BUILD=ON` [Source: #11913] - -10. **C10_USING_CUSTOM_GENERATED_MACROS** — When building as a separate CMake project, - define this macro to avoid `c10/macros/cmake_macros.h not found`. [Source: #11999] - -11. **Use TOSAQuantizer, not XNNPACKQuantizer** — For Arm targets, use - `TOSAQuantizer(TosaSpecification.create_from_string("TOSA-0.80+BI+u55"))`. - XNNPACK quantizer has different numerics. [Source: #1161] - -12. **YOLO models work on Ethos-U85** — YOLOv12 tested on Ethos-U85 MAC-256 / Corstone-320 - with fixed input sizes. See `examples/models/yolo12`. [Source: #12237] - -13. **Observer sharing bug at residual junctions** — Arm Ethos quantizer incorrectly - shares observers at Conv-ReLU + residual add nodes. Can cause quantization errors - in ResNet/MobileNet-like architectures. [Source: #12959] - -14. **PReLU not supported on Ethos-U** — `nn.PReLU` decomposes to `torch.where` - which is not supported. No workaround available. [Source: #16902] - -15. **BatchNorm without Conv not delegated** — Standalone `BatchNorm2d` fails - Ethos-U delegation. Workaround: manually decompose to mul+add. [Source: #17241, #17397] - -16. **conv→relu→permute→reshape(5D) crashes** — This graph pattern crashes - the partitioner during `to_edge_transform_and_lower`. [Source: #16739] - -17. **Softmax amax is slow on NPU** — Softmax decomposition uses `aten::amax` - on the elementwise engine (not MACs). Don't trust Vela cycle estimates — - profile on FVP or real hardware. [Source: #18319] - -18. **LayerNorm quantization is accuracy-sensitive** — Use `--stable_softmax` - for transformer models. Epsilon value (default 1e-5) can cause accuracy drops - in int8 quantization of LayerNorm-heavy models. [Source: #16426, #18306] - -19. **tosa module requires setup.sh** — `pip install executorch` does NOT install - tosa dependencies. Run `examples/arm/setup.sh` after pip install, or you'll get - `No module named 'tosa'`. [Source: #13901] - -20. **arm_executor_runner object lifetime bug** — `BufferCleanup` used `free()` on - non-malloc memory. Fixed in PR #16339. Crashes on real hardware but not FVP. [Source: #16225] - -21. **Ethos-U base_addr mismatch on real hardware** — Output buffers may remain - unchanged on real MCUs despite reported success. Works on FVP. [Source: #16784] - -22. **GRU/RNN not supported** — GRU decomposition fails during Ethos-U lowering. - LSTM CMSIS-NN support planned but not yet implemented. [Source: #12270, #17753] - -23. **ConvTranspose2d fallback failure** — Fails to fall back to CPU when NPU - can't run it, giving "Non-passthrough operation could not run on NPU". [Source: #17668] - -24. **setup.sh dependency conflicts are benign** — flatbuffers and numpy version - conflicts between vela and tosa-tools are known. Backend works despite warnings. [Source: #10899] - -25. **SharedQuantizationSpec recursion** — Certain graph topologies cause infinite - recursion. Fixed in pytorch/ao#3011. [Source: #13842] +# Arm Backend — Debugging Guardrails + +These guardrails were checked against this checkout, not all released versions. +Use the [reference map](../backends/arm/overview.md) for current documentation. +For implementation work, use the [Arm skill](../../.claude/skills/arm/SKILL.md). + +1. **Identify the target first.** TOSA reference execution, VGF execution, + Ethos-U execution, and Cortex-M/CMSIS-NN are distinct paths. Passing one does + not validate the others. +2. **Shape support is conditional.** Ethos-U currently requires resolved tensor + shapes; TOSA/VGF shape support depends on specification and operator checks. +3. **Explain non-delegation before changing support.** Inspect rejection reasons + and delegation tags. Accepted nodes must actually lower, and fallback nodes + still need compatible runtime kernels. +4. **Use current support evidence.** Consult generated operator documents, + support checks, and tests for the selected configuration. + Coverage of one configuration is not universal support. +5. **Preserve quantization and layout semantics.** Inspect annotation and Q/DQ + metadata for accuracy failures. Layout permutes are introduced around + channels-last nodes and can cancel during propagation; remaining conversions + are not automatically bugs. +6. **Separate simulator, hardware, and compiler results.** Check I/O dtype, + memory configuration, and the deployed runner when hardware differs from an + FVP. Compiler estimates are not measured execution performance. +7. **Match setup to the checkout.** Use current requirements, optional extras, + and setup documentation. Resolve package conflicts rather than assuming + incompatible dependency versions will work together. +8. **Verify input contracts.** Match the program's input dtype, shape, and + quantization parameters. An int8-I/O program needs correctly quantized input + values, not the floating-point preprocessing used for a different program.