[borys3kk]

Summary

Two XNNPACK lowering improvements found while profiling the fp16 MI-GAN inpainting model. A zero constant_pad_nd before a convolution costs a full extra pass over the activation, which is noticeably slow in NHWC. In MI-GAN these pads, one before every 4x4 blur filter, took 13-22% of the forward. The fp32 I/O casts of an fp16 graph were also the only ops left outside the delegate.

1. Fold zero pads into convolutions (FusePadIntoConvPass).
A zero-valued constant_pad_nd on the two spatial dims whose only user is a non-transposed convolution is folded into that convolution's padding. XNNPACK takes top/right/bottom/left padding independently, so asymmetric pads such as F.pad(x, (2, 1, 2, 1)) before a 4x4 filter fold too. The graph and its shape metadata stay as they are: the pass only tags nodes. The pad visitor aliases the pad output to its input, and the conv visitor serializes the combined padding.

The pass runs last, after ChannelsLastTaggedReshapePass and RemoveRedundantCopyPass. If a layout copy sits between the pad and the conv, the conv is no longer the pad's only user and nothing folds. Quantized pads are left alone. InsertPadQDQPass runs earlier and puts q/dq around pads fed by a dequantize, and the fold skips any pad whose input is a dequantize or whose user is a quantize. A pad in a quantized context therefore still becomes a quantized static pad, as before. Non-zero pad values and transposed convolutions are also not folded.

2. Lower fp32 <-> fp16 casts as XNNConvert.
ToDimOrderCopyConfig now accepts a _to_dim_order_copy that only changes dtype between fp32 and fp16 with the default dim order. Other dtype conversions, and a dtype cast combined with a dim order change, are still rejected. op_to_copy.py serializes such a cast as XNNConvert. ChannelsLastTaggedReshapePass treats the cast as layout-agnostic, so it inherits its input's layout. RemoveRedundantCopyPass no longer removes the cast as a "same-layout copy" and never pairs it with a layout copy as the first half of an NCHW/NHWC round trip. It still removes a same-layout copy that follows the cast.

No runtime change is needed: XNNConvert and TRBL conv padding already exist in the runtime.

Results measured on an ExecuTorch 1.4.1-based branch. They include the upsample_nearest2d lowering from #24113, without which a layout copy sits between every MI-GAN pad and its conv and nothing folds. Timings are Method::execute for MI-GAN 256 / 512 / faces_256 fp16, 10 timed runs after 2 warm-ups, output bit-identical:

	256	512	faces_256
Mac M4 Pro, 14 threads	34 -> 27 ms (-21%)	58 -> 41 ms (-30%)	33 -> 27 ms (-17%)
Galaxy S23 (SD 8 Gen 2), 4 threads	108 -> 105 ms (-3%)	189 -> 182 ms (-3%)	109 -> 107 ms (-2%)

The host gain is larger because XNNPACK's Constant Pad is unusually slow on the M4 Pro. On the device, the casts as Convert nodes alone are within noise. Their benefit is that 100% of the forward now runs inside the delegate.

#24113 (upsample_nearest2d lowering) adds a static method at the same spot in channels_last_tagged_reshape_pass.py. Whichever lands second gets a trivial conflict there.

Review order:

1. fuse_pad_into_conv_pass.py, op_static_constant_pad.py and op_conv2d.py (the fold).
2. generic_node_configs.py, op_to_copy.py, channels_last_tagged_reshape_pass.py and remove_redundant_copy_pass.py (the casts).
3. The tests.

Test plan

New tests:

• backends/xnnpack/test/passes/test_fuse_pad_into_conv_pass.py: conv-pad-conv in fp32 and fp16 and with a depthwise conv. Each asserts through the pass stage that the pad is tagged as folded and that the conv has the expected combined TRBL padding (existing conv padding included), then lowers, runs and compares outputs. Pads that must stay unfused (non-zero value, transposed conv, pad on the graph input, qs8 graph with InsertPadQDQPass in the pipeline) are checked the same way and must stay correct.
• backends/xnnpack/test/passes/test_remove_redundant_copy_pass.py: fp32 I/O around an fp16 conv and a bare fp16/fp32 round trip keep their casts. A channels-last copy after an fp16 cast is still removed.
• backends/xnnpack/test/ops/test_to_copy_dtype.py: fp32 I/O around an fp16 conv and a cast-only graph are fully delegated with no cast left outside. An int cast and a dtype plus dim order cast are not partitioned.

# new tests: 17 passed (7 fuse-pad, 6 remove-redundant-copy incl. 3 existing, 4 to_copy dtype)
python -m pytest backends/xnnpack/test/passes/test_fuse_pad_into_conv_pass.py \
  backends/xnnpack/test/passes/test_remove_redundant_copy_pass.py \
  backends/xnnpack/test/ops/test_to_copy_dtype.py

# with the existing tests of the touched modules: 114 passed
python -m pytest backends/xnnpack/test/passes/test_fuse_pad_into_conv_pass.py \
  backends/xnnpack/test/passes/test_remove_redundant_copy_pass.py \
  backends/xnnpack/test/ops/test_to_copy_dtype.py \
  backends/xnnpack/test/passes/test_channels_last_tagged_reshape.py \
  backends/xnnpack/test/passes/test_insert_pad_qdq.py \
  backends/xnnpack/test/ops/test_conv1d.py backends/xnnpack/test/ops/test_conv2d.py \
  backends/xnnpack/test/ops/test_static_constant_pad.py \
  backends/xnnpack/test/ops/test_to_copy.py backends/xnnpack/test/ops/test_slice_copy.py

# all XNNPACK pass and op tests: 336 passed
python -m pytest -n 8 backends/xnnpack/test/passes backends/xnnpack/test/ops

Run on macOS (Apple Silicon, Python 3.13) against the prebuilt pybindings. lintrunner on the changed files reports no issues.

cc @nil-is-all @GregoryComer @digantdesai @cbilgin @JakeStevens

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24112

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 Unclassified Failures

As of commit ce27dca with merge base 422cc39 ():

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

• periodic / gather-models (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
• periodic / Resolve CI docker image / resolve (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[borys3kk]

@pytorchbot label "release notes: xnnpack"

[JakeStevens]

Add software mansion copyright, not Meta. we will add if an eng modifies too

[JakeStevens]

couple comments that codify this fp32 <-> fp16 cast, I would prefer not codifying this into comments as they will become stale for example if/when we add fp32 <-> bf16

[JakeStevens]

same as above

[JakeStevens]

this compares the output dim_order to the identity order, not to the input's dim order

we can check it with an explicit x.to(torch.float16, memory_format=torch.contiguous_format) on a channels_last input, we would expect it to not be delegated, while just x.to(torch.float16) should be