Discussion issue for customizable targets

[jamesmunns]

Up to now, we've handled targets with differing capabilities and ABI details by defining separate targets, such as thumbv7m-none-eabi and thumbv7em-none-eabihf. For Arm v6m and v7m targets (Cortex-M0 to Cortex-M7), this has worked okay, as the number of options is relatively low.

However, newer v8m targets, as well as architectures like RISC-V, have a much larger volume of potential options.

It is often necessary (or strongly preferrable) for targets to have consistent compilation options for core/alloc/std and the main application, either for ABI reasons (e.g. not mixing hard and soft float ABIs), or for performance reasons (more efficient intrinsics for bit operations, memcpy, etc.)

At RustNL, it was informally discussed whether we could explore options outside of "one target per configuration", including potentially using build-std to recompile the standard library to match selected features for "customizable" targets.

On a Zulip thread regarding ABI-relevant flags, it was also discussed that Rust for Linux may also have uses where the Kernel can be configured with options that affect the ABI (and necessitate building a consistent standard library), such as -Zfixed-x18.

It is not clear yet exactly how this would be specified, what guarantees would be given for stability, etc. This discussion issue is aimed at:

1. Capturing the concrete facts and potential use cases as of today, for example a list of targets, and the kind of configuration they need (regardless of how we "solve"/"implement" this configuration), or other relevant scenarios
2. Discussing potential ways to solve this, and any "pre-requisites" for these solutions, such as what would need to be stabilized or implemented in rustc or cargo to enable these options.

It's likely this will need to be discussed with a wider group of Rust teams, but I wanted to start a tracking/discussion issue for now as it has become relevant.

[romancardenas]

I personally would like to explore an alternative approach to "one target per configuration". In RISC-V, the number of possible combinations is huge. For instance, the new ratified Zaamo and Zalrsc extensions introduce different degrees of atomic operations support.

[bjorn3]

Do these extensions affect the ABI? If not, -Ctarget-feature should be enough as is, right? Separate targets are only strictly necessary when the ABI changes.

[jamesmunns]

@bjorn3 there are cases where it's not required to use a new target, but cases where it's extremely useful to have core/alloc/std using the selected features as well, for example cases where instructions (or knowledge of things like additional registers) are available that can significantly improve performance for memcpy, string/byte comparisons, or other intrinsics defined in the standard library.

I don't have a good answer of what extensions or target customizations are certainly ABI breaking, but I think there are reasonable uses "extremely nice to have" use cases here too.

[bjorn3]

For those cases either an entirely new target or using -Zbuild-std with RUSTFLAGS="-Ctarget-feature=+..." would work, right? Customizable targets would fundamentally require -Zbuild-std anyway, so performance optimizations are not really a good motivating reason for customizable targets IMHO. On the other hand for things that are ABI breaking, customizable targets could be a pretty good solution.

[jamesmunns]

I think that's the general thing we're discussing. For example today, thumbv7m-none-eabihf and thumbv7em-none-eabihf are ABI compatible, but we still have separate targets because the additional instructions in M3 vs M4/M7 are useful to have for performance. There are also FPU options for these targets that I don't think change the ABI, but allow for hardware f64 operations (not available on ALL M7 targets!), which can be pretty make or break for certain DSP use cases.

I agree that the "how we impl this" is probably "change some flags and use build-std", but IMO it doesn't make sense to ONLY have this capability for ABI relevant config. In both cases, you want your application AND the stdlib to have consistent feature enablement, and it would be nice if it didn't require use of nightly/unstable features.

Totally aware this would require RFCs/decisions from the compiler/libs/cargo teams, it was my intent to start that discussion in this issue, as there was at least some support for "propose what you want stabilized and we'll discuss it" from team members at RustNL.

[Darksonn]

In the kernel, we have several places where this kind of flag would be useful. The flag that we have discussed the most is the -Zfixed-x18 flag, which had an MCP at rust-lang/compiler-team#748. There is also more information about the ABI impact at rust-lang/rust#124323. To summarize, it is UB to mix code compiled with -Zfixed-x18 AND -Zsanitizer=shadow-call-stack together with code compiled using neither flag. Code using just one of the two flags cannot cause in UB by being mixed with other flag combinations.

Some other cases:

• Retpoline support in rustc (target features or dedicated flags) rust-lang/rust#116852
• Tracking Issue for -Zfunction-return rust-lang/rust#116853
• SLS support in rustc (target features or dedicated flags) rust-lang/rust#116851
• -Zregparm=3 support in rustc rust-lang/rust#116972
• Tracking issue: -Zreg-struct-return support in rustc rust-lang/rust#116973
• See up-to-date info at rustc wanted features & bugfixes Rust-for-Linux/linux#355

I think these all have ABI concerns if mixed, but I could be wrong.

I imagine that -Zsanitizer also has several other examples of ABI effects? At the very least, I know that you must recompile std when enabling sanitizers in some cases.

there are cases where it's not required to use a new target, but cases where it's extremely useful to have core/alloc/std using the selected features as well

I could list way more flags if we include such things. Note that we don't have std in our build, so we don't use -Zbuild-std. I think it's only std that needs special things when you build it yourself (?).

[9names]

On riscv I believe F extension (or lack thereof) is the only extension that changes ABI.
So if we were to have targets for ABI breaks only for unknown-none-elf that would be:
RV32E, RV32I, RV32IF, RV32ID, RV64I, RV64IF, RV64ID, RV64Q (assuming we ignore RV64E, RV64EF, RV64ED and RV128

[edit]
Oh, maybe D extension does as well 🤔
List updated

[2nd edit]
I missed Q, too.
Officially recognised ABIs listed here: https://github.com/riscv-non-isa/riscv-elf-psabi-doc/blob/master/riscv-cc.adoc#named-abis

[jamesmunns]

I could list way more flags if we include such things. Note that we don't have std in our build, so we don't use -Zbuild-std. I think it's only std that needs special things when you build it yourself (?).

The build-std flag is also used for recompiling core and alloc crates. Like I mentioned, this can impact how intrinsics (memcpy, string comparison) are compiled.

[thejpster]

Ok, first up, I'd like to make a point of order and ask that we name the Arm architectures correctly.

• Armv6-M
• Armv7-M
• Armv7E-M
• Armv8-M
  • Baseline
  • Mainline

Edit: I'm a microcontroller person and these are the M-Profile targets. There are obviously R-profile and A-profile targets as well.

As far as I can recall:

• Armv8-M Mainline is Armv7E-M compatible (that is, an Armv8-M CPU will run Armv7E-M firmware)
• Armv7E-M is Armv7-M compatible
• Armv7-M is Armv6-M compatible
• Armv8-M Baseline is Armv6-M compatible

I think it's also worth noting that 'EABI' is sort of hand waving over a bunch of Arm standards and specifications:

(https://github.com/ARM-software/abi-aa/blob/2982a9f3b512a5bfdc9e3fea5d3b298f9165c36b/bsabi32/bsabi32.rst)

Perhaps it would help to introduce the idea of interior target features and exterior target features. An interior feature affects only the interior of a function, and does not affect its ABI - that is, how it is called and how it returns. An exterior target feature, conversely, does affect the ABI.

The only exterior feature I know of is whether the FPU is used as part of the ABI - what we call eabi and eabihf. Note that any Arm M-profile FPU can be used to carry double precision values into a function - even one that only supports single precision instructions. That's why they're called things like "vfp4d16sp" - it's a single precision version of the Vector Floating Point Unit version 4, with room for 16 doubles (or 32 singles).

There are many interior features, which vary according to which revision of the Arm Architecture you are looking at, and which particular processor you are using (e.g. the Cortex-M7) and how it implements those features, or doesn't, or optionally implements them depending on how the silicon vendor chooses to implement that Arm IP in their SoC. I have tried to outline all of these, and their corresponding rustc target feature flags, in the platform documentation at https://doc.rust-lang.org/nightly/rustc/platform-support/arm-none-eabi.html. Be aware that if you pick a target-cpu, LLVM enables the maximal set of features for that CPU, even if your particular implementation of the CPU doesn't have them. You then have to turn off the features you don't have, which is messy.

I agree with notes above that many of these interior features are related to performance, and it is advantageous to have libcore pre-compiled with them - but only because we don't let users (easily) compile the standard library from source. In particular, it's likely that M-Profile Vector Extensions (MVE, aka Helium) would be pretty useful for some of the Unicode text handling functions.

I final point I want to make is that currently cortex-m-rt mixes up the use of eabihf with the presence of an FPU. If you use a soft-float target but add a target option to enable the FPU (which is entirely valid), cortex-m-rt won't turn the FPU on during the assembly start-up code and you can quickly run into problems. I think it would be better to split FPU support from the use of the hard-float ABI.

[thejpster]

A good source for what GCC supports on bare-metal Arm is: https://wiki.segger.com/GCC_floating-point_options

Also interesting to see what Arm's own compiler does: https://developer.arm.com/documentation/dui0741/f/Overview-of-the-ARM-Compiler-6-Toolchain/ARM-architectures-supported-by-the-toolchain?lang=en

They seem to have a single arm-arm-none-eabi target, and require -march= and/or -mcpu= to specify exactly which architecture or processor you support. I assume they either don't ship the C library pre-compiled, or they have a (target, march, mcpu) tuple for each pre-compiled C library.

[cr1901]

At RustNL, it was informally discussed whether we could explore options outside of "one target per configuration"

For those cases either an entirely new target or using -Zbuild-std with RUSTFLAGS="-Ctarget-feature=+..." would work, right? Customizable targets would fundamentally require -Zbuild-std anyway,

I was under the impression that this was the appropriate way to locally override your CPU variant. For instance, I used "-Ctarget-feature=+c" on riscv32i-unknown-none-elf to fake a riscv32ic-unknown-none-elf target, and it worked just fine.

If the concern is getting core/alloc/std optimized for said targets (big savings for a fake riscv32ic-unknown-none-elf target!), well indeed that's what build-std is for :P.

[jamesmunns]

@cr1901 there are some cases where target features could potentially change the ABI (mentioned above). These are the cases where it would not just work, as compared to just "missing optimization from core/alloc/std components".

[jamesmunns]

Noting that this was discussed in the meeting today, chat logs here: https://libera.irclog.whitequark.org/rust-embedded/2024-09-03#1725387483-1725389567;

Also CC rust-lang/rust#129893, which is what kicked this whole discussion off today.

[FlyGoat]

As discussed in: https://lore.kernel.org/linux-mips/CANiq72mvTTgyTjDCWBz_kOdY1f4gopAtWxyC4P4c+Lr0YVkzLA@mail.gmail.com/

MIPS also needs such facility:

Options are:

• mips1
• mips2
• mips3
• mips4
• mips5
• mips32
• mips32r2
• mips32r5
• mips32r6
• mips64
• mips64r2
• mips64r5
• mips64r6

Each of these needs a soft float and hard float target.

We probably needs triples for microMIPS as well.