RFC: #[export_visibility = ...] attribute
#3834
Conversation
anforowicz
force-pushed
the
export-visibility
branch
2 times, most recently
from
a79823e to
ab37907
Compare
|
|
||
| ## Benefit: Smaller binaries | ||
|
|
||
| One undesirable consequence of unnecessary public exports is binary size bloat. |
There was a problem hiding this comment.
Should only be the case for libraries. For binaries all functions are already made not-exported.
There was a problem hiding this comment.
True, though on (weird/stunt) occasions there might be reasons to have public symbols in a binary.
(That case is not common and not important, just mentioning it in case someone figured it was categorically impossible and never happened.)
There was a problem hiding this comment.
I think @joshtriplett's comment implies that there actually might exist a use case for #[export_visibility = "public"] (or interposable) on binary crates. This should be noted in the RFC. We can still choose not to support it initially.
text/0000-export-visibility.md
Outdated
| (when the freeing allocator expects that the pointer it got was earlier | ||
| allocated by the same allocator instance). | ||
|
|
||
| This is what happened in https://crbug.com/418073233. In the smaller repro |
There was a problem hiding this comment.
I would have expected all of Chromium to use a single rust allocator rather than use a different one for each DSO. Why is that not the case?
There was a problem hiding this comment.
I would have expected all of Chromium to use a single rust allocator rather than use a different one for each DSO. Why is that not the case?
Is that really a requirement if foo.so doesn't export any functions that return pointers to Rust-related objects? I would expect in such a case that which Rust allocator / standard library / etc is used would be an internal implementation detail of foo.so. IIUC this detail leaks out only because of an unintentional export of a cxx-generated, internal symbol.
But to try to answer the question - the same allocator is statically linked into Chromium binaries. This means that an executable and an .so may end up with a separate copy of the same global data structures of the allocator.
There was a problem hiding this comment.
Is that really a requirement if foo.so doesn't export any functions that return pointers to Rust-related objects?
It is not a requirement. I'm just surprised that Chromium copies the entire rust standard library between the dylib and executable rather than using the copy from the dylib in the executable to save space.
There was a problem hiding this comment.
Is that really a requirement if foo.so doesn't export any functions that return pointers to Rust-related objects?
It is not a requirement. I'm just surprised that Chromium copies the entire rust standard library between the dylib and executable rather than using the copy from the dylib in the executable to save space.
That is indeed a bit unfortunate. I think this is to some extent based on the following:
- Chromium requirement to use an external linker
- Assumption that only
rlibs /static_libs can be linked by an external linker, and that an external linker wouldn't be able to handledylibs
But thank you for bringing this up - maybe this should indeed be treated as an alternative fix for https://crbug.com/418073233. I am not sure what the next steps should be for this aspect:
- Maybe I should open a bug (either under https://github.com/rust-lang/rust/issues, or under https://crbug.com) to move this discussion elsewhere?
- Maybe before opening a bug I should first learn more about
dylibs...
|
For most use cases rather than specifying the exact symbol visibility (which may not even be supported by the object file format, like interposable on pe/coff or (with the default two-level namespaces) mach-o) I think having just a way to force SymbolExportLevel::Rust rather than the default SymbolExportLevel::C would be a better idea. This causes it to still be exported from rust dylibs (as necessary to avoid linker errors depending on the exactly when rustc decides to codegen functions), but prevents it from being exported from cdylibs. It doesn't work for staticlibs currently, but for those if you want to limit symbol visibility you have to specify your own version script during linking anyway to prevent exporting all rust mangled symbols too. |
The fact that you are distinguishing between
That's not 100% correct - instead of using a version script, one may also use |
The Chromium case is effectively equivalent to using staticlibs, not to using rust dylibs/cdylibs.
That doesn't apply to the standard library unless you go out of your way using unstable features to recompile the standard library. |
Ack / agreed.
Thank you for bringing up this point. This probably should be explicitly addressed by the RFC (*), but I am not sure if I agree with your conclusions so far. This is because:
(*) I am not sure what the right process is here. Should I add commits to the RFC as we keep discussing here? Should I first give people an opportunity to review the first draft? |
ehuss
added
the
T-lang
|
I think this is a good opportunity to expand the design space (and documentation) of "various levels of exportendess" a bit, even if the resulting proposal for There are multiple attributes that targets this similar space ( What I'd like to see is a table of "levels of exportedness" combined with the kinds of end artifacts, and how we can users can express all those levels with the attributes listed above.
In particular, one of my requirements is that |
This is something only rustc must be allowed to do (other than for symbols defined in inline asm called from within the same inline asm block). Only rustc knows if all callers will end up in the same object file as the definition and it doesn't provide any guarantees around when this happens. So exposing this to the user is a stability hazard.
For regular functions and
For rlib this doesn't make sense. There is no way to make rlibs a symbol export boundary without introducing an expensive link/object file rewrite step for each individual rlib. For staticlib it would be nice to have a symbol export boundary, but unfortunately we don't have one right now even for
This makes sense to me. See the end of my comment.
This has to always be the case if it is visible outside of the object file. The very point of rust dylibs is that rust code in a separate DSO can call any public function, which thanks to cross-crate inlining can call effectively every function that rustc wouldn't make private to the current object file. And again, rustc doesn't provide any guarantees when this happens, so allowing you to not export symbols from a rust dylib is a stability hazard.
Yes.
No
Not really aside from the visibility information we already tell the linker (export from rust dylib, don't export from cdylib). Currently rustc internally works with three different symbol export levels:
It makes sense to me to allow |
I think this probably should be captured somehow as one of the alternatives in the RFC. Is there a specific syntax that you have in mind here? I guess one option would be to have a |
👍
I don't think this is a good name as it is still meant to be usable from C, just not outside of the linked DSO.
This would be an option, although ideally if we manage to stop exporting all symbols from staticlibs, I would like the same attribute to be usable to prevent export from both cdylib and staticlib, so it should probably not mention cdylib in the name. I don't have suggestions for a better name though. |
|
Frankly, if we have |
|
In Rust 1.87 and before on GNU/Linux, this code: #![feature(rustc_attrs)]
#[rustc_std_internal_symbol]
#[no_mangle]
pub extern "C" fn blah(i: u32) -> u32 {
i+1
}produced an object with the What this issue is not applicable to:
Omitting the In C++, mangling (controlled by The analog to what This errors now in 1.88, the only way i'm aware of being injecting an assembly declaration |
|
@bjorn3 - thank you for proposing a narrower fix, focusing on setting
From my perspective both
|
|
I think the current RFC does a good job of creating a conceptual framework for symbol visibility that can abstract over platform differences, and the remaining issues seem like they can be addressed. Perhaps the issue with dylibs can be solved by linting on calls to non-inlined, hidden-linkage functions from inlined or generic functions. My feeling is that having two export levels, "Rust" and "C", isn't quite the right abstraction – or if it is, we should be able to spell it as I love the idea @petrochenkov raised in #3834 (comment) of having a table of all the things you might want to accomplish and how you would spell them. I wouldn't put the onus on this RFC to completely fill out that table, but it would be helpful to know which gaps exist and which are getting filled. It's also important to me that the users who really know what they're doing when it comes to symbol visibility should be able to exercise direct control. While those users are a small percentage of Rust users, I do see this as a significant benefit to them and to unlocking Rust usage in more arcane contexts. That control can be mediated by a table that maps from "how it's spelled in C compilers and linker scripts" to "how to spell it in Rust"; it doesn't require us to adopt the existing models or terminology wholesale. There are places we might reasonably want to place limits on this (as brought up by @bjorn3 above, setting visibility to an individual codegen unit is almost certainly a bad idea). Otherwise I think we should have some path that favors flexibility and transparency, and avoid forcing outside experts who come to Rust to guess at complex logic the compiler might be doing and why. |
|
@rfcbot fcp merge |
|
Team member @tmandry has proposed to merge this. The next step is review by the rest of the tagged team members: Concerns:
Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up! cc @rust-lang/lang-advisors: FCP proposed for lang, please feel free to register concerns. |
rust-rfcbot
added
proposed-final-comment-period
traviscross
added
the
I-lang-radar
traviscross
changed the title
#[export_visibility = ...] attribute.#[export_visibility = ...] attribute
Hmmm... after having sent the above, now I think it is a bit inaccurate. I guess |
I hope that the inliner can check if the inlined code makes cross- |
|
In general, it can't codegen an inline function that calls a hidden symbol, which is bad because |
During codegen it is not yet known if two crates will end up getting linked into the same dylib, so it would need to inhibit inlining for any cross-crate calls into hidden symbols, not just cross-dylib calls. |
| /// gets inlined into another `dylib` then the call to the internal helper | ||
| /// will cross `dylib` boundaries - this will **not** work if the internal | ||
| /// helper is hidden from dynamic linking. | ||
| #[inline] |
There was a problem hiding this comment.
Can we solve this with a check that public #[inline] and generic functions do not call a function with #[export_visibility = "hidden"]? I don't think we need any kind of transitive call-graph analysis, just to check which functions are directly called.
I think we should also prevent pub fns from being marked as #[export_visibility = "hidden"].
cc @bjorn3 whose perspective I'm particularly interested in on this.
There was a problem hiding this comment.
I saw this comment: #3834 (comment)
During codegen it is not yet known if two crates will end up getting linked into the same dylib, so it would need to inhibit inlining for any cross-crate calls into hidden symbols, not just cross-dylib calls.
This makes sense. My expectation would be for us to explicitly error/lint if you try to do this with public #[inline] or generics, and silently inhibit inlining of the function calling a hidden symbol otherwise.
There was a problem hiding this comment.
Fundamentally, trying to avoid these errors by restricting how source programs are written means committing to details of rustc’s codegen strategy that are currently unstable implementation details. To give just one concrete example not covered by the above comments, rustc currently has some logic to treat small functions as-if they were #[inline] for codegen purposes even if they weren’t declared as such in the source code. I expect it’ll be really hard to figure out a set of rules that reliably avoids these errors and doesn’t restrict desirable current or future cleverness in the compiler. (Edit: … and isn’t too restrictive to be useful — #[no_mangle] is basically a guarantee that it’ll get codegen’d but people probably want to factor out calls to visibility=hidden without adding more unmangled symbols to their library.)
There was a problem hiding this comment.
Thanks @tmandry for proposing linting as a potential answer to the hidden-vs-dylibs problem in this RFC.
Thanks @hanna-kruppe for pointing out that basing the proposed lint on inlineability would lead to an undesirable outcome (changing codegen implementation details could result in breaking existing Rust code by making the warning fire more or less than before).
FWIW I tried to cover this in an updated text of the RFC - see a4ab948. (Rendered)
The discussion also made me think about having a lint that fires in presence of any Rust callers of a hidden Rust function. As I say in the new text of the RFC, this seems very drastic, but in practice #[no_mangle] are oftentimes called only from another, non-Rust language. This is definitely the case for FFI thunks used as one of motivating examples in this RFC.
There was a problem hiding this comment.
I expect it’ll be really hard to figure out a set of rules that reliably avoids these errors and doesn’t restrict desirable current or future cleverness in the compiler.
The rule I'm thinking about is "if a function references a hidden-visibility symbol it is not eligible for inlining". That seems straightforward to add on top of the existing heuristics based on function size. Is this too simplistic?
#[no_mangle]is basically a guarantee that it’ll get codegen’d but people probably want to factor out calls to visibility=hidden without adding more unmangled symbols to their library.
Seems fine, they can do that.
#[unsafe(no_mangle)]
#[export_visibility = "hidden"]
extern "C" fn hook_callable_from_c() {}
pub fn hook() {
hook_callable_from_c()
}I guess the question is why you would want to do this, and whether it's worth the double indirect call. I don't have a good sense of the use case here.
There was a problem hiding this comment.
The problems here apply to "interposable" if you expect the function to be globally overridden. That is, if the interposable function gets inlined it can no longer be overridden in the contexts where it's inlined. The difference is that you can get unexpected behavior instead of a linker error, and that doesn't seem better.
Seems like a function explicitly marked interposable should disable inlining by default, if we keep that name. We can allow users to re-enable it explicitly if they know what they're doing.
Alternatively we can go with a name that promises less like "public".
edit: Now that interposable has been removed from the RFC we don't have to solve this problem now, but I still want to check my understanding of the solution space here.
There was a problem hiding this comment.
The rule I'm thinking about is "if a function references a hidden-visibility symbol it is not eligible for inlining". That seems straightforward to add on top of the existing heuristics based on function size. Is this too simplistic?
You'd have to put this check in at least two distinct places (the MIR inliner, and the place that decides to put MIR into metadata even if the function otherwise wouldn't be). And then you have to somehow ensure that "this function does not reference a hidden-visibility symbol so I can inline it" remains true between the time you checked it and the time artifacts get linked. That goes against the grain of how an optimizing compiler wants to work so there's probably many ways to subvert it. To give one simple example, this...
fn generic<T>() {
let cb = get_callback();
cb(type_name::<T>())
}
#[inline]
fn this_can_be_inline_right() {
let cb = get_callback();
cb("hello");
}
fn get_callback() -> fn(&str) {
return hidden_visibility_fn;
}... can easily turn into the following by interprocedural constant propagation, even if get_callback isn't considered eligible for inlining:
fn generic<T>() {
let tmp = hidden_visibility_fn as fn(&str);
tmp(type_name::<T>());
}
#[inline]
fn this_can_be_inline_right() {
let tmp = hidden_visibility_fn as fn(&str);
tmp("hello");
}
fn get_callback() -> fn(&str) {
return hidden_visibility_fn;
}I don't think this is the sort of issue that can be fixed by adding more restrictions in the same vein to even more passes besides inlining-related ones. The core problem is that trying to maintain a "lexical" property of the code (here: whether certain symbols are mentioned directly) clashes with the usual framework for reasoning about compiler optimization correctness: define semantics of program in IR, check that each individual optimization pass refines program behavior. To a first approximation, if you want some property to be true about the machine code that gets linked in the end, you should probably find a way to represent that property in the IR semantics.
Finally, I'd like to emphasize again that I'm just pointing out single counter-examples that come to mind easily, I can't provide an exhaustive checklist. If you find a nice way to solve this, I can't promise I won't come up with a different "challenge" after thinking about it some more. One thing that already came to mind while I was drafting this comment is that it might make sense for a dylib to also contain LLVM bitcode for everything linked into the dylib to enable better LTO of binaries linked against the dylib. I haven't checked if rustc already does that but if it (ever) does, this would give even more ways for a reference to a hidden symbol to cross a dylib boundary.
There was a problem hiding this comment.
Note that we already compile everything with -fno-semantic-interposition just like clang. This causes LLVM to assume that any interposed functions behave identically to the original and optimize accordingly. In other words interposition with -fno-semantic-interposition is effectively UB afaik.
text/0000-export-visibility.md
Outdated
| ## Cross-platform behavior | ||
| [cross-platform-behavior]: #cross-platform-behavior | ||
|
|
||
| We don't really know | ||
| whether the `hidden` / `protected` / `interposable` visibilities | ||
| make sense across different target platforms and/or map to distinct entities | ||
| (see | ||
| [a Zulip question here](https://rust-lang.zulipchat.com/#narrow/channel/233931-t-compiler.2Fmajor-changes/topic/.60.23.5Bexport_visibility.20.3D.20.2E.2E.2E.5D.60.20attribute.20compiler-team.23881/near/522491140)). | ||
|
|
||
| One weak argument is that these visibilities are supported by LLVM and Clang, so hopefully | ||
| they would also make sense for Rust: | ||
|
|
||
| * **LLVM**: Those visibilities are ultimately mapped from | ||
| [`rustc_target`'s `SymbolVisibility`](https://github.com/rust-lang/rust/blob/81a964c23ea4fe9ab52b4449bb166bf280035797/compiler/rustc_target/src/spec/mod.rs#L839-L843), | ||
| through | ||
| [`rustc_middle`'s `Visibility`](https://github.com/rust-lang/rust/blob/81a964c23ea4fe9ab52b4449bb166bf280035797/compiler/rustc_middle/src/mir/mono.rs#L396-L407), | ||
| and into | ||
| [`rustc_codegen_llvm`'s `Visibility`](https://github.com/rust-lang/rust/blob/81a964c23ea4fe9ab52b4449bb166bf280035797/compiler/rustc_codegen_llvm/src/llvm/ffi.rs#L153-L160). | ||
| So all the values make some sense at | ||
| [the LLVM level](https://llvm.org/docs/LangRef.html#visibility-styles). | ||
| * **Clang** and **GCC** support those 3 visibilities | ||
| (see the "Parity with C++" subsection in the "Motivation" section above). | ||
|
|
||
| OTOH, ideally we would somehow check what happens on some representative subset | ||
| of target platforms (maybe: Posix, Windows, Wasm?): | ||
|
|
||
| * TODO: what exactly do we want to verify on these target platforms? |
There was a problem hiding this comment.
I don't think we can merge the RFC without any due diligence here. The RFC should demonstrate that the parameters it gives for each option can be implemented for at least our Tier 1 platforms, or specify where there are exceptions or particular unknowns.
text/0000-export-visibility.md
Outdated
|
|
||
| * TODO: what exactly do we want to verify on these target platforms? | ||
|
|
||
| ## Rust standard library |
There was a problem hiding this comment.
Changing this strikes me as a future possibility and not within the direct scope of the RFC. Do you agree? If so we should move it out of this section.
|
I did another pass over the RFC and remembered a few things that need to be resolved before merging. I'm still positive on the RFC overall. @rfcbot concern specify how inlining interacts with hidden symbols
@rfcbot concern due diligence on tier 1 platforms
@rfcbot concern UB potential of interposable
|
|
My apologies for not replying earlier - I somehow managed to miss that there was additional feedback on this PR. Let me acknowledge that there are a few action items that I should take care of at this point:
I’ll try to work on those items (editing this comment to mark things as done when I make progress). |
anforowicz
force-pushed
the
export-visibility
branch
from
c656e67 to
2d5eea8
Compare
|
This PR was rebased onto a different master commit. Here's a range-diff highlighting what actually changed. Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers. |
This comment was marked as resolved.
This comment was marked as resolved.
|
RE: @chorman0773
Maybe changing this behavior is an option on the table? To me the main problem with inlining suppression is that it may suppress many legitimate, non-
I think that generics may be a red herring here, because IIUC 1a) generics can't be marked as FWIW I tried capturing the concerns and discussion above in the text of the RFC - see db7c119 RE: @tmandry
The RFC proposes 3 potential answers for the hidden-vs-dylibs problem. So far we have identified significant concerns about one of those potential answers (avoiding inlining in some cases). OTOH I think that I haven't heard any feedback about the other 2 potential answers - would those 2 answers be acceptable and let us proceed with the RFC? Or are there arguments for explicitly rejecting one or two of those other 2 answers? In particular, the RFC proposes either:
|
|
@tmandry, can you please clarify the concern below?
IIUC the RFC does not add any new kinds of visibility to If you think that delegating the responsibility to LLVM is not okay, then can you please help me understand what kind of test results or what kind of data you'd like to see added to the RFC? |
that's incorrect https://play.rust-lang.org/?version=stable&mode=debug&edition=2024&gist=acebbccd2aeb3384a353161e9be5f801 |
|
I would note that generics being Also, the point about generics is that a generic function may call an |
| OTOH, ideally we would somehow check what happens on some representative subset | ||
| of target platforms (maybe: Posix, Windows, Wasm?): |
There was a problem hiding this comment.
As far as I'm aware, there is no Windows equivalent of the "protected" visibility class.
There was a problem hiding this comment.
Doesn't COFF have protected as normal visibility class, with no equivalent for ELF default visibility? And same for Mach-O. ELF default visibility allows another dylib to overwrite exported functions of the local dylib for calls from within the local dylib.
The Rust language does not assume LLVM, and rustc supports multiple backends. Saying that these map to LLVM definitions is not enough on its own. Those definitions might have meaning or not on various platforms and in various combinations. The way I see it, there are two directions this RFC could go:
1 is my preferred outcome. The goals of cross-platform and intuitive are in tension, however, and not necessarily related to the way that LLVM surfaces its options. Looking over the LLVM and GCC docs, some of the visibility options strike me as leaky abstractions with a list of platform-specific caveats. We can simplify the problem by restricting scope. As you suggest, it's valid to cut out "hidden" and focus on a single "protected" option, with more options as future possibilities. I would like to know we have a plausible path to stabilizing future options, i.e. we'll need to discuss some tradeoffs but there's at least one option that looks usable. With my current understanding I think "hidden" meets this bar. Another future possibility is that we stabilize both "abstract" and "platform-specific" options under the same attribute. To clarify something, am I correct in understanding that your particular use case is solved with either "hidden" or "protected"? |
My usecase (i.e. fixing https://crbug.com/418073233) can be addressed by either:
I think that my usecase would not be addressed by |
This RFC proposes to add
#[export_visibility = …]attribute, which seems like a reasonable way to address the following issues:#[no_mangle]symbols are exported from acdylibrust#98449This RFC complements the
-Zdefault-visibility=...command-line flag, which is tracked in rust-lang/rust#131090This PR replaces the Major Change Proposal (MCP) at rust-lang/compiler-team#881
(/cc @bjorn3, @ChrisDenton, @chorman0773, @joshtriplett, @mati865, @workingjubilee, and @Urgau who have kindly provided feedback in the Zulip thread associated with that MCP)
/cc @tmandry from rust-lang/rust-project-goals#253, because one area where this RFC seems needed is FFI tooling
Rendered