Introduce a new Buffer trait.

[sunfishcode]

I'm experimenting with a Buffer trait similar to #908, however I've run into a few problems. See the questions in examples/new_read.rs for details.

@notgull @SUPERCILEX

[SUPERCILEX]

If we go with this approach, I feel like there's a pretty good chance compiler folks will improve the error messages for us if we file bug reports. The story around passing in a mutable array but needing a slice instead has annoyed me for quite some time (I usually just as_mut_slice everything). It was pretty confusing to figure out initially, so I'm sure better error messages would be welcome.

[SUPERCILEX]

Looking at our APIs, I think we'd want to consider adding uninit methods to roughly these methods (the filtering is not great, sorry):

• https://docs.rs/rustix/latest/rustix/?search=%26mut%20%5Bu8%5D
• https://docs.rs/rustix/latest/rustix/?search=Vec%3Cu8%3E

[sunfishcode]

Ok, I've now updated everything in rustix to use the new Buffer trait, except:

• readlink, readlinkat, ptsname, ttyname - These return CString, which is a little different than a buffer of u8.
• The ancillary data API for sendmsg/recvmsg. These are tricky. And not needed due to Add uninit buffer ancillary APIs #1108.
• IoSliceMut, because we reuse std's IoSliceMut type.
• IoSliceRaw, because it's potentially tricky.

I'm not opposed to having them use Buffer, I've just run out of steam for doing them myself. If anyone would like to see them migrated, PRs would be welcome.

[SUPERCILEX]

Are we sure we want this one? I feel like that's almost always wrong and you meant to use spare capacity. Maybe there's a use case I'm missing?

[SUPERCILEX]

This seems very footgunny... is the thinking that users might create a big buffer initialized to zero and then pass that in? I can see that making sense, but it still scares me.

[sunfishcode]

I added this because rustix's current APIs already support using Vec like this, because Vec has a DerefMut to &mut [T]. This just ensures that any existing users doing this don't need to add an extra * to keep working.

[SUPERCILEX]

Fair enough

[SUPERCILEX]

Thank you so much for polishing this idea! We'll see if the pitch forks come out, but I'm very excited that MaybeUninit is no longer a second class citizen.

[SUPERCILEX]

My thoughts on the APIs that weren't updated.

readlink, readlinkat, ptsname, ttyname - These return CString, which is a little different than a buffer of u8.

The "problem" here is that all of these methods grow the buffer that's passed in. So I think these methods are fundamentally different and thus there's no need to try and shoehorn them into using the buffer trait.

The ancillary data API for sendmsg/recvmsg

This seems fine because these APIs are like inotify::Reader and RawDir: they produce well typed values and are using the buffer as scratch space rather than returning it to you for your consumption. You gain nothing by passing in non-MaybeUninit buffers and simply shouldn't be allowed to do that, so not supporting the buffer trait is fine.

IoSlice

I have no opinion on these right now. I agree that more thought will be needed, so punting and creating separate methods in 1.x and then fully replacing in a distant v2 seems fine.

[sunfishcode]

I found another confusing error message when testing with this patch:

error[E0521]: borrowed data escapes outside of closure
   --> src/sys/linux_macos.rs:124:35
    |
118 |     let listxattr_func = if deref {
    |         -------------- `listxattr_func` declared here, outside of the closure body
...
124 |     let vec = allocate_loop(|buf| listxattr_func(&*path, buf))?;
    |                              ---  ^^^^^^^^^^^^^^^^^^^^^^^^^^^ `buf` escapes the closure body here
    |                              |
    |                              `buf` is a reference that is only valid in the closure body

Offhand, I'm not sure what it means.

The code is in https://github.com/sunfishcode/xattr (which is a fork of https://github.com/Stebalien/xattr with fixes for rustix 1.0.0).

[sunfishcode]

I found a way to work around that error in that particular case, so I'm going to merge this and proceed with the next prerelease. If that error or other confusing errors pop up in more places, we can reevaluate.

[sunfishcode]

This is now prereleased in rustix 1.0.0-prerelease.1.

[SUPERCILEX]

What was the fix to that error? My guess is that again you'd need to reborrow.

[kevinmehall]

The MaybeUninit impl doesn't play well with retry_on_intr because the borrow checker can't see that the closure won't be called again after it has returned Ok with the borrow from the buffer.

let mut event_buf = [MaybeUninit::<epoll::Event>::uninit(); 4];
let (events, _) = retry_on_intr(|| epoll::wait(epoll_fd, &mut event_buf, None)).unwrap();

error: captured variable cannot escape `FnMut` closure body
  --> src/platform/linux_usbfs/events.rs:93:40
   |
92 |     let mut event_buf = [MaybeUninit::<epoll::Event>::uninit(); 4];
   |         ------------- variable defined here
93 |     let (events, _) = retry_on_intr(|| epoll::wait(epoll_fd, &mut event_buf, None)).unwrap();
   |                                      - ^^^^^^^^^^^^^^^^^^^^^^^^^^^---------^^^^^^^
   |                                      | |                          |
   |                                      | |                          variable captured here
   |                                      | returns a reference to a captured variable which escapes the closure body
   |                                      inferred to be a `FnMut` closure
   |
   = note: `FnMut` closures only have access to their captured variables while they are executing...
   = note: ...therefore, they cannot allow references to captured variables to escape

I think you pretty much always want to ignore EINTR on epoll::wait.

This works, and isn't too bad, though:

let events = match epoll::wait(epoll_fd, &mut event_buf, None) {
    Ok((events, _)) => events,
    Err(Errno::INTR) => &mut [],
    Err(e) => panic!("epoll::wait failed: {e}"),
};

[SUPERCILEX]

Yup, it's the reborrow bug again. You need to use event_buf.as_mut_slice() instead.

[SUPERCILEX]

Here are the minimal repros for the two bugs we've found:

• https://play.rust-lang.org/?version=stable&mode=debug&edition=2024&gist=8a42a839db9ca11dc04dc13c85cbcc8f
• https://play.rust-lang.org/?version=stable&mode=debug&edition=2024&gist=cc44131bdcd2981de98bf3e2dbb2ebd3

I'll see if I can start a zulip thread about improving this.

[sunfishcode]

What was the fix to that error? My guess is that again you'd need to reborrow.

Moving the let listxattr_func = ... into the inner closure: Stebalien/xattr@2fa7d57 fixes the error.

[sunfishcode]

* https://play.rust-lang.org/?version=stable&mode=debug&edition=2024&gist=cc44131bdcd2981de98bf3e2dbb2ebd3

This example gets the same error message as the error in xattr, however the code is a little different. This example is fixed by adding &mut *, while the error in xattr is not.

[sunfishcode]

I filed #1356 to add an example with standalone testcases of all the various interesting error messages we've seen with Buffer, along with ways to fix them.

[kevinmehall]

The snippet I posted uses &mut [MaybeUninit<T>] where epoll::wait returns (&mut [T], &mut [MaybeUninit<T>]), while the example in #1356 demonstrates retry_on_intr with a &mut [T] buffer where the Output would be usize if the example function used it. I don't think re-borrowing helps with the MaybeUninit case, because it needs the full lifetime for the return value.

As I said, it's not necessarily bad to have to use match instead of retry_on_intr here, but it negates some of the value of retry_on_intr.

[sunfishcode]

@kevinmehall Ah, thanks for pointing that out. I've now filed #1375 to add an example that uses MaybeUninit with retry_on_intr. I unfortunately was also unable to find a better workaround, so the advice I went with is, use an explicit loop instead of using retry_on_intr.

[SUPERCILEX]

@kevinmehall Ah, I didn't realize this was specific to the MaybeUninit variants. Indeed, this time the compilation error is correct in that it prevents a potential soundness issue if rety_on_iter did something liek this:

fn uh_oh(f: impl FnMut() -> T) -> (T, T) { (f(), f()) }

let mut event_buf = [MaybeUninit::<u8>::uninit(); 4];
let (mut_ref_one, mut_ref_two) = uh_oh(|| &mut event_buf);

Honestly I wonder if we should just get rid of retry_on_intr—it doesn't handle EAGAIN and I usually prefer the loop style since it lets you break for different reasons.

[sunfishcode]

Agreed that retry_on_intr is often not useful, however it is occasionally handy. It doesn't check for EAGAIN because that usually means the code should return to its event loop rather than spinning until data arrives.

[sunfishcode]

I've now written a blog post about the Buffer trait: https://blog.sunfishcode.online/writingintouninitializedbuffersinrust/

[SUPERCILEX]

Great blog post, thanks for sharing!