The bugs in zero-copy related API

[renzibei]

First, I'm talking about the FSTACK_ZC_SEND related API here (although FF_USE_PAGE_ARRAY related api cannot work for my NIC neither).

I found that the f-stack will crash or hang when one try to send mbufs filled with a lot of data. After some research, I think the origin is a designing problem. When we want to use zero copy API, we often want to solve the copying problem when the data is quite large. However, f-stack uses the pattern of non-blocking socket, which means that with every ff_write we can only write a limited number of data. When this limited size is smaller than the total size of mbuf, this sending will not be able to send the whole mbuf. We can neither start from the unsent data position in the next sending because this position may be in the middle of a mbuf, and freebsd doesn't handle this case.

[renzibei]

If we want to make some improvements, I think some there are a few directions.

The first direction is to limit the maximum size of mbufs that f-stack grants to the users at one time to make sure f-stack can send this mbuf as a whole in the future. And we need to guarantee that in every WRITE event, the writable size is larger than this size.

The second direction is to offer a tool function to split the one mbuf chain into two chains, and the first splited chain has the specified length, which is set by the user in order to fit the writable size. Then when a writable event is invoked, the user can split the mbufs to fit the writable size.

The third is to modify the freebsd tx logic, probably we will need to add more information to let freebsd know that it shouldn't send data from the beginning of mbuf but from an offset to it.

I think we can learn some designing from other zero-copy APIs like io_uring ZC send .

[renzibei]

In addition, regarding the current zero copy performance, I think there are some limitations. For example, every time the memory obtained with ff_zc_mbuf_get will be reclaimed by freebsd during ff_write. The user has to get the memory again via ff_zc_mbuf_get. Although the running speed of ff_zc_mbuf_get based on m_getm2 is not particularly slow, there is still a lot of overhead. I think it's possible to use a dedicated memory pool for these zero-copy buffers instead of letting freebsd manage them. We could use special codes in ff_write to reclaim this memory instead of letting freebsd reclaim it. This can greatly reduce the overhead of re-applying memory.

[renzibei]

First, I'm talking about the FSTACK_ZC_SEND related API here (although FF_USE_PAGE_ARRAY related api cannot work for my NIC neither).

I found that the f-stack will crash or hang when one try to send mbufs filled with a lot of data. After some research, I think the origin is a designing problem. When we want to use zero copy API, we often want to solve the copying problem when the data is quite large. However, f-stack uses the pattern of non-blocking socket, which means that with every ff_write we can only write a limited number of data. When this limited size is smaller than the total size of mbuf, this sending will not be able to send the whole mbuf. We can neither start from the unsent data position in the next sending because this position may be in the middle of a mbuf, and freebsd doesn't handle this case.

A new finding is that we seem to be able to ignore kevent.data, i.e. current writable data size, since ff_write doesn't seem to fail in the FF_ZC_SEND case.

[jfb8856606]

Great content!

We will seriously consider how to optimize, but because this is not our main usage scenario, there is no timetable for the time being，and very welcome to submit Pull request.