[Question] Difference between SEND_ZC and provided buffers

[nikneym]

Hi, I couldn't wrap my mind around this specific situation, please correct me if I'm mistaken. SEND_ZC tries to skip internal buffers and puts the buffer directly to the wire if possible. Whereas the goal of registered I/O is to share the buffer between kernel and userspace to get rid of copy overhead.

Essentially these two seem to do the same thing, which is reducing copies. One extra cost of SEND_ZC could be the additional CQE it posts but other than that, I couldn't find a difference. Which one of them should be preferred over the other in what scenarios?

[axboe]

SEND_ZC is to avoid copying memory into the kernel just for sending purposes. It maps and pins the userspace buffer and sends than, posting a 2nd notification whenever the data has been acked and is safe to reuse.

Registered buffers pre-map/pin a userspace buffer.

So no, they do not do the same thing at all, but registering a buffer saves a pin/map and unpin/unmap for each SEND_ZC. That operation is something that both SEND_ZC and O_DIRECT read/write on a file needs to do, pre-registrering the buffers does that once rather than per operation.

[axboe]

If it's SEND_ZC, yes it'd help. If it's a normal SEND, no it won't do anything.

[isilence]

That's too broad of a recommendation. @cmazakas, take a profile w/o it, and we can try to estimate from there. I saw a meaningful speed up from registered buffers when I was wiring up SEND_ZC, but it'll depend on your workload. For example, large sends with huge pages will likely supersede any registered buffer benefit.

[axboe]

That's too broad of a recommendation

The statement is that registered buffers will make send/sendmsg ZC faster. That's not broad, that's just a fact. How much will vary depending on how much data is set. For big sends, yeah won't make much of a difference, it'll just reduce CPU cycles a bit. For tiny sends, it'll make a big difference.

[cmazakas]

Alright, you guys sold me. I'll start considering how to add it to my project.

I'm going to wind up keeping both methods (registered buffers vs not), as there's a lot of utility in both approaches.

[isilence]

The statement is that registered buffers will make send/sendmsg ZC faster. That's not broad, that's just a fact.

Not if the userspace overhead to use the feature would outshadow it. And there is a dev time aspect as well. It's a good thing to mention that it doesn't work with non-zc sends, but I don't want to be setting wrong expectations for people; there are cases where it's beneficial, and there are where it might not worth the effort, which is why I mentioned profiles.

How much will vary depending on how much data is set. For big sends, yeah won't make much of a difference, it'll just reduce CPU cycles a bit. For tiny sends, it'll make a big difference.

On the opposite, larger sends with multiple pages get more out of it, smaller ones are dominated by the net stack overhead, but that depends on the particular constant.

[axboe]

Moved this to a discussion, as it's not an issue.

[isilence]

@nikneym, pretty much as folks already mentioned, SEND[MSG]_ZC avoids copies of the payload buffer. Registered buffers (i.e. SEND[MSG]_ZC with IORING_RECVSEND_FIXED_BUF) save on page pinning and also other internal accounting. Provided buffers mentioned in the title is a third thing, and doesn't help with either aforementioned factor.

As for what's faster, normal sends can be faster than send zc depending on the payload size and other circumstances. Last time I benchmarked it, the parity for TCP was somewhere around 3K per send [1]. I'd recommend to try it for yourself and compare. And if you're using send zc, then also try to enable registered buffers, there are no (kernel) downsides in having them.

[1] https://lore.kernel.org/all/3b06ebe5-509e-45d2-9a41-5f2af67a36a4@gmail.com/

[nikneym]

@isilence thank you for the detailed response, I'll definitely benchmark both options but I feel fixed buffers together with SEND_ZC is what I'm looking for since the program mostly do small yet frequent writes.

Provided buffers mentioned in the title is a third thing, and doesn't help with either aforementioned factor.

By provided buffers, I was talking about ring-mapped provided buffers (io_uring_setup_buf_ring). I thought it supersedes the earlier registered buffer implementation, is it not the case for network I/O?

[axboe]

Registered and provided buffers have nothing to do with each other. Registered buffers enable what is explained above, provided buffers is a way to hand buffers to the kernel ahead of time so that eg a recv can pick a suitable buffer when the socket has data to be received.

For provided buffers, there's the old/legacy approach, which was indeed superseded by ring provided buffers. They provide the same functionality, ring provided buffers are just more efficient than the old approach.

[nikneym]

For provided buffers, there's the old/legacy approach, which was indeed superseded by ring provided buffers. They provide the same functionality, ring provided buffers are just more efficient than the old approach.

I understood, the confusion from my end was that I thought io_uring_setup_buf_ring is a replacement for the io_uring_register_buffers, not the io_uring_prep_provide_buffers. It seems io_uring_register_buffers is a whole different concept.

[axboe]

Exactly, provided buffers can be either the old/legacy style or the new ring buffer style. Those are basically the same concept in terms of what they provided, even if the new provided ring buffers provide more features than the old ones. Both are very different from registered buffers, which do the pre map/pin and keeps buffers pinned for DMA until they get unregistered.

I blame myself for the naming...

[isilence]

@isilence thank you for the detailed response, I'll definitely benchmark both options but I feel fixed buffers together with SEND_ZC is what I'm looking for since the program mostly do small yet frequent writes.

Small writes is where you usually don't need zerocopy. You can try it out and see, it depends on the IO pattern and how it pushed onto the wire, but often it's faster to copy those 100 bytes of data in the kernel and tightly pack pages with it than doing zerocopy.