Zero copy send and handling of -ENOMEM

[normanmaurer]

Hi there,

I am the project lead of netty and we are working and adding zero copy send support to it. During doing some tests I tried to send a 10 MB buffer via zero copy send and received an -ENOMEN as the result. So now I am wondering what would be the correct way to handle this. I (wrongly) assumed if the buffer is too large to use zero copy send it would just copy it and do a "regular" send but it seems like this is not happening. Just I just retry it by myself via a regular send in this situation or what would be the best way to handle this ? Also is there a way to know what the maximum buffer size is that I can send via zero copy send operations ?

Tagging @isilence as requested by @axboe .

[isilence]

Not supposed to be different from normal sends in this sense, liburing/test/send-zerocopy even tries to do over 1MB sends (but only with hugepages), but maybe it's hitting some limit, e.g. while pinning pages. I'll try it out later, but I'm curious where ENOMEM comes from. Is it consistently reproducible from the first send?

[normanmaurer]

Seems like it is always happen on the first send... no huge-pages used here.

[axboe]

Maybe it's a hw limit of sorts in terms of memory segments? When you send 1MB for zc, how exactly are you doing it? Is it one single chunk in the iov with sendmsg, multiple vecs, or using send with a single buffer?

[isilence]

Maybe it's a hw limit of sorts in terms of memory segments? When you send 1MB for zc, how exactly are you doing it? Is it one single chunk in the iov with sendmsg, multiple vecs, or using send with a single buffer?

It'll get split into multiple skbs if too large. More likely it's pinned memory accounting or just some difference in the socket memory accounting. @normanmaurer, I'd assume it's not registered buffers (i.e. not IORING_RECVSEND_FIXED_BUF)?

[axboe]

Or perhaps it's the ulimit -l limit? That would also cause -ENOMEM if it exceeds the amount lockable by the user.

[normanmaurer]

Sorry I had a typo... -ENOMEM happens on a 10MB buffer. It's a single buffer with send_zc.

[normanmaurer]

Just to add some more details there... ulimit -l shows 8192 which might be just too small for the send_zc to work.