Is it possible to achieve 100% reliable synchronous cancellation for immediate resource destruction (RAII)?

[hn-sl]

Context: While developing a C++20 Coroutine-based I/O framework using liburing, I encountered a significant challenge regarding reliable cancellation.

The Motivation (Why Sync Cancel?): If a truly synchronous cancellation were possible inside the Awaitable's destructor, the framework implementation would become significantly simpler and more efficient. It would allow us to treat I/O requests with strict RAII semantics: the Awaitable owns the request, and when it goes out of scope, the request is guaranteed to be gone. This eliminates the need for complex, overhead-heavy state management often found in other async frameworks (such as extending object lifetimes via shared_ptr, maintaining "tombstone" states, or incurring the cost of additional runtime state checks inside the event loop's hot path).

The Intended Design & Problem: My design relies on a direct mapping where the user_data submitted to the ring points to a callback specifically bound to the Awaitable instance. Consequently, the Awaitable must remain alive for as long as its corresponding CQE might exist in the ring.

To satisfy this constraint, I intend to use io_uring_register_sync_cancel() within the Awaitable's destructor to strictly finalize the operation before destruction.

However, an asynchronous gap prevents this design: Although io_uring_register_sync_cancel() waits for the cancellation command to be processed, it does not consume or remove the CQE of the original request. By design, the original request will inevitably post a CQE to the ring (either as -ECANCELED or as a successful completion if it finished just in time).

Consequently, even after a successful sync-cancel return, a "Phantom CQE" will eventually appear in the completion queue. If the destructor proceeds to destroy the Awaitable, the main loop will later fetch this Phantom CQE and dereference the invalid pointer, leading to UB.

The "CQE Stealing" Idea (My Question): To enforce strict RAII, I am considering a manual "CQE Stealing" approach inside the destructor.

Since the destructor itself runs within the context of the main event loop (inside an io_uring_for_each_cqe block), simply iterating from the head again is problematic (unknown start point, iterator conflict). Therefore, I am thinking of iterating backwards from the tail:

Logic:

Call io_uring_register_sync_cancel().

Manually scan the CQ ring buffer in reverse order (from tail to head) inside the destructor.

Find the CQE that matches my user_data.

Patch/Swap the user_data in that CQE to a harmless dummy_callback.

Return from the destructor (safely destroying the Awaitable).

Performance Note: While linear scanning might sound inefficient, I expect this to be fast (close to O(1)) in practice. Since the scan is performed immediately after the cancellation, the target CQE should be located at or very close to the tail of the ring.

Questions:

Feasibility: Is this "Reverse Scanning & Swapping" strategy technically feasible and safe? I am concerned about the timing issue where io_uring_register_sync_cancel() returns but the CQE has not yet been visible in the ring buffer.

Safety: Is it valid to access and modify the ring buffer in this manner while the outer loop is iterating?

Alternative: Is there any existing mechanism to perform a cancellation that strictly guarantees "no future CQE will appear for this user_data" before returning?

[noteflakes]

The trick with io_uring is that you need to take into account the lifetime of each operation in both userland and the kernel.

If the user_data references an object on the stack, you'll have to wait for the corresponding CQE to arrive before returning from your op function. If user_data references an object in the heap then you can mark it as cancelled and free it when the corresponding CQE is processed.

[romange]

I confirm it's a valid concern. I have the same issue with life-cycle management when using io_uring_register_sync_cancel together with IORING_OP_POLL_ADD with IORING_POLL_ADD_MULTI flag.

1. Scenario A: filling IORING_OP_POLL_ADD and then immediately io_uring_register_sync_cancel for the same userdata id.
   if io_uring_submit has not been called, then we will have a race where io_uring_register_sync_cancel will be called before IORING_OP_POLL_ADD is processed. yes, it is possible to call io_uring_submit each time we add IORING_OP_POLL_ADD SQE but I am not sure it's a 100% correct fix, and whether it's guaranteed that kernel registers the polling before io_uring_submit returns.

2. Scenario B (more complicated): there is already a completion event for this id somewhere in kernel which will eventually surface as a userspace notification after io_uring_register_sync_cancel returns . If we delete the object assigned to the multi-shot id right after the call to io_uring_register_sync_cancel we will segfault when the completion will be processed.

I would expect that io_uring_register_sync_cancel will serve as a hard barrier after which no completions for this id will be issued.

I am trying to come up with a workaround as even if kernel behaviour will be improved, it won't help us as we need the solution for currently released kernels.

My next idea:

1. Use IORING_OP_POLL_REMOVE asynchronously with hopes it will linearize completions, i.e all the completions for previous submissions for that socket will happen earlier. I do not have strong confidence with this approach. Btw, based on my experiments IORING_OP_POLL_REMOVE together with IO_DRAIN - it is broken and I saw comments here that using IO_DRAIN is not advised.
2. Give up on removing POLL entirely and somehow manage the clean-up at the callback level by identifying the "last" callback invocation. I am not sure it is possible to identify the last invocation easily.

@axboe @isilence maybe I am thinking all wrong - would love to hear you thoughts as well.