ev-threads sometimes get stuck in busy-loop

[itrofimow]

So, I was benchmarking samples/hello-service the other night, hoping to see some nice numbers since I've got my hands on a couple of 192-cores machines, however what I've encountered was this:

After the benchmark is finished, some of the ev-threads are left stuck in busy loop:

, with the following backtrace:

(lldb) * thread #11, name = 'event-worker_9', stop reason = signal SIGSTOP
    frame #0: 0x00000000004f03e4 userver-samples-hello_service`userver::v2_14_rc::concurrent::impl::IntrusiveMpscQueueImpl::TryPopBlocking() [inlined] userver::v2_14_rc::compiler::RelaxCpu::operator()(this=<unavailable>) at relax_cpu.hpp:15:9
   12  	#if defined(__x86_64__) || defined(__amd64__) || defined(__i386__)
   13  	        __builtin_ia32_pause();
   14  	#elif defined(__aarch64__)
-> 15  	        __asm__ __volatile__("yield");
   16  	#else
   17  	#error "Unsupported CPU architecture"
   18  	#endif
bt
* thread #11, name = 'event-worker_9', stop reason = signal SIGSTOP
  * frame #0: 0x00000000004f03e4 userver-samples-hello_service`userver::v2_14_rc::concurrent::impl::IntrusiveMpscQueueImpl::TryPopBlocking() [inlined] userver::v2_14_rc::compiler::RelaxCpu::operator()(this=<unavailable>) at relax_cpu.hpp:15:9
    frame #1: 0x00000000004f03e4 userver-samples-hello_service`userver::v2_14_rc::concurrent::impl::IntrusiveMpscQueueImpl::TryPopBlocking() [inlined] userver::v2_14_rc::concurrent::impl::(anonymous namespace)::BlockThreadUntilNotNull(next=<unavailable>) at intrusive_mpsc_queue.cpp:20:9
    frame #2: 0x00000000004f03e4 userver-samples-hello_service`userver::v2_14_rc::concurrent::impl::IntrusiveMpscQueueImpl::TryPopBlocking() [inlined] userver::v2_14_rc::concurrent::impl::IntrusiveMpscQueueImpl::TryPop(this=0x0000ffffb2e51200, mode=kRarelyBlocking) at intrusive_mpsc_queue.cpp:121:12
    frame #3: 0x00000000004f0370 userver-samples-hello_service`userver::v2_14_rc::concurrent::impl::IntrusiveMpscQueueImpl::TryPopBlocking(this=0x0000ffffb2e51200) at intrusive_mpsc_queue.cpp:57:12
    frame #4: 0x00000000004ef03c userver-samples-hello_service`userver::v2_14_rc::engine::ev::Thread::UpdateLoopWatcherImpl() [inlined] userver::v2_14_rc::concurrent::impl::IntrusiveMpscQueue<userver::v2_14_rc::engine::ev::AsyncPayloadBase>::TryPopBlocking(this=0x0000ffffb2e51200) at intrusive_mpsc_queue.hpp:114:65
    frame #5: 0x00000000004ef034 userver-samples-hello_service`userver::v2_14_rc::engine::ev::Thread::UpdateLoopWatcherImpl(this=0x0000ffffb2e51200) at thread.cpp:163:52
    frame #6: 0x00000000004eef04 userver-samples-hello_service`userver::v2_14_rc::engine::ev::Thread::RunEvLoop(this=0x0000ffffb2e51200) at thread.cpp:140:9
    frame #7: 0x0000ffffb3dadd2c libstdc++.so.6`execute_native_thread_routine + 28
    frame #8: 0x0000ffffb40948b8 libpthread.so.0`start_thread + 392
    frame #9: 0x0000ffffb3aacb7c libc.so.6`thread_start + 12

This reproduces for me on current master (83e0706), and also reproduces on the parent of e61f96e

[itrofimow]

@Anton3 this is probably your wheelhouse