pmr::synchronized_pool_resource is affected by thread priority inversion

[enniobarbaro@…]

In our setup, we have several high priority threads that occasionally need to allocate memory using synchronized_pool_resource::allocate. Such memory is later released by a lower priority thread using synchronized_pool_resource::deallocate.

It might sometimes happen that several allocate calls happen simultaneously with deallocate. Because of the spinlock implementation used by dlmalloc, the high priority threads are in a busy loop trying to acquire the lock, but the kernel might have no resources left to awake the "cleaner" thread that is currently holding the lock.

Attached is a minimum example that Linux SCHED_FIFO scheduler. Running the program, on my system I can see:

1) The program prints 'failed' several times, meaning that some calls to allocate blocked for more than a second.

2) The total execution time is approximately 120 times slower than a version that does not use any thread priority.

Inspecting the program with perf one can see that most of the time is spent inside boost_cont_sync_lock, in particular in sched_yield.

A possible solution could be to switch to a pure pthread implementation by defining USE_SPIN_LOCKS=0 before including dlmalloc_2_8_6.c

System: Linux SPC-LT48 4.9.0-4-amd64 #1 SMP Debian 4.9.65-3 (2017-12-03) x86_64 GNU/Linux

[Ion Gaztañaga]

I can't see any sensible fix, synchronized_pool_resourc is not thought to be prepared to handle priority inversion and several platforms don't support something like PTHREAD_PRIO_INHERIT and the default pthread implementation might not support priority inheritance protocol. I don't know if also setting this flag can affect performance for the common case.

Maybe you should build the library using USE_SPIN_LOCKS=0 and this should be at least documented.