Thread safety: mmio read and write keep lock when being interrupted by pthread_cancel()

[fhaverkamp]

We created a pthread which was supposed to poll a hardware register. Once we did not need this thread anymore, we canceled it by doing a pthread_cancel(). That caused in some cases deadlocks, due to a lock in a subsequent call to mmio_read/write which required the lock which was not unlocked.

To circumvent the problem we used

pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &oldstate);
do something ...
cxl_mmio_read32()
...
pthread_setcancelstate(oldstate, NULL);

Please review and fix functions which acquire locks and which are likely to be used in multithreaded code which can be stopped with pthread_cancel().

Thanks

Frank

[fhaverkamp]

Here is the original bug and our own circumvention for the underlying problem:
open-power/snap#543