Custom thread pool integration

[kripergvg]

Hello.
Thanks a lot for this great library!
I wanted to ask one thing - suppose we have a custom ThreadPool and custom task types. Is there any difference to how coyote performs if we put SchedulingPoint.Interleave(); manually comparing to if we replace our custom tasks with the coyote's ones? Is there any benefits in doing so?
Additionally - do you have any guidance where SchedulingPoint.Interleave(); should be put in case of custom Task types and ThreadPool? For me it seems that for coyote to work properly we can leave our custom ThreadPool with custom worker threads and just put SchedulingPoint.Interleave(); at places where work is scheduled or executed and then use coyote rewriting for everything else. Is it a correct assumption?

[pdeligia]

Hi @kripergvg, sadly your assumption does not hold. Coyote would not work with a custom ThreadPool or custom worker threads, as these require custom instrumentation of their lifetime (e.g., when they start executing, when they complete, when they pause and for what reason they paused so they can get re-enabled, and all these goe much beyond adding SchedulingPoint.Interleave*).

Coyote today works with either:

(1) [recommended] binary instrumentation (i.e. using coyote rewrite) to automatically instrument the most common thread/task-related types that are known & supported by Coyote today (such as Task) in your binaries so that they can be controlled by the Coyote runtime and their interleavings can be explored during testing, or

(2) [at your own risk, not supported] by manually instrumenting your cystom types (or in general types not instrumented already via coyote rewrite above) using the experimental Coyote Operation extensibility API. This API is a subset (and wrapper) of the same internal APIs that Coyote uses during binary instrumentation, which are very robust, but what we publicly expose via Operation is experimental (neither battle-tested nor complete), so might not work for your custom threading scenario, and thus YMMV (we would like to eventually support manually instrumenting custom scenarios, but there is no roadmap/ETA as it is low-priority for our team).

*Regarding using SchedulingPoint.Interleave it will not achieve what you want, as it requires a thread to already be controlled which is done through (1) or (2) above (all the Interleave API does is give a hint to Coyote to potentially "explore pausing the currently executing controlled thread, and continue executing a previously paused controlled thread").

[kripergvg]

Hi @pdeligia, thank you very much for the answer.
I think I didn't explain what I meant properly. I was thinking about using coyote rewrite for all of our binaries and then add SchedulingPoint.Interleave at some points in thread pool code to make sure there are interleaving points at these important sections. So now I am wondering if using coyote rewrite is enough for custom thread pool to work (because all custom workers will be instrumented by coyote rewrite )?

[pdeligia]

Thanks for clarifying @kripergvg, this makes much more sense to me now, but sadly my point from above still stands about custom thread pools! It actually depends how your thread pool is creating worker threads. If it creates threads using regular C# APIs like thread.start which Coyote already understands and rewrites during coyote rewrite there is a good chance it will work, but might require some extra SchedulingPoint.Interleave. It also depends if you introduce new synchronization constructs or not. If your custom thread pool synchronizes the threads somehow (or e.g. queues them up, etc) this could interfere with Coyote runtime logic, unless also instrumented (with the more complex APIs that I mentioned above). It is really hard for me to tell you how easy/hard/possible this is, without deeper knowledge of your thread pool design, as its very situational :)

Another thought: what is the primarily reason behind your custom threadpool? Is it for performance? Or changing runtime semantics? If it is the former, could you remove the custom thread pool for testing only? This would reduce the complexity required to control and explore interleavings here.

[kripergvg]

Additionally to custom ThreadPool we have custom awaitable Task type that has a bit different API. I think it might be possible to replace ThreadPool with standard one but not sure about Task type. Ideally I would like to run tests using our types so it's closer to production environment.
I will try to go with coyote rewrite + SchedulingPoint.Interleave. Thank you for explanation.

[pdeligia]

Oh, interesting that you are also having custom task types alongside the threadpool, makes your scenario even more complex than I originally imagined :)

I fully agree that ideally you should be able to rewrite and run Coyote on the production concurrency types (both custom tasks and threadpool), if semantics are indeed different from the .NET versions, else behavior may diverge and you lose coverage. In general, I find your scenario super interesting, but just sadly this is a non-standard/supported scenario for Coyote at this moment as I explained above (due to the complexity of controlling arbitrary user-defined concurrency types, so although we eventually would like to make our experimental extensibility API complete & robust so users can achieve what you are trying to do, it's currently low priority), so YMMV really!

IMO, the best approach might still be for testing to revert to using the original .NET types: one option for a custom Task type with different API would be to have a "mock" wrapper around the original .NET task that implements the same methods (even if it's not as efficient as your production version, as long as the semantics are the same, it doesn't matter since it's for testing). (This way you also avoid the other issue you raised in your PR with thread renaming, which is an assumption that Coyote runtime takes today.) But I am saying this without fully understanding the scope of your custom types, plus might be non-trivial to achieve in your case.

Let me know how it goes!

[kripergvg]

For sure, thank you very much for guidance!