Next Task Optimization for the Mulithreaded Executor

[james7132]

Objective

Whenever the multithreaded executor spawns system tasks, it always starts a new task when a system is supposed to run. This presents a potential context switch if the task is stolen by another thread. Even with #11906, its still pretty common to see systems and the mulithreaded executor ping-pong between multiple threads, particularly when systems becomes serialized due to conflicting accesses or explicit ordering, with each bounce incurring a context switch overhead. This PR aims to address that overhead.

Solution

This PR ensures that at least one thread is running a scheduled system or the multithreaded executor at any given time. This is done by withholding one of the systems that are slated to run, and instead of spawning a task, directly run the system inside the same task the mulitthreaded executor is running in after releasing the lock on the executor. So long as there is at least one system the executor can run, and the task can grab the lock on the multithreaded executor, it will continue to run in a loop without yielding to the async task executor.

In strictly serialized schedules, there should now be close to zero OS-level overhead between running a single threaded executor and the multithreaded executor.

Performance

The performance difference here is more noticeable with schedules with a large number of serialized or short-lived systems. Tested on many_foxes (yellow is main, red is this PR), there is about a consistent 5% performance improvement on the main schedule.

On the Render schedule, this yields a more meager 1% improvement on the entire schedule:

[re0312]

result with benchmark #12873

[chescock]

I experimented with something similar to this recently, and found a few potential issues when looking at the traces more closely.

On my 4-core machine, most of the benefit came from the fact that there were extra threads running. The main and render threads currently sleep while running a schedule, but with this change they will run systems alongside the compute task pool threads.

My understanding is that some architectures need to send rendering tasks to the main thread, so running more of the main schedule there could delay rendering (until #9122 is merged).

Also, if you have tasks saturating the Io and AsyncCompute task pools, then keeping the main thread running would give you more threads than cores and cause thrashing. That won't show up in many_foxes since it doesn't do much on those pools, although it might become a problem if #1907 is resolved.

The other potential issue is that system tasks sometimes get picked up by the scope running a par_iter. With this change, system tasks can run much longer and could block the scope for longer. It would be possible for a par_iter in the Render schedule to block while it runs the entire PostUpdate schedule, or vice versa!

[chescock]

If other threads failed to lock the executor, running the task before looping will delay spawning dependent systems until the cached system is run.

For example, if you have add_systems(Update, ((a, b).chain(), (c, d).chain())), then b and d can run concurrently. But if c completes while a holds the executor lock, then the thread that completed a will run b before running the executor to spawn d.

You may want to do something like

let mut cached_system = None;
loop {
    if let Ok(mut guard) = self.environment.executor.state.try_lock() {
        guard.tick(self, &mut cached_system);
        drop(guard);
        if !self.environment.executor.system_completion.is_empty() {
            continue;
        }
    }
    match cached_system.take() {
        Some(cached_system) => unsafe { cached_system.run_system(self) },
        None => break,
    }
}