Two self-hosted runners incorrectly run the same job

[DolceTriade]

Hello,

I'm seeing an issue with our self hosted runner setup. We have a github action that kicks off 3 workflows that run on self hosted runners with the same labels. We create these ephemeral runners on demand via a webhook that activates when it receives workflowrun events in the queued state and matching the labels we select. After the runner completes the workflow, it deletes itself.

This webhook will instantiate runners to service these requests. Unfortunately, I'm seeing issues where occasionally, two runners end up running the exact same job starving one of the tasks of their runner.

When testing this, we allocate 3 runners for 3 jobs, but one job ends up running on two runners. The GitHub Actions page only receives output from one of the runners (idk where the output from the other runner is going, but it's clearly running the same action). Since the runner deletes itself after, the last task never gets run and stays queued forever.

Is this a bug in my set up or is this a quirk of the runner application?