Investigate frequent failures of the Delete Tugboat Preview workflow

[JunTaoLuo]

User Story or Problem Statement

As the maintainer of the CI in this repo, we need to ensure that there is no resource starvation that will prevent CI from running correctly under high loads of checkins so developers can verify their changes in a timely manner.

Description or Additional Context

This is a followup to the initial GHA Tugboat integration created as part of #13834.

Currently the "Delete Tugboat Preview" workflow fails often, see examples here (success, failure). The failures are due to the cache entries not being found for the branches of the PRs being deleted.

The issue is that this works for closing PRs but not merging PRs and the culprit is which cache scopes are used for the different triggers. From the official documentation:

The value of GITHUB_REF varies for a closed pull request depending on whether the pull request has been merged or not. If a pull request was closed but not merged, it will be refs/pull/PULL_REQUEST_NUMBER/merge. If a pull request was closed as a result of being merged, it will be the fully qualified ref of the branch it was merged into, for example /refs/heads/main.

It turns out that GHA cache uses GITHUB_REF for the scope of the cache entries so the Tugboat Preview ID cached during the PR creation with the refs/pull/PULL_REQUEST_NUMBER/merge scope is not available during PR merge which uses the refs/heads/main scope.

Steps for Implementation

Investigate the reason for failure of these workflows and change the mechanism to more reliably delete tugboat previews once PRs are merged or closed.

Acceptance Criteria

• Tugboat previews of merged/closed PRs are successfully deleted

Team

Please check the team(s) that will do this work.

• CMS Team
• Public Websites
• Facilities
• Accelerated Publishing

[JunTaoLuo]

There are two approaches worth investigating here

1. We'll revert the changes and go back to using webhooks for Tugboat integration with GitHub. This approach is the most familiar to the team and has proven reliable previously. However, the lack of documentation for this unofficial feature and lack of flexibility will remain.

2. Continue fixing issues that come up. I'll work on a prototype in the test repo that will fix this by relying on the caching mechanism on the main branch instead of the PR integration branches.

I'll create the PRs for both approaches and we'll evaluate which approach to take.

[JunTaoLuo]

As per our discussion we've decided to take the former approach of reverting back to using webhooks integration due to its known reliability. I'll have the PRs updated with verification results shortly.

cc @EWashb @BerniXiongA6 @ndouglas @maortiz-27-80