Upstream revert revert auth token fix

[pmahindrakar-oss]

Why are the changes needed?

Fixes race condition in token auth cache #5387

What changes were proposed in this pull request?

Fixes the racy behavior in auth token cache using mutexes and condition variables.

In a nutshell, the change does the following.

• Allows all parallel token requesters to try to acquire a lock
• One of the requester gets the lock and rest get waitlisted and wait for notification from the requester who acquired the lock
• The requester who acquired the lock refreshes the token and save to the token cache
• He also notifies the waitlisted requesters to now go and read the saved token cache and proceed with the request

The test plan below summarizes how this was tested

How was this patch tested?

This was tested locally using integration test and was ran using 20 parallel routines

go test -v -tags=integration -run '^TestAPISuite$/^Test_RandomNameWorkflow$'

• All routines get unauthenticated in the first attempt of using inmemory cached token provider

cat t |grep "If it's an unauthenticated error, we will attempt to establish an authenticated context" |wc -l
      20

• All routines try to lock

cat t |grep "try lock"  |wc -l                                                         python
      20

• One of the routine is able to acquire the lock

cat t |grep "Locked : true"  |wc -l                                                    python
       1

• Rest fail to lock

cat t |grep "Locked : false"  |wc -l                                                   python
      19

• The ones who fail to lock go into waiting state . i.e they get added to condition variables waitlist

cat t |grep "Waiting" |grep -v "Coming" |wc -l                                         python
      19

eg : log for ref

{"json":{"src":"token_cache_inmemory.go:77"},"level":"info","msg":"Coming out of Waiting","ts":"2024-05-22T23:52:47-07:00"}

• The one who has locked goes and saves the token

cat t |grep "Saved"  |wc -l                                                            python
       1

• Broadcasts to all the others waiting using single notification which notifies all the waiters.

cat t |grep "Broadcasted"  |wc -l                                                      python
       1

• Post which they come out of waiting

 cat t |grep "Coming out of Waiting"|wc -l                                              python
      19

And go ahead and read the token

Log file for reference

t.txt

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

Docs link

[codecov]

Codecov Report

Attention: Patch coverage is 34.61538% with 51 lines in your changes missing coverage. Please review.

Project coverage is 61.07%. Comparing base (1e61f4e) to head (595de20).
Report is 135 commits behind head on master.

Files with missing lines	Patch %	Lines
flytectl/pkg/pkce/token_cache_keyring.go	6.25%	30 Missing ⚠️
flyteidl/clients/go/admin/auth_interceptor.go	48.27%	10 Missing and 5 partials ⚠️
...admin/tokenorchestrator/base_token_orchestrator.go	55.55%	2 Missing and 2 partials ⚠️
flyteidl/clients/go/admin/token_source_provider.go	33.33%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5407      +/-   ##
==========================================
- Coverage   61.09%   61.07%   -0.03%     
==========================================
  Files         793      793              
  Lines       51169    51226      +57     
==========================================
+ Hits        31264    31288      +24     
- Misses      17033    17062      +29     
- Partials     2872     2876       +4     

Flag	Coverage Δ
unittests-datacatalog	69.31% <ø> (ø)
unittests-flyteadmin	58.90% <ø> (+0.04%)	⬆️
unittests-flytecopilot	17.79% <ø> (ø)
unittests-flytectl	67.97% <16.66%> (-0.38%)	⬇️
unittests-flyteidl	79.04% <50.00%> (-0.26%)	⬇️
unittests-flyteplugins	61.94% <ø> (ø)
unittests-flytepropeller	57.32% <ø> (ø)
unittests-flytestdlib	65.82% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.