Await logic for deletion is sometimes slow

[EronWright]

It seems like the deletion await logic occasionally fails to observe the actual deletion of the resource, and waits until the timeout (and is then successful). I don't have a repro but here's the example:

View in Browser (Ctrl+O): https://app.pulumi.com/eron-pulumi-corp/nested/dev/updates/7

     Type                                                           Name                              Status              
 -   pulumi:pulumi:Stack                                            nested-dev                        deleted (0.33s)     
 -   ├─ kubernetes:yaml/v2:ConfigGroup                              stack                             deleted (0.15s)     
 -   │  └─ kubernetes:pulumi.com/v1:Stack                           stack:default/stack-tidy-raven    deleted (20s)       
 -   ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding  crb                               deleted (600s)      
 -   ├─ kubernetes:yaml/v2:ConfigGroup                              program                           deleted (0.10s)     
 -   │  └─ kubernetes:pulumi.com/v1:Program                         program:default/stack-tidy-raven  deleted (0.24s)     
 -   ├─ kubernetes:core/v1:ServiceAccount                           ksa                               deleted (0.24s)     
 -   └─ random:index:RandomPet                                      name                              deleted (0.17s)   
Resources:
    - 8 deleted

Duration: 10m24s

The program resembled:

name: example
runtime: yaml
resources:
  ksa:
    type: kubernetes:core/v1:ServiceAccount
    properties:
      metadata:
        namespace: default
  crb:
    type: kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding
    properties:
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: ClusterRole
        name: system:auth-delegator
      subjects:
        - kind: ServiceAccount
          name: ${ksa.metadata.name}
          namespace: default 

There's a possible clue in the diagnostic output. Observe that the phrase "Resource scheduled for deletion" did not appear for the slow resource (the CRB). The phrase comes from cli-utils library here. I would guess that the object was gone before the cli-utils got a chance to observe it.

Diagnostics:
  kubernetes:core:ServiceAccount (ksa)
    Resource scheduled for deletion
    
 
  kubernetes:rbac.authorization.k8s.io:ClusterRoleBinding (crb)
    
 
  kubernetes:yaml/v2:ConfigGroup$kubernetes:pulumi.com/v1:Program (program:default/stack-tidy-raven)
    Resource scheduled for deletion
    
 
  kubernetes:yaml/v2:ConfigGroup$kubernetes:pulumi.com/v1:Stack (stack:default/stack-tidy-raven)
    Resource scheduled for deletion