Defer containers

[dhilipkumars]

What this PR does / why we need it:
Introduces a new Termination Semantic for pod.

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #

Implements this feature. The implementation has diverged reasonably from its orignal proposal.
We will update the proposal to reflect what has been implemented.

Special notes for your reviewer:

NOTE: This is an unpolished implementation, has a lot of rough edges. This PR is raised just to get some consensus on the approach we have taken.

Here are some of the highlights of the approach

1. It is designed to work across kubelet restarts
2. The default and only restart policy of DeferContainer is 'OnFailure'.
3. If deferContainers are configured for a POD it disables (overrides) PreStopHooks
4. If all the containers during deferContainers are completed well ahead of GracePeriod, the POD would be killed without waiting anyfurther.
5. deferContainer's images if that is different from the other containers in the POD, it will be pre-pulled while the pod is running.
6. the property of killPod or killPodwithSyncResults functions to be blocking is maintained.
7. Implemented as an alpha annotation.

Here is a sample pod spec configured with deferContainers and with a gracePeriod of 60 secs

#LifeCycle Demo
apiVersion: v1
kind: Pod
metadata:
  name: defer-demo
  annotations:
    pod.beta.kubernetes.io/init-containers: '[
       {
         "name": "first",
         "image": "bash",
         "command": ["/usr/local/bin/bash", "-c", "echo started"],
         "volumeMounts": [
            {
               "name": "config",
               "mountPath": "/config"
            }
         ]
       }
    ]'
    pod.alpha.kubernetes.io/defer-containers: '[
       {
         "name": "last-but-one",
         "image": "bash",
         "command": ["/usr/local/bin/bash", "-c", "echo entered deferContainers"],
         "volumeMounts": [
            {
               "name": "config",
               "mountPath": "/config"
            }
         ]
       },
       {
         "name": "last",
         "image": "busybox",
         "command": ["/bin/echo", "finished"],
         "volumeMounts": [
            {
               "name": "config",
               "mountPath": "/config"
            }
         ]
       }
    ]'
spec:
  terminationGracePeriodSeconds: 60
  containers:
  - name: prestop
    image: bash
    command: [ "/usr/local/bin/bash", "-c", "--" ]
    args: [ "while true; do sleep 1; echo Prestop1:`date`;  cat /config/test.file; done;" ]
    volumeMounts:
      - name: config
        mountPath: /config
    lifecycle:
      preStop:
        exec:
          command: ["/usr/local/bin/bash","-c", "echo preExitHook_executing > /config/test.file; sleep 2; echo preExithook_finished > /config/test.file"]
  volumes:
      - name: config
        emptyDir: {}

Here is the events for the pod. As you can see the deferContainer named last has a different image which is getting pre-pulled

  FirstSeen     LastSeen        Count   From                    SubObjectPath                           Type            Reason                  Message
  ---------     --------        -----   ----                    -------------                           --------        ------                  -------
  2m            2m              1       kubelet, kube-node-3                                            Normal          SuccessfulMountVolume   MountVolume.SetUp succeeded for volume "config"
  2m            2m              1       kubelet, kube-node-3                                            Normal          SuccessfulMountVolume   MountVolume.SetUp succeeded for volume "default-token-7xgh1"
  2m            2m              1       default-scheduler                                               Normal          Scheduled               Successfully assigned defer-demo to kube-node-3
  2m            2m              1       kubelet, kube-node-3    spec.initContainers{first}              Normal          Started                 Started container with id a8ed41f882604468d6564fb3fd0696369344d188b1ef510f31ea65988217245d
  2m            2m              1       kubelet, kube-node-3    spec.initContainers{first}              Normal          Created                 Created container with id a8ed41f882604468d6564fb3fd0696369344d188b1ef510f31ea65988217245d
  2m            2m              1       kubelet, kube-node-3    spec.containers{prestop}                Normal          Created                 Created container with id 4a5cd7168ccc1d5d51c2c60c77e34f6a2348bac12797513a3dc11a411be87eb2
  2m            2m              1       kubelet, kube-node-3    spec.containers{prestop}                Normal          Started                 Started container with id 4a5cd7168ccc1d5d51c2c60c77e34f6a2348bac12797513a3dc11a411be87eb2
  53s           53s             1       kubelet, kube-node-3    spec.deferContainers{last}              Normal          Pulling                 Defer-Container image is being pre-pulled
  2m            16s             3       kubelet, kube-node-3    spec.initContainers{first}              Normal          Pulling                 pulling image "bash"
  2m            10s             3       kubelet, kube-node-3    spec.initContainers{first}              Normal          Pulled                  Successfully pulled image "bash"
  53s           10s             2       kubelet, kube-node-3    spec.deferContainers{last}              Normal          Pulling                 pulling image "busybox"
  10s           10s             1       kubelet, kube-node-3    spec.deferContainers{last-but-one}      Normal          Created                 Created container with id 9ed5e37058012476bb42769e488dbd39dc07d47cb5b90c0d108898558909cf0a
  10s           10s             1       kubelet, kube-node-3    spec.deferContainers{last-but-one}      Normal          Started                 Started container with id 9ed5e37058012476bb42769e488dbd39dc07d47cb5b90c0d108898558909cf0a
  45s           5s              2       kubelet, kube-node-3    spec.deferContainers{last}              Normal          Pulled                  Successfully pulled image "busybox"
  5s            5s              1       kubelet, kube-node-3    spec.deferContainers{last}              Normal          Created                 Created container with id e334389c25b40e69e0b2f6e8c7b6897fddd237a1586ead3b8bc3b81fa50b942e
  5s            5s              1       kubelet, kube-node-3    spec.deferContainers{last}              Normal          Started                 Started container with id e334389c25b40e69e0b2f6e8c7b6897fddd237a1586ead3b8bc3b81fa50b942e
  3s            3s              1       kubelet, kube-node-3    spec.containers{prestop}                Normal          Killing                 Killing container with id docker://4a5cd7168ccc1d5d51c2c60c77e34f6a2348bac12797513a3dc11a411be87eb2:Need to kill Pod

Also the pod does not wait to run-out of all of the GracePeriod, as soon as all the deferContainers are completed the pod is deleted. Even though the grace period is configured for 60seconds since all the deferContainers completed successfully in 7 seconds, so the pod is deleted without waiting further.

CC: @dchen1107 @smarterclayton @erictune @brendandburns @Kargakis @quinton-hoole @irfanurrehman @kow3ns @sig-node-reviewers

TODO:

1. Tests
2. E2E and Integration tests
3. better integration with kubctl like for status and logs etc.,
4. Documentation

Release note: none

[huawei-k8s-bot]

@dhilipkumars: Adding do-not-merge/release-note-label-needed because the release note process has not been followed.

One of the following labels is required "release-note", "release-note-action-required", or "release-note-none".
Please see: https://github.com/kubernetes/community/blob/master/contributors/devel/pull-requests.md#write-release-notes-if-needed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[huawei-k8s-bot]

@dhilipkumars: The following tests failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
pull-kubernetes-bazel-test	2e7f91c	link	/test pull-kubernetes-bazel-test
pull-kubernetes-bazel-build	2e7f91c	link	/test pull-kubernetes-bazel-build
pull-kubernetes-unit	2e7f91c	link	/test pull-kubernetes-unit
pull-kubernetes-verify	2e7f91c	link	/test pull-kubernetes-verify

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.