#start AD
In this project there is presented an usage of Kubernetes operators to create closed loops. Here there are presented two closed loops: responsive - fast and deliberative - slow. Fast manages same of resources as cpu and memory and slow manages the parameters of a fast closed loop as eg. a prority in serving resources, when simultaneous serving is needed.
One closed loop which we treat as operator with closedloop name consists of three operators: monitoring, decision and execution. Separation of this functions is purely conventional. This makes easy to define a closedloop. The deliberative closedloop has added suffix "_d" to his name in order to make different names compare to the responsive closedloop. Then we have for responsive closedloops the following operators: closedloop_d, monitoring_d, decision_d and execution_d.
You’ll need a Kubernetes cluster to run against. You can use KIND to get a local cluster for testing, or run against a remote cluster.
Note: Your controller will automatically use the current context in your kubeconfig file (i.e. whatever cluster kubectl cluster-info shows).
We use kubebuilder (https://book.kubebuilder.io/quick-start) to build and run our operators. This tool require the following (state on November 2023):
- go version v1.20.0+
- docker version 17.03+. To run docker without sudo, follow post-installation steps for Docker
- kubectl version v1.11.3+. To run kubectl without sudo, change the ownership and permissions of ~/.kube
sudo mv /root/.kube $HOME/.kube # this will write over any previous configuration
sudo chown -R $USER $HOME/.kube
sudo chgrp -R $USER $HOME/.kube- Access to a Kubernetes v1.11.3+ cluster. For kubernetes cluster you can use KIND described above
Run the installation procedure. Run this as sudo user. Check instalation by
kubebuilder versionUse git to clone our project to a machine with above configuration.
git clone [ link to git repository ]Let assume that the project is cloned into closedloop directory. So we can compile and run using bellow commands. This run controller locally. In order to run on k8s, build and deploy a docker container in k8s following instructions described bellow in Running on the cluster.
cd closedloop
#Generate your CRD and a configuration file
make generate && make manifests && make install
#run controller localy
make run
#install CR instances of responsive closedloop
kubectl apply -f config/samples/closedlooppooc_v1_closedloop3.yaml
#install CR instances of deliberative closedloop
kubectl apply -f config/samples/closedlooppooc_d_v1_closedloop3.yamlAlso you need a RESTPod-Listen as proxy from managed system to closedloop, and our managed system RestSys which simple create data and send them (data-send) toRESTPod-Listen. Detailed descriprion in C) Deploy the operator image and the two Managed Systems
If you want to create you own closedloop project go to Kubebuilder ClosedLoop on Minikube
#stop AD
- Install Instances of Custom Resources:
kubectl apply -f config/samples/kubectl apply -f config/samples/WhatYouWantTo- Build image :
make docker-build IMG=controller:latest - Save image as file to then send it to minikube
docker save -o ./savedimage controller:latestthen on minikube :
docker load -i savedimage- Deploy the controller to the cluster with the image specified by
IMG:
make deploy IMG=controller:latestTo delete the CRDs from the cluster:
make uninstallUnDeploy the controller from the cluster:
make undeploy IMG=controller:latestThis project aims to follow the Kubernetes Operator pattern.
It uses Controllers, which provide a reconcile function responsible for synchronizing resources until the desired state is reached on the cluster.
- Install the CRDs into the cluster:
make install- Run your controller (this will run in the foreground, so switch to a new terminal if you want to leave it running):
make runNOTE: You can also run this in one step by running: make install run
If you are editing the API definitions, generate the manifests such as CRs or CRDs using:
make manifestsNOTE: Run make --help for more information on all potential make targets
More information can be found via the Kubebuilder Documentation
This Part explains how to reproduce this project starting from kubebuilder init (starting from scratch: initialisation of a project in kubebuilder, writing all code on your own, and running the code locally in kubebuilder, i.e., not in kubernetes cluster). Running the loop in real cluster is described in a separate section later on in this document.
I. Init your Project
username:~/closedloop$ kubebuilder init --domain closedloop.io --repo closedloop //Init folder
username:~/closedloop$ kubebuilder create api --group closedlooppooc --version v1 --kind ClosedLoop //create API and Controller
username:~/closedloop$ kubebuilder create api --group closedlooppooc --version v1 --kind Monitoring //create API and Controller
username:~/closedloop$ kubebuilder create api --group closedlooppooc --version v1 --kind Decision //create API and Controller
username:~/closedloop$ kubebuilder create api --group closedlooppooc --version v1 --kind Execution //create API and Controller
username:~/closedloop$ kubebuilder create api --group closedlooppooc --version v1 --kind Monitoringv2 //create API and ControllerII. Complet your API SPEC
Go to the folder : api/yourVersion and complete all the _types.go file to describe your CR Spec and Status
III. Generate your CRD and configuration file based on what you did on the _types.go files
username:~/closedloop$ make generate && make manifests && make installIV. Complete the logic of your controller
Complete code of the controller files in the "/controllers" folder
V. Run your Project to test it localy (This is not like in production, refer to VII)
username:~/closedloop$make runVI. Create your CR Ressources
Complete/fill in the files on /config/samples as a example and excecute the command:
username:~/closedloop$ kubectl apply -f config/samples/closedlooppooc_v1_closedloop.yaml //(Example)VII. Deploy your Operator like in production
Excecute the commands:
username:~/closedloop$ make docker-build IMG=controller:latest && docker save -o ./savedimage controller:latestFor Minikube : Transfert the savedimage file to your minikube VM and build it : example
Run From Minikube (ssh) to retreive from where your build the image
$scp Username@IP:/Path/To/savedimage ./ // Copy the file in local
$docker load -i savedimage // Load the Image in Minkube
Run on the Kubebuilder Host to Deploy your Operator, RBAC file, ..) :
username:~/closedloop$ make deploy IMG=controller:latest
VIII. Load the Proxy Pod
Run From Minikube (ssh)
$scp username@IP:/Path/To/closedloop/RESTPod-Listen/* ./ && docker build -t restpod:latest . //This will retreive and build the image needed for the proxy pod
VIV. Deploy the 2 Managed Systems
- Exporter :
Run From Minikube (ssh)
$scp username@IP:/Path/To/closedloop/exporter/* ./ && docker build -t exporter . //This will retreive and build the image needed for the exporter
Run on the Kubebuilder Host
username:~/closedloop$ kubectl create -f ./exporter/exporter.yaml //This will create the exporter
- PodToPushData to Proxy-Pod :
Run From Minikube (ssh)
$scp username@IP:/Path/To/closedloop/RESTSys/* ./ && docker build -t data-send:latest . //This will retreive and build the image needed for the POdToPushData to Proxy-Pod
Run on the Kubebuilder Host
username:~/closedloop$ kubectl create -f ./RESTSys/data-send-deployment.yaml //This will create the PodToPushData
Note: The following description contains also details of manual configuration not mentioned before. They are needed to tune the data sender (data.go application) as we do not use DNS service for local name resolution.
make undeploy IMG=controller:latestmake uninstallWe assume all code has already been provided
make generate && make manifests && make install
make docker-build IMG=controller:latest && docker save -o ./savedimage controller:latest2. ssh to minikube (you sh to the master node): create operator image and load the image; check images
scp [email protected]://home/minikube/demos/closedloop-ad/closedloop/savedimage ./
docker load -i savedimage
docker image list~/.../closedloop/make deploy IMG=controller:latestNote: PodToPushData and Proxy-Pod together correspond to (represent) one of the two managed systems while exporter represents the second managed system.
Note: PodToPushData and Proxy-Pod work together to feed respective instance of a closed loop with monitoring data (by their design and the instantiation process, both of them correspond to one common instance of closed loop). PodToPushData generates random numbers for CPU, RAM and Disk usage and sends them to the Proxy-Pod. Proxy-Pod runs Python Simple HTTP Server that receives (PUT) the requests form PodToPushData Pod and resends them to the closed loop by accessing and modifying the value of parameter Data (and also Time) in the spec section of the Monitoring Custom Resource. This custom resource represents a given instance of the closed loop. Changing the value of Data/Time parameter pair triggers the reconciliation loop of the Monitoring operator thereby propelling the whole closed loop to run.
scp [email protected]://home/minikube/demos/closedloop-ad/closedloop/RESTPod-Listen/* ./ &&
socker build -t restpod:latest .Note: exporter is a Deployment running nginx web server togehter with a Python script that cyclically generates random values for the usage of CPU, RAM and Disk and writes then into the index.html of the server. The server can then be queried (GET) for the contents of the index page. However, currently we do not use exporter in our demos.
scp [email protected]://home/minikube/demos/closedloop-ad/closedloop/exporter/* ./ && docker build -t exporter .kubectl create -f ./exporter/exporter.yaml3. prepare the image for the data-sender Pod (i.e., PodToPushData that sends data to the Proxy-Pod) and create the data-sender Pod (PodToPushData)
scp [email protected]://home/minikube/demos/closedloop-ad/closedloop/RESTSys/* ./ && docker build -t data-send:latest .Below, we create a CRB (Cluster Role Binding) to allow ProxyPod accessing (i.e., editing) the Monitoring CR (with somewhat confusing name of the CR being closedloop-v2-monitoring-xyz...).
~/demos/closedloop/RESTPod-Listen$ kubectl apply -f .kubectl apply -f config/samples/closedlooppooc_v1_closedloop3.yaml#AD
kubectl apply -f config/samples/closedlooppooc_d_v1_closedloop3.yamlkubectl logs -f -n closedloop-system closedloop-controller-manager-7d9bf7cffd-b4g7nkubectl create -f ./RESTSys/data-send-deployment.yamlTo be done each time for a newly created data-sender instance !!!
look for POST message and notice the ProxyPod service name (for DNS resolution) in the form: closedloop-v2-monitoring-deployment-service.com:80
cat data.go
ip atake note of the eth0 IP address above - this is the k8s node address to be used in the NodePort service type for the ProxyPod
(alternatively to the above, you can simply run "$ minikube ip" on the minikube/kubebuilder host)
on kubebuilder, login to the data-sender Pod to set the NodePort IP address for the ProxyPod service (remember to adjust the name of your data-send-deployment Pod)
kubectl get pods -A ==> check the name of data-sender Pod
kubectl exec --stdin --tty data-send-deployment-6c9f7dd689-qstdr -- /bin/bashin the data-sender Pod, insert a DNS entry in the hosts file (adjust the address to your environment)
nano /etc/hostsand add a line as follows:
192.168.49.2 closedloop-v2-monitoring-deployment-service.comNote: closedloop-v2-monitoring-deployment-service.com is the FQDN of the ProxyPod service as hardcoded in the data-sender Pod program. If one sets a local DNS server able to resolve that FQDN onto the minikube node IP address than the above change is not needed. Configuring the receiver of the monitoring data is always specific and can be troublesome. Future work could focus on integrating with Prometheus, etc. But for now we are fine with workarounds as the one above.
For visibility reasons, it is recommended to open 3 terminals of k9s and in each of them observe (Ctrl-D) the spec section of the custom resource Monitoring2, Decision and Excecution, respectively. One then will be able to easily trace the change of the spec properties involved in the message flow. Leverage on the use of Kubernetes ecosystem tools!
~/demos/closedloop/$ kubectl apply -f ./RESTPod-Listen/kubectl exec --stdin --tty data-send-deployment-6c9f7dd689-qstdr -- /bin/bashgo run projects/data.go#AD
Note: single run of data.go results in sending a single "mesurement" message. Emulating a stream of mesurement messages requires multiple runs of data.go.
go run projects/data.go [cpu] [memory] [disk]Use kubectl or k9s tool. Display available CRs
kubectl get crd
Next display CR which you want to see, for example:
kubectl -o yaml get monitoringv2s.closedlooppooc.closedloop.ioCopyright 2023.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Note: In this part of the guide, the steps illustrating the operation of the reactive and deliberative loops are described. The demo uses loop instances created as described in the previous part of this guide. Actually, two separate runs are presented: PART 1: for the reactive loop running in isolation, and PART 2: for the reactive and deliberative loops running in parallel and interacting with each other so that the deliberative component monitors the reactive loop and tunes its parameters according to deliberative loop policy. We remind that acording the the model of our loops described in the report (see a separate Orange-internal document) the parameters of the deliberative loop policy for a particular instance of that loop are specified in the corresponding CRD while the logic of the policy of the loop is hardcoded in source code of the deliberative loop operator.
The workflow of operations within the demo is presented in the figure below. The steps of the workflow are marked with consecutive integers, each step comprising one or two "operations" (symbolically represented as "messages" exchanged between particular functional blocks). The figure given below covers a complete demo workflow, i.e., two loops running in parallel. However, the actions relevant to PART 1 (isolated reactive loop) and PART 2 (both loops are interworking) are easily distinguishable in the figure and we will refer to respective steps in the descriptions that follows.
This run illustrates the basic worflow within a loop. In particular, one can observe how particular modules engaged in the loop exchange information by modifying dedicated parts of appropriate custom resource (CR). Each change of the CR invokes respective reconciler that executes appropriate logic of a given function of the loop.
The Managed System is modelled by the data-sender being a containerized application that sends a report to the Proxy-Pod. This report contains a triplet describing the usage of resources. As shown in the figure below, the triplet is now equal to {CPU:2 Memory:40 Disk:196}. (Note: running data.go application is described in section (H) of the CONSOLIDATED ACTION SET section above.) The containerized app data.go (models the Managed System) always sends a triplet, but in this demo only the tuple CPU and Mem is used; Disk is ignored as early as in the Proxy-Pod). Physically, this report is sent as a REST message to the Proxy-Pod, and in our case Proxy-Pod has forwarded a tuple {CPU=2 Mem=40} to the Monitoring function. All those operations are covered by step 1 in the figure.
The latter step is confirmed by checking the values (CPU:2, Memory:40) in Spec.Data of Monitoring CR (Custom Resource). This is shown in the screenshot presented below. Additionally, Monitoring operator complements the data tuple with a tag containing the time of receiving the message from Proxy-Pod. This can be seen in the screenshot as parameter Spec.Time with the value 2023-11-27 11:1:32.386296 which in subsequent steps will be passed on and recorded in all CRs taking part in this instance on the loop. This time tag serves as a unique identifier of message instance so that it is always possible to distinguish between different message instances even if the data carried in those messages (e.g., CPU/Mem/Disk usage) is the same.
Now consider the values received in the context of the monitoring policy applicable in our closed loop. In this case Cpu=2 which is lower than the CPU threshold set to 5 (see MonitorinData-1-thresholdvalue: 5 in the figuge below), and Memory=40 is compared to the memory threshold 50 (MonitorinData-2-thresholdvalue: 50). In our convention, MonitorinData-x-thresholkind: inferior in the monitoring policy means that if, e.g., cpu<MonitorinData-1-thresholdvalue then the state of CPU is considered to be "Low"; similar interpretation applies to Memory.
According to the interpretation of the thresholds and the comparison conventions of the monitoring policy described above, obtaining the values of resource usage Cpu=2 and Memory=40 results in sending a notification form the Monitoring to the Decision function indicating "Low cpu, Low memory". This can be confirmed by inspecting the value of the field Spec.Message in the Decision CR - see the figure below.
The value PriorityRank.rank-1: cpu (flull name Spec.Decisionpolicies.Priorityspec.Priorityrank.rank-1: cpu) in Decision CR shown above indicates that Cpu has higher rank than Memory. Therefore, as both resource types have been reported as being in shotage ("Low" state for both of them) and only one can be scaled in a given iteration of the loop, it is CPU that is going to be scaled this time (see our Report for more detailed description of the scaling algorithm). Accordingly, "React to cpu" message is sent to Execution which can be verified by inspecting the value of parameter Spec.Action in the Execution CR - see the figure below.
Step 4 belongs to the deliberative loop and will be referred to in PART 2 below.
Step 5 and step 6 cover subsequent reaction of Execution to receiving "React to cpu" request from Decision. In the demo, step 5 in Execution does not lead to sending explicit requestr/control message to the Managed System. As we mentioned in the Report, the demo focuses on showing the basic mechanisms feedback operations in the reactive loop have not been implemented (that's also why the data sender application data.go has to be triggered manually to emulate sending consecutive monitoring reports).
This run illustrates the worflow in a setup where a deliberative loop monitors the operation of a reactive loop and in case when the reactive loop does not behave according to the expectation of (policy imposed by) the deliberative component that the latter . In particular, one can observe how particular modules engaged in the loop exchange information by modifying dedicated parts of appropriate custom resource (CR). The steps of each component (reactive loop, deliberative loop) are interleaved in such an order so that the presentation reflects the real flow of operations in the most natural way.
Naming convention: The components of the deliberative loop in our example cooperate based on similar pronciples as in the reactive loop described above. Therefore, the whole structure of the loop is replicated after the reactive loop. In the implementation, all components of this loop are named adding suffix "D" at the end on respective name.
In the following figure, we informatively show the master CR of the reactive loop. Important in the context of current experiment are fields Spec.Decisionpolicies.Priorityspec.Priorityrank.rank-1: cpu and Spec.Decisionpolicies.Priorityspec.Priorityrank.rank-2: memory. They will be subject to changes based on the decision of the deliberative loop. The latter is a new component making the whole setup more "autonomous" (by observing and tunning the operation of the reactive loop). We will observe the changes of those fields in the course of loop operation.
Starting the experiment: similary to the previous case of isolated reactive loop, the tuple CPU:5 Memory:28 is sent to Monitoring. This is shown in the figure below.
Because this time only emory is below respective threshold (memory shortage), the message "Low memory" is sent to Decision, which is confirmed by inspecting the field Spec.Message in the Decision CR whose value is set to "Low memory" - see the figure below (we skipped Monitoring CR to shorten the presentation).
In reaction to receiving "Low memory", Decision sends "React to memory" notification to Execution. This can be confirmed by inspecting the valu of parameter |Spec.ction: React to memory in Executiuon CR - see the figure below. Also, and according to the detailed description of the deliberative loop operation form section 5 of the Report, in parallel to triggering Execution, the indication of the scaled resource Metric=memory is sent by Decision to MonitoringD that runs in the deliberative loop. This will be shown in the next figure.
In the MonitoringD CR (see the figure below), we can now check the value Spec.Data.Metric: memory - this is what MonitoringD has just received from the reactive loop component Decision. Also, in the field Spec.Time, the value 2023-12-01 21:18:35.936615 received from the reactive loop has been stored (again, it identifies a message, but also certain "threat" in the loop operation). This value (the tag) will next be sent to DecisionD where it will be also saved in a list Spec.Data.Memory containing memory scaling times. - We will see that in the next figure that follows.
Here we see a normal work of a deliberate closedloop. Because execution realizes "React to cpu", metric cpu is send to a deliberate closedloop to MonitoringD CR.
Metric cpu is written to MonitoringD, but there is not decision yet in DecisionD, so Not metric is set in ExecutionD and nothing is send to a master reactive Closedloop
Now a new data is processed by reactive closedloop. React to cpu is triggered en metric cpu is send to a deliberative MonitoringD
But now a new time of metric cpu is sufficient to take a decision by DecisionD and in ExecutionD is Spec.Action=Increase rank and Spec.Metric=memory. This values are send to master reactive ClosedLoop
And we see that Status.Increaserank=memory in ClosedLoop. This value is propagated to Monitoring and Decision for changing priority policy.
Here we see a work of both closedloop with a new priority policy set.
