README.md

Lab: Adding Prometheus and Grafana to AKS Cluster

This lab will walkthrough using the Core OS Prometheus Operator to add Monitoring and Visualization capabilities to our AKS Cluster. The Operator will be installed using HELM.

Prerequisites

• Complete previous labs:
  • Azure Kubernetes Service
  • Build Application Components in Azure Container Registry
  • Helm Setup and Deploy Application

Instructions

1. Deploy Prometheus Operator

   # Go to the prometheus lab directory
   cd ~/kubernetes-hackfest/labs/monitoring-logging/prometheus-grafana
   
   # Create a new Monitoring Namespace to deploy Prometheus Operator too
   kubectl create namespace monitoring
   
   # Add the stable repo for Helm 3
   helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
   helm repo add stable https://charts.helm.sh/stable
   helm repo update
   
   # Install Prometheus Operator
   helm install prometheus-operator prometheus-community/kube-prometheus-stack -f values.yaml --namespace monitoring
   kubectl -n monitoring get all -l "release=prometheus-operator"
   
   # Check to see that all the Pods are running
   kubectl get pods -n monitoring
   
   # Other Useful Prometheus Operator Resources to Peruse
   kubectl get prometheus -n monitoring
   kubectl get prometheusrules -n monitoring
   kubectl get servicemonitor -n monitoring
   kubectl get cm -n monitoring
   kubectl get secrets -n monitoring

Note: The following explains how to expose Prometheus, Alert Manager and Grafana dashboards via a public IP. This is only for lab purposes, and would not be recommended for production. A more secure alternative would be to use a Kubernetes port-forward. (Ex. kubectl port-forward services/prometheus-operator-prometheus -n monitoring 9090:9090)

1. Expose Services to Public IP's

   # use your VI skills to change the below snippet. The type should be "LoadBalancer" and not "ClusterIP"
   
   kubectl edit service prometheus-operator-prometheus -n monitoring

   spec:
     clusterIP: 10.0.79.78
     ports:
     - name: http
       port: 9090
       protocol: TCP
       targetPort: 9090
     selector:
       app: prometheus
       prometheus: kube-prometheus
     sessionAffinity: None
     type: LoadBalancer

   # repeat for Alert Manager
   kubectl edit service prometheus-operator-alertmanager -n monitoring

   # repeat for Grafana
   kubectl edit service prometheus-operator-grafana -n monitoring

   Note: These settings should not generally be used in production. The endpoints should be secured behind an Ingress. This just aides the lab experience.

2. Interact with Prometheus (Prometheus and Alert Manager Dashboards)

   # Get your public IP address for the Prometheus dashboard (if <pending>, you must wait...)
   kubectl get service prometheus-operator-prometheus -n monitoring

   Open up a brower to http://:9090 and you will see the Prometheus dashboard

   • Screenshot of Default Prometheus UI

     

   # Get your public IP address for the Prometheus Alert Manager (if <pending>, you must wait...)
   kubectl get service prometheus-operator-alertmanager -n monitoring

   Open up a brower to http://:9093 and you will see the Prometheus dashboard

   • Screenshot of Default Alert Manager UI

     

3. Interact with Grafana Dashboard

   # Get your public IP address for Grafana (if <pending>, you must wait...)
   kubectl get service prometheus-operator-grafana -n monitoring

   Open up a brower to http://:80 and you will need to log in to see the Prometheus dashboard. Username: admin Password: prom-operator

   • Screenshot of Kubernetes Capacity Planning Dashboard

     

4. Deploy Sample App with Integrated and Custom Prometheus Metrics

   • Create Namespace for Sample GO App

     # Create custom Namespace to deploy sample app to.
     kubectl create namespace sample-app

   • Build Sample GO App Container & Update Deployment Manifest

     # 1. Use ACR Build to create Container and Push to ACR
     # 2. Update Container Image in Deployment manifest (prom-graf-sample-go-app.yaml) 
     # Deploy the Sample GO Application with Updated Container Image
     kubectl apply -f prom-graf-sample-go-app.yaml -n sample-app
     # Deploy the ServiceMonitor to Monitor the Sample GO App
     kubectl apply -f prom-graf-servicemonitor.yaml -n monitoring
     # Deploy the ConfigMap to Raise Alerts for the Sample GO App
     kubectl apply -f prom-graf-alert-rule.yaml -n monitoring

   • If there is interest in how Prometheus Metrics and Custom Metrics can be added to an existing application take a look at the GO Code.

   • Get the IP address of the sample and send some requests to it to get some metrics loaded.

     kubectl get svc -n sample-app
     
     # Either pop out to a browser or curl http://<ExternalIP>:8080
     curl -H 'Cache-Control: no-cache' <ExternalIP>:8080
     
     # To view the metrics endpoint
     curl <ExternalIP>:8080/metrics

5. Check Metrics and Alerts are Working for Sample GO App

   • Using the technique above, port-forward to the Prometheus Dashboard.

   • Check custom metric requests_counter_total in the deployed sample GO App:

     

   • Check Replica Count custom alert for the sample GO App:

     

6. Fix Replica Count Custom Alert

   • Scale the Deployment to 3 replicas to stop the Alert from FIRING.

     kubectl scale deploy sample-go -n sample-app --replicas=3

   • Using the technique above, port-forward to the Prometheus Dashboard and check that the Alert is now Green and not FIRING. Be patient, this will take a couple of minutes for the metrics to be updated and the evaluation process to respond accordingly.

     

Docs / References

• Prometheus Operator
• Crash Course to Monitoring K8s
• Prometheus Operator Alerting

Next Lab: Service Mesh w/ Distributed Tracing