AAW Prod: Resource Utilization

[Jose-Matsuda]

Similar to #1998

Track issues related to resource utilization in AAW prod in order to minimize cloud costs. This includes investigation and action items for them.

The general steps will be as follows

Investigate, we want to baseline costs, investigate what our current node usage is at, investigate daemonsets, investigate machine types that we can migrate to.
Re-size workloads. Change requests for those that have been specified (there's no need to add requests if not specified) to something that better reflects its actual usage (use grafana). This will need to be broken down per nodepool
Re-size nodepools. We need to resize nodepools to use machines that better reflect our usage, can we downsize some to say Standard_D2ds_v5?
Review Node selection logic: are pods being scheduled where they shouldn't be?
Investigate Shutting down node pools: shut down machines when not in use

Issues

• AAW Prod: Script getting resources by a label or annotation #2009
• AAW Prod: system nodepool find resources we manage #2006
• AAW Prod: system nodepool suggest new requests for resources #2005
• AAW Prod: System nodepool resize workloads identified #2007
• AAW Prod: general nodepool list pods we manage  #2008