AAW Dev: Resource Utilization

[Jose-Matsuda]

Track issues related to resource utilization in AAW dev in order to minimize cloud costs. This includes investigation and action items for them.

The couple teams we have to involve or have involved are;
FinOps (has been reached out to by me and we have a machine size to go off of, Standard_D2ds_v5
CNS so we can modify our terraform (we can do this already, we do not need to reach out anymore)

The general steps will be as follows

1. Investigate, we want to baseline costs, investigate what our current node usage is at, investigate daemonsets, investigate machine types that we can migrate to.
2. Re-size workloads. Change requests for those that have been specified (there's no need to add requests if not specified) to something that better reflects its actual usage (use grafana). This will need to be broken down per nodepool
3. Re-size nodepools. We need to resize nodepools to use machines that better reflect our usage
4. Review Node selection logic: are pods being scheduled where they shouldn't be?
5. Investigate Shutting down node pools: shut down machines when not in use

Issues

• AAW Infra: scale down gerenal nodepool #1965
• AAW Dev: Determine what machine types dev nodes are on and re-size if necessary. #1993
• Inspect Grafana dashboards #1990
• AAW Dev: Re-size workloads scheduled on system nodepools #1992
• AAW Dev: Re-size workloads scheduled on general nodepools #1997
• AAW dev: change requests for default notebooks  #2000
• AAW dev: system nodepools track resource usage #2001
• AAW dev: general nodepools track resource usage #2002
• AAW Dev: Review node selection logic #2003
• AAW dev: cloudmainsys nodepools track resource usage #2004