Adjust node pools in data-processing clusters

[gfr10598]

The most recent change to etl k8s configs failed to launch pods in staging, because I had set the 8 core node-pool to 0 instances.

Also, in all projects, the utilization is low, because there are more nodes than needed for the requested pods.

All the pools should be updated to use appropriate auto-scaling configs.

[gfr10598]

Today I am:
changing sandbox default pool to allow 1 node per zone, and remove zone us-east1-d
changing staging 8 core parser-pool1 to allow 0-2 nodes per zone.
changing staging 4 core parser-pool to allow 0-1 nodes per zone.

Tomorrow, I intend to change prod to set up auto-scaling for parser, default, and gardener pools.

[gfr10598]

removing zone us-east1-d from the default pool in mlab-sandbox made gardener unschedulable, because of persistent volume location. Restored us-east1-d, and adjusted auto-scaling to allow 1-2 per zone.

Later changed to 0-2 per zone

[gfr10598]

Subsequently updated mlab-sandbox parser-pool1 to allow 0-2 nodes per zone as well.

[laiyi-ohlsen]

@gfr10598 can you enumerate the steps that are needed before the next gardener release to production?

[gfr10598]

I just deleted the mlab-staging parser-pool1 node pool from data-processing cluster. K8S had restarted parsers, and they were instantiated in the wrong node pool. Deleting the pool should prevent this happening again.

[gfr10598]

See m-lab/etl#985 related to propagating errors from etl to gardener.