[FEA] Adjust the spill size heuristic rule

[viadea]

Is your feature request related to a problem? Please describe.
Current spill size heuristic rule is 10G and it can be changed by env var RAPIDS_USER_TOOLS_SPILL_BYTES_THRESHOLD .
This does not consider the factor that how many worker nodes and how many CPU cores this CPU job uses.
The more resources(especially the more SSDs used by each executor), the negative impact of the spill is less.

Describe the solution you'd like
Use the Total_Spill_Size/Total_number_of_CPU_COREs > certain_threashold as the new rule.
Eg. If the per_CPU_CORE_spill is >1GB, then we disqualify this job.
(Of course, if you have better algorithm, feel free to share)

[amahussein]

Possibly duplicate of #1344
@viadea which proposal do you want to keep?

[tgravescs]

It would be good to have eventlog or more information where the current heuristic didn't work?

agree there are a lot of factors in this, its impossible for us to know exactly what the GPU will do so the more characteristics we can get of jobs we see where it is an issue and not an issue will help us tune the heuristic