[FEA] Enable AQE related recommendations in Profiler Auto-tuner

[cindyyuanjiang]

Fixes #576

This PR added the following list of settings AQE optimization for auto-tuner:

Spark Property	Recommendation
spark.rapids.sql.batchSizeBytes	Set to just under 2GB (default is 1GB)
spark.sql.adaptive.autoBroadcastJoinThreshold	If the setting is above 100MB (default is 10MB), recommend user to set to a lower number.
spark.sql.adaptive.advisoryPartitionSizeInBytes	If Input Size > 35KB and Shuffle Read > 50KB: - For A100, set to 64MB - For T4, set to 32MB Otherwise, set to 128MB.
spark.sql.adaptive.coalescePartitions.initialPartitionNum	If Input Size > 35KB, Shuffle Read > 50KB and value < 200 (low value) - For A100, set to 400 - For T4, set to 800
spark.sql.adaptive.coalescePartitions.parallelismFirst	If Input Size > 35KB and Shuffle Read > 50KB: - Set to 'false' to prioritize 'advisoryPartitionSizeInBytes' over 'minPartitionSize' for better performance

[amahussein]

Thanks @parthosa and @cindyyuanjiang !
I made some few comments

Can you please check also how the AQE recommendations look like in the user-tools?
One of the things to keep an eye on is that the tables of the profiler stdout are readable and match the expected from the profiler core .

[cindyyuanjiang]

Filed follow up issue, tracked here: #719

[revans2]

Switching my review to approved. I saw the benchmark results and I am okay with the change now. I am a little concerned with what happens if the the number of shuffle partitions is very small, or the AQE target shuffle size is very large, or if the maxPartitionBytes is very large. But as long as we have benchmarks that we are running we can improve the benchmarks over time as we see more corner cases from customers show up.

[amahussein]

the PR has conflicts. Needs upmerge.

[cindyyuanjiang]

Switching my review to approved. I saw the benchmark results and I am okay with the change now. I am a little concerned ...

Thank you for the feedback @revans2!