Update the config doc

And make the uvm config of Python internal.

docs/configs.md

@@ -38,11 +38,10 @@ Name | Description | Default Value
 <a name="memory.pinnedPool.size"></a>spark.rapids.memory.pinnedPool.size|The size of the pinned memory pool in bytes unless otherwise specified. Use 0 to disable the pool.|0
 <a name="memory.uvm.enabled"></a>spark.rapids.memory.uvm.enabled|UVM or universal memory can allow main host memory to act essentially as swap for device(GPU) memory. This allows the GPU to process more data than fits in memory, but can result in slower processing. This is an experimental feature.|false
 <a name="python.concurrentPythonWorkers"></a>spark.rapids.python.concurrentPythonWorkers|Set the number of Python worker processes that can execute concurrently per GPU. Python worker processes may temporarily block when the number of concurrent Python worker processes started by the same executor exceeds this amount. Allowing too many concurrent tasks on the same GPU may lead to GPU out of memory errors. >0 means enabled, while <=0 means unlimited|0
-<a name="python.gpu.enabled"></a>spark.rapids.python.gpu.enabled|Enable (true) or disable (false) the support of running Python Pandas UDFs on the GPU. When enabled, Pandas UDFs can call cuDF APIs for acceleration. This is an experimental feature.|false
-<a name="python.memory.gpu.allocFraction"></a>spark.rapids.python.memory.gpu.allocFraction|The fraction of total GPU memory that should be initially allocated for pooled memory for all the Python workers. It supposes to be less than (1 - $(spark.rapids.memory.gpu.allocFraction)), since the executor will share the GPU with its owning Python workers.|None
-<a name="python.memory.gpu.maxAllocFraction"></a>spark.rapids.python.memory.gpu.maxAllocFraction|The fraction of total GPU memory that limits the maximum size of the RMM pool for all the Python workers. It supposes to be less than (1 - $(spark.rapids.memory.gpu.maxAllocFraction)), since the executor will share the GPU with its owning Python workers. when setting to 0 means no limit.|0.0
-<a name="python.memory.gpu.pooling.enabled"></a>spark.rapids.python.memory.gpu.pooling.enabled|Should RMM in Python workers act as a pooling allocator for GPU memory, or should it just pass through to CUDA memory allocation directly.|None
-<a name="python.memory.uvm.enabled"></a>spark.rapids.python.memory.uvm.enabled|Similar with `spark.rapids.python.memory.uvm.enabled`, but this conf is for python workers. This is an experimental feature.|None
+<a name="python.gpu.enabled"></a>spark.rapids.python.gpu.enabled|This is an experimental feature and is likely to change in the future. Enable (true) or disable (false) support for scheduling Python Pandas UDFs with GPU resources. When enabled, pandas UDFs are assumed to share the same GPU that the RAPIDs accelerator uses and will honor the python GPU configs|false
+<a name="python.memory.gpu.allocFraction"></a>spark.rapids.python.memory.gpu.allocFraction|The fraction of total GPU memory that should be initially allocated for pooled memory for all the Python workers. It supposes to be less than (1 - $(spark.rapids.memory.gpu.allocFraction)), since the executor will share the GPU with its owning Python workers. Half of the rest will be used if not specified|None
+<a name="python.memory.gpu.maxAllocFraction"></a>spark.rapids.python.memory.gpu.maxAllocFraction|The fraction of total GPU memory that limits the maximum size of the RMM pool for all the Python workers. It supposes to be less than (1 - $(spark.rapids.memory.gpu.maxAllocFraction)), since the executor will share the GPU with its owning Python workers. when setting to 0 it means no limit.|0.0
+<a name="python.memory.gpu.pooling.enabled"></a>spark.rapids.python.memory.gpu.pooling.enabled|Should RMM in Python workers act as a pooling allocator for GPU memory, or should it just pass through to CUDA memory allocation directly. When not specified, It will honor the value of config 'spark.rapids.memory.gpu.pooling.enabled'|None
 <a name="shuffle.transport.enabled"></a>spark.rapids.shuffle.transport.enabled|When set to true, enable the Rapids Shuffle Transport for accelerated shuffle.|false
 <a name="shuffle.transport.maxReceiveInflightBytes"></a>spark.rapids.shuffle.transport.maxReceiveInflightBytes|Maximum aggregate amount of bytes that be fetched at any given time from peers during shuffle|1073741824
 <a name="shuffle.ucx.managementServerHost"></a>spark.rapids.shuffle.ucx.managementServerHost|The host to be used to start the management server|null

sql-plugin/src/main/scala/com/nvidia/spark/rapids/python/PythonConfEntries.scala

@@ -16,14 +16,16 @@
 
 package com.nvidia.spark.rapids.python
 
+import com.nvidia.spark.rapids.RapidsConf.{POOLED_MEM, UVM_ENABLED}
 import com.nvidia.spark.rapids.RapidsConf.conf
 
 object PythonConfEntries {
 
   val PYTHON_GPU_ENABLED = conf("spark.rapids.python.gpu.enabled")
-    .doc("Enable (true) or disable (false) the support of running Python Pandas UDFs" +
-      " on the GPU. When enabled, Pandas UDFs can call cuDF APIs for acceleration." +
-      " This is an experimental feature.")
+    .doc("This is an experimental feature and is likely to change in the future." +
+      " Enable (true) or disable (false) support for scheduling Python Pandas UDFs with" +
+      " GPU resources. When enabled, pandas UDFs are assumed to share the same GPU that" +
+      " the RAPIDs accelerator uses and will honor the python GPU configs")
     .booleanConf
     .createWithDefault(false)
 
@@ -40,7 +42,7 @@ object PythonConfEntries {
     .doc("The fraction of total GPU memory that should be initially allocated " +
       "for pooled memory for all the Python workers. It supposes to be less than " +
       "(1 - $(spark.rapids.memory.gpu.allocFraction)), since the executor will share the " +
-      "GPU with its owning Python workers.")
+      "GPU with its owning Python workers. Half of the rest will be used if not specified")
     .doubleConf
     .checkValue(v => v >= 0 && v <= 1, "The fraction value for Python workers must be in [0, 1].")
     .createOptional
@@ -49,21 +51,24 @@ object PythonConfEntries {
     .doc("The fraction of total GPU memory that limits the maximum size of the RMM pool " +
       "for all the Python workers. It supposes to be less than " +
       "(1 - $(spark.rapids.memory.gpu.maxAllocFraction)), since the executor will share the " +
-      "GPU with its owning Python workers. when setting to 0 means no limit.")
+      "GPU with its owning Python workers. when setting to 0 it means no limit.")
     .doubleConf
     .checkValue(v => v >= 0 && v <= 1, "The value of maxAllocFraction for Python workers must be" +
       " in [0, 1].")
     .createWithDefault(0.0)
 
   val PYTHON_POOLED_MEM = conf("spark.rapids.python.memory.gpu.pooling.enabled")
     .doc("Should RMM in Python workers act as a pooling allocator for GPU memory, or" +
-      " should it just pass through to CUDA memory allocation directly.")
+      " should it just pass through to CUDA memory allocation directly. When not specified," +
+      s" It will honor the value of config '${POOLED_MEM.key}'")
     .booleanConf
     .createOptional
 
   val PYTHON_UVM_ENABLED = conf("spark.rapids.python.memory.uvm.enabled")
-    .doc("Similar with `spark.rapids.python.memory.uvm.enabled`, but this conf is for " +
-      "python workers. This is an experimental feature.")
+    .doc(s"Similar with '${UVM_ENABLED.key}', but this conf is for" +
+      s" python workers. When not specified, it will honor the value of config" +
+      s" '${UVM_ENABLED.key}'. This is an experimental feature.")
+    .internal()
     .booleanConf
     .createOptional