Use environment variables to set thresholds in static yaml configurations

[amahussein]

Signed-off-by: Ahmed Hussein [email protected]

Fixes #1387

This code change aims at parsing yaml configuration file and resolving the environment variables.

• spillThresholdBytes defined in qualification-conf.yaml can be overridden by an env-var RAPIDS_USER_TOOLS_SPILL_BYTES_THRESHOLD.
• totalCoreSecThreshold defined in qualification-conf.yaml can be overridden by an env-var RAPIDS_USER_TOOLS_CORE_SECONDS_THRESHOLD.

These changes improve the flexibility and maintainability of the configuration management in the user_tools package.

Usage:

export RAPIDS_USER_TOOLS_CORE_SECONDS_THRESHOLD=1024
spark_rapids <args>

Code changes

This pull request introduces several changes to the user_tools package, mainly focusing on dependency updates and configuration management enhancements. The key changes include updating dependencies in pyproject.toml, integrating the pyaml_env library for environment variable parsing in YAML files, and modifying configuration files to support environment variable substitution.

Documentation Update

Filed an issue internally to add the new env_variables to the documentation

Dependency updates and integration:

• user_tools/pyproject.toml: Updated the pyYAML dependency to a version range and added pyaml_env to handle environment variables in YAML files.

Code changes for environment variable parsing:

• user_tools/src/spark_rapids_pytools/common/prop_manager.py: Imported parse_config from pyaml_env and replaced yaml.safe_load with parse_config in the __open_yaml_file method. [1] [2]
• user_tools/src/spark_rapids_tools/utils/propmanager.py: Imported parse_config from pyaml_env and replaced yaml.safe_load with parse_config in the load_yaml function. [1] [2]

Configuration file enhancements:

• user_tools/src/spark_rapids_pytools/resources/qualification-conf.yaml: Modified configuration parameters to support environment variable substitution using the !ENV tag. [1] [2]

[amahussein]

CC: @viadea FYI to control spillThreshold or coreSeconds in a specific environment.

[tgravescs]

overall looks good. are we documenting these for user, or mean for more advanced configuration?

[amahussein]

overall looks good. are we documenting these for user, or mean for more advanced configuration?

It is meant for advanced configuration.
We have an internal issue opened to document all the environment-variables. Thus, if users need specific tuning for their environment we can point them to the env-variables.

[parthosa]

Thanks @amahussein. This is an interesting approach. During runtime, we can configure our configs.

This will also be helpful if we want to configure other thresholds in future (such as Top Candidate cut-off speedup etc)