[FEA] AutoTuner warns that non-utf8 may not support some GPU expressions #736
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Ahmed Hussein (amahussein) <[email protected]> Fixes NVIDIA#713, Fixes NVIDIA#734 This PR changes the behavior of the AutTuner to display a comment when the file-encoding of an application is set to a value that is not "utf-8". The changes also improves the extraction of the RAPIDS jars values. *Changes for 713*: - Added a new field in `ApplicationSummaryInfo` that represents the systemProperties - Capture SystemProperties in the App in order to be able to check the file-encoding - Moved map properties to `CacheableProps` so that it can be used by the Qualification as well. - Moved String-conversion methods from AutTuner to StringUtils - Moved `getEventFromJsonMethod` to EventUtils object - Added a new UnitTest for the AutoTuner - Updated ApplicationInfoSuite unitTests *Changes for 734*: - Fixed the implementation of `CollectInformation.getRapidsJARInfo`
amahussein
added
feature request
parthosa
reviewed
Jan 22, 2024
| // No need to capture all the properties in memory. We only capture important ones. | ||
| systemProperties ++= event.environmentDetails("System Properties").toMap.filterKeys( | ||
| RETAINED_SYSTEM_PROPS.contains(_)) | ||
|
|
Comment on lines
756
to
757
| case encoding => | ||
| if (!ToolUtils.isFileEncodingRecommended(encoding)) { |
There was a problem hiding this comment.
Can we move if condition to case encoding statement?
| def handleEnvUpdateForCachedProps(event: SparkListenerEnvironmentUpdate): Unit = { | ||
| val sparkProperties = event.environmentDetails("Spark Properties").toMap | ||
|
|
Signed-off-by: Ahmed Hussein (amahussein) <[email protected]>
nartal1
reviewed
Jan 22, 2024
There was a problem hiding this comment.
Overall LGTM. Just a couple of Nits. Thanks @amahussein for refactoring some of the code!
core/src/main/scala/com/nvidia/spark/rapids/tool/profiling/CollectInformation.scala
Outdated
Show resolved
Hide resolved
core/src/test/scala/com/nvidia/spark/rapids/tool/profiling/AutoTunerSuite.scala
Outdated
Show resolved
Hide resolved
Signed-off-by: Ahmed Hussein (amahussein) <[email protected]>
nartal1
approved these changes
Jan 23, 2024
This was referenced Jan 23, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Ahmed Hussein (amahussein) [email protected]
Fixes #713, Fixes #734
This PR changes the behavior of the AutoTuner to display a comment when the file-encoding of an application is set to a value that is not "utf-8".
The AutoTuner will generate the folllowing line:
The changes also improves the extraction of the RAPIDS jars values.
Changes for 713:
ApplicationSummaryInfothat represents the systemPropertiesCacheablePropsso that it can be used by the Qualification as well.getEventFromJsonMethodto EventUtils objectApplicationInfoSuiteunitTestsChanges for 734:
CollectInformation.getRapidsJARInfo