Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEA] Generate an output file with runtime information #699

Closed
Tracked by #697
amahussein opened this issue Dec 19, 2023 · 0 comments · Fixed by #705
Closed
Tracked by #697

[FEA] Generate an output file with runtime information #699

amahussein opened this issue Dec 19, 2023 · 0 comments · Fixed by #705
Assignees
@amahussein
Labels
core_tools Scope the core module (scala) feature request New feature or request

Comments

@amahussein
Copy link
Collaborator

amahussein commented Dec 19, 2023 •
edited
Loading

Is your feature request related to a problem? Please describe.

It is will be helpful if the Qualification/Profiling generates a runtime.info file that contains information about the execution. This will help us troubleshooting and identifying features related to the releases.

Some of the information may include:

  • The user-tools jar version
  • The plugin jar-version (if any)
  • Spark Runtime version loaded in classPath
  • java Runtime version
  • Scala runtime verion

Describe the solution you'd like

Generate a file that lists runtime properties of the tools.

Describe alternatives you've considered

  • We cannot rely on the file-name of the jar to extract this information as the file name can change.
  • Logging the runtime information without being a part of the generated file is not practical since the output files are stored and can be used for further processing.

Additional context

This is one of the requirements in #697
@amahussein amahussein added feature request New feature or request ? - Needs Triage core_tools Scope the core module (scala) labels Dec 19, 2023
@amahussein amahussein self-assigned this Dec 19, 2023
This was referenced Dec 19, 2023
Closed
Closed
amahussein added a commit to amahussein/spark-rapids-tools that referenced this issue Dec 26, 2023
@amahussein
Generate an output file with runtime and build information
90f2997 
Signed-off-by: Ahmed Hussein (amahussein) <[email protected]>

Fixes NVIDIA#699

This PR is to dump the runtime/build information of the tools jar to the
output folder. By knowing which Tools version has been used and which
Spark version has been used in the runtime, developers can have more
insights about the output and the numbers generated by the Tools.

For Qualification:

- `rapids_4_spark_qualification_output/runtime.properties`

For Profiling:

- `rapids_4_spark_profile/runtime.properties`

A sample of the generated file is as follows:

Notice that

- `runtime.spark.version` is the spark version loaded
during runtime, while `build.spark.version` is the version used to build
the tools jar.

- `build.verion` represents the Tools jar version

```
build.hadoop.version=3.3.6
build.java.version=1.8.0_322
build.scala.version=2.12.15
build.spark.version=3.1.1
build.version=23.10.2-SNAPSHOT
runtime.spark.version=3.3.3
```
@amahussein amahussein mentioned this issue Dec 26, 2023
Merged
amahussein added a commit that referenced this issue Jan 8, 2024
@amahussein
Generate an output file with runtime and build information (#705)
dedc8d7 
* Generate an output file with runtime and build information


Fixes #699

This PR is to dump the runtime/build information of the tools jar to the
output folder. By knowing which Tools version has been used and which
Spark version has been used in the runtime, developers can have more
insights about the output and the numbers generated by the Tools.

For Qualification: `rapids_4_spark_qualification_output/runtime.properties`

For Profiling: `rapids_4_spark_profile/runtime.properties`

A sample of the generated file is as follows:

Notice that

- `runtime.spark.version` is the spark version loaded
during runtime, while `build.spark.version` is the version used to build
the tools jar.

- `build.verion` represents the Tools jar version

```
build.hadoop.version=3.3.6
build.java.version=1.8.0_322
build.scala.version=2.12.15
build.spark.version=3.1.1
build.version=23.10.2-SNAPSHOT
runtime.spark.version=3.3.3
```

---------

Signed-off-by: Ahmed Hussein (amahussein) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@amahussein amahussein

Labels
core_tools Scope the core module (scala) feature request New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Generate an output file with runtime and build information amahussein/spark-rapids-tools
2 participants
@mattahrens @amahussein