Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for disk spill metrics in Photon jobs #1405

Conversation

parthosa
Copy link
Collaborator

@parthosa parthosa commented Oct 29, 2024

Issue #251

Following up on #1390, this PR adds parsing for Disk Spill Size metrics for Photon Jobs.

  • Photon event logs report disk spill size using the metric name "num bytes spilled to disk".
  • Similar to 1390, we now calculate the disk spill size using accumulators instead of task metrics for Photon jobs.
  • Note: Could not find any metrics for in-memory spills in Photon event logs.

Changes

Enhancements to Photon-specific metrics handling:

Test expectation updates:

@parthosa parthosa added bug Something isn't working core_tools Scope the core module (scala) labels Oct 29, 2024
@parthosa parthosa self-assigned this Oct 29, 2024
@parthosa parthosa marked this pull request as ready for review October 29, 2024 22:29
@parthosa
Copy link
Collaborator Author

This PR is no more required since spill metrics are parsed correctly by default.

@parthosa parthosa closed this Oct 30, 2024
@parthosa parthosa deleted the spark-rapids-tools-add-spill-metrics-photon branch October 30, 2024 00:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working core_tools Scope the core module (scala)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant