Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding GC Metrics #12

Merged
merged 5 commits into from
Jul 5, 2024
Merged

Adding GC Metrics #12

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f557976
Adding GC Metrics
bilalbari Jul 2, 2024
4eb9997
Review comment changes
bilalbari Jul 3, 2024
f1664df
Correcting output format + refactoring
bilalbari Jul 3, 2024
58a7d9f
Output Formatting Changes
bilalbari Jul 5, 2024
94f7f93
Formatting + Making qual bench single threaded
bilalbari Jul 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 25 additions & 6 deletions core/src/main/scala/org/apache/spark/rapids/tool/benchmarks/Benchmark.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,9 @@
package org.apache.spark.rapids.tool.benchmarks

import java.io.{OutputStream, PrintStream}
import java.lang.management.ManagementFactory

import scala.collection.convert.ImplicitConversions.`collection AsScalaIterable`
import scala.collection.mutable
import scala.collection.mutable.ArrayBuffer
import scala.concurrent.duration.NANOSECONDS
Expand Down Expand Up @@ -101,20 +103,30 @@ class Benchmark(
// The results are going to be processor specific so it is useful to include that.
out.println(RuntimeUtil.getJVMOSInfo.mkString("\n"))
val nameLen = Math.max(40, Math.max(name.length, benchmarks.map(_.name.length).max))
out.printf(s"%-${nameLen}s %14s %14s %11s %10s\n",
name + ":", "Best Time(ms)", "Avg Time(ms)", "Stdev(ms)", "Relative")
out.printf(s"%-${nameLen}s %14s %14s %11s %10s %10s %10s\n",
name + ":", "Best Time(ms)", "Avg Time(ms)", "Stdev(ms)","Max GC Time(ms)",
"Max GC Count", "Relative")
out.println("-" * (nameLen + 80))
results.zip(benchmarks).foreach { case (result, benchmark) =>
out.printf(s"%-${nameLen}s %14s %14s %11s %10s\n",
out.printf(s"%-${nameLen}s %14s %14s %11s %10s %10s %10s\n",
benchmark.name,
"%5.0f" format result.bestMs,
"%4.0f" format result.avgMs,
"%5.0f" format result.stdevMs,
"%8d" format result.maxGcTime,
"%5d" format result.maxGcCount,
"%3.1fX" format (firstBest / result.bestMs))
}
out.println()
}

private def getGcMetrics: (Any, Any) = {
val gcBeans = ManagementFactory.getGarbageCollectorMXBeans
val gcTime = gcBeans.map(_.getCollectionTime).sum
val gcCount = gcBeans.map(_.getCollectionCount).sum
(gcTime, gcCount)
}

Copy link
Owner

@amahussein amahussein Jul 3, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we move this to the RuntimeUtil, then we can use any where as a utility function regardless it is part of the benchmark or not.
Also, we can return a case class


case class JVMGCInfo(val gcCount: Long, val gcTime: Long, val: totalHeap: Long, val usedHeap: Long, val freeHeap: Long)

After we do that we can also define another object to manage snapshots (similar to the timer)
But we probably need another manager to handle start/end to return the diff instead of flatenning the code. We can do the manager class together after you update the PR.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added changes to handle the above. Added memory tracker class. Also added tracking of average used memory, average free memory and max heap allocation

/**
* Runs a single function `f` for iters, returning the average time the function took and
* the rate of the function.
Expand All @@ -127,17 +139,23 @@ class Benchmark(
val minIters = if (overrideNumIters != 0) overrideNumIters else minNumIters
val runTimes = ArrayBuffer[Long]()
var totalTime = 0L
var maxGcCount = 0L
var maxGcTime = 0L
for (i <- 0 until minIters) {
val timer = new ToolsTimer(i)
val (gcTimeBefore, gcCountBefore) = getGcMetrics
f(timer)
val runTime = timer.totalTime()
runTimes += runTime
totalTime += runTime

val (gcTimeAfter, gcCountAfter) = getGcMetrics
maxGcTime = math.max(maxGcTime, gcTimeAfter.asInstanceOf[Long] - gcTimeBefore.asInstanceOf[Long])
maxGcCount = math.max(maxGcCount, gcCountAfter.asInstanceOf[Long] - gcCountBefore.asInstanceOf[Long])
if (outputPerIteration) {
// scalastyle:off
println("*"*80)
println(s"Iteration $i took ${NANOSECONDS.toMicros(runTime)} microseconds")
println(s"Iteration $i GC time ${gcTimeAfter.asInstanceOf[Long] - gcTimeBefore.asInstanceOf[Long]}")
println("*"*80)
// scalastyle:on
}
Expand All @@ -153,12 +171,13 @@ class Benchmark(
val stdev = if (runTimes.size > 1) {
math.sqrt(runTimes.map(time => (time - avg) * (time - avg)).sum / (runTimes.size - 1))
} else 0
Benchmark.Result(avg / 1000000.0, best / 1000000.0, stdev / 1000000.0)
Benchmark.Result(avg / 1000000.0, best / 1000000.0, stdev / 1000000.0, maxGcTime, maxGcCount)
}
}


object Benchmark {
case class Case(name: String, fn: ToolsTimer => Unit, numIters: Int)
case class Result(avgMs: Double, bestMs: Double, stdevMs: Double)
case class Result(avgMs: Double, bestMs: Double, stdevMs: Double,
maxGcTime: Long = 0, maxGcCount: Long = 0)
}