Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Profiling tool throws NPE when appInfo is null and unchecked #639

Closed
kuhushukla opened this issue Oct 27, 2023 · 0 comments · Fixed by #640
Closed

[BUG] Profiling tool throws NPE when appInfo is null and unchecked #639

kuhushukla opened this issue Oct 27, 2023 · 0 comments · Fixed by #640
Assignees
Labels
bug Something isn't working core_tools Scope the core module (scala)

Comments

@kuhushukla
Copy link
Collaborator

kuhushukla commented Oct 27, 2023

Describe the bug

2023-10-26 16:26:57,410 DEBUG rapids.tools.cmd_driver: executing CMD:
	<CMD: java -Xmx24g -cp /home/kuhu/Downloads/dbfs_eventlog/prof_20231026212649_b53B14Ba/work_dir/rapids-4-spark-tools_2.12-23.08.2.jar:/home/kuhu/Downloads/dbfs_eventlog/prof_20231026212649_b53B14Ba/work_dir/hadoop-azure-3.3.4.jar:/home/kuhu/Downloads/dbfs_eventlog/prof_20231026212649_b53B14Ba/work_dir/spark-3.3.3-bin-hadoop3/jars/* com.nvidia.spark.rapids.tool.profiling.ProfileMain --output-directory /home/kuhu/Downloads/dbfs_eventlog/prof_20231026212649_b53B14Ba /home/kuhu/Downloads/dbfs_eventlog/eventlog/1024-202047-zgdzmm84_10_139_64_5/7353421840558998897/eventlog>[
	<STDOUT>
	| 23/10/26 16:26:56 INFO Profiler: Threadpool size is 16
	| 23/10/26 16:26:56 INFO ApplicationInfo: Parsing Event Log: file:/home/kuhu/Downloads/dbfs_eventlog/eventlog/1024-202047-zgdzmm84_10_139_64_5/7353421840558998897/eventlog
	| 23/10/26 16:26:56 WARN ToolUtils: ClassNotFoundException while parsing an event: DBCEventLoggingListenerMetadata
	| 23/10/26 16:26:56 WARN Utils: Your hostname, kuhu-Precision-7865-Tower resolves to a loopback address: 127.0.1.1; using 10.28.9.183 instead (on interface enp3s0)
	| 23/10/26 16:26:56 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
	| 23/10/26 16:26:57 INFO ApplicationInfo: Total number of events parsed: 1664 for file:/home/kuhu/Downloads/dbfs_eventlog/eventlog/1024-202047-zgdzmm84_10_139_64_5/7353421840558998897/eventlog
	| 23/10/26 16:26:57 INFO EventLogPathProcessor: ==============   (index=1)  ==============
	| 23/10/26 16:26:57 INFO Profiler: Took 886ms to process file:/home/kuhu/Downloads/dbfs_eventlog/eventlog/1024-202047-zgdzmm84_10_139_64_5/7353421840558998897/eventlog
	| 23/10/26 16:26:57 WARN Profiler: Exception occurred processing file: eventlog
	| java.lang.NullPointerException
	| 	at com.nvidia.spark.rapids.tool.profiling.CollectInformation.$anonfun$getAppInfo$1(CollectInformation.scala:39)
	| 	at scala.collection.immutable.List.map(List.scala:293)
	| 	at com.nvidia.spark.rapids.tool.profiling.CollectInformation.getAppInfo(CollectInformation.scala:37)
	| 	at com.nvidia.spark.rapids.tool.profiling.Profiler.com$nvidia$spark$rapids$tool$profiling$Profiler$$processApps(Profiler.scala:315)
	| 	at com.nvidia.spark.rapids.tool.profiling.Profiler$ProfileProcessThread$1.run(Profiler.scala:249)
	| 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	| 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	| 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	| 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	| 	at java.lang.Thread.run(Thread.java:750)
	| 23/10/26 16:26:57 INFO ToolTextFileWriter: Profile summary output location: /home/kuhu/Downloads/dbfs_eventlog/prof_20231026212649_b53B14Ba/rapids_4_spark_profile/profile.log
	| Profile Tool Progress 100% [======================================================] (0 succeeded + 1 failed + 0 N/A) / 1
	| 
	| Profile Tool execution time: 911ms
	| 	process.success.count = 0
	| 	process.failure.count = 1
	| 	process.NA.count = 0
	| 	execution.total.count = 1]; [
	<STDERR>
	| Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties]

Steps/Code to reproduce bug
Some eventlogs do not have the appropriate event (AppStart?) to instantiate AppInfo correctly.

Expected behavior
Not throw a null pointer exception

Environment details (please complete the following information)
I tried it on Azure but it can happen on other platforms too

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working core_tools Scope the core module (scala)
Projects
None yet
2 participants