Poor Inference Parallelism From Different JVMs On Same Host

[malcolm-mccarthy]

Hello, we are running multiple copies of a Java application on the same host that use DJL/PyTorch/TorchScript to generate inferences. We had expected to see good parallel behaviour for the inferences as the JVMs are independent of one another. However, the odd thing is that we are seeing behaviour that is consistent with there being some sort of blocking activity within DJL/PyTorch/TorchScript that causes the inference calls from different JVMs to stack up behind each other.

We are running on Red Hat 8 with the following grade

implementation "ai.djl.pytorch::pytorch-engine:0.16.0"
implementation "ai.djl.pytorch::pytorch-jni:1.10.0-0.16.0"
implementation "ai.djl.pytorch::pytorch-native-cpu-precxx11:1.10.0:linux-x86_64"

[lanking520]

Thanks for your reporting. Could you share any code snippet for your setup? And share any stack or flamechart of your JVM process.

Please also feel free to email [email protected] to schedule any meeting with us to help you unblock.

[malcolm-mccarthy]

Setting the num_threads and num_interop_threads to 1 looks promising so am going to continue down that route for now

[frankfliu]

@malcolm-mccarthy
First of all, you don't have to use multiple JVM just to improve the throughput. Not like python, in JVM you can run multi-threading inference. Multiple process will have the same or less performance compare to multi-threading.

Secondly, if you run inference parallel, you need to tune your pytorch thread pool to avoid thread contention (even in multi-process case), see: https://docs.djl.ai/master/docs/development/inference_performance_optimization.html#thread-configuration
We recommend you to set:

System.setProperty("ai.djl.pytorch.num_threads", "1");
System.setProperty("ai.djl.pytorch.num_interop_threads", "1");

[malcolm-mccarthy]

I have tried this and it looks very promising - the separate JVMs appear to be more or less independent and the performance is good. I will continue to do further testing.