Run plone.restapi jMeter/Locust performance tests on Plone Foundation infrastructure?

[tisto]

@fredvd @gforcada some time ago I set up jMeter and Locust performance tests on our kitconcept Jenkins infrastructure. I just realized that the tests were broken since quite some time and I lost the old data.

I'd like to set this up again because I think this gives us very valuable insights over time.

Do you see any chance to set up those tests on the existing jenkins.plone.org infrastructure?

I'd be ok with continuing to run this as my own pet project on our infrastructure. The major downside of this is that it is hard to share information and let other participate in my efforts to monitor and improve plone.restapi performance.

[fredvd]

@tisto +1 Let's see what is in the tests/setup and how easy we can move it to jenkins. We're still in the process of rebooting the CI-team and then gather feedback on the future of our CI/CD and where what runs.

But for the time being It would be at least valuable to have the test setup stored somewhere for the community and also indeed to keep an eye on plone.restapi performance. And for the 'base' / backend distribution we still rely on jenkins for all testing (i.e. on which Maurits decided to cut a release for the backend/classicUI.

I asssume for performance tests it would be nice to have consistent performance cpu/io/mem on the test runner available?

cc: @plone/ci-team

[gforcada]

I wouldn't care too much about having perfect isolation first, but rather have the setup running.

Given that we are moving, slowly, and probably never completely, to GHA, is that something that could be done there? 🤔

Otherwise, getting a jenkins job in jenkins.plone.org that checkouts plone.restapi and runs a script(?) with probably one or two extra jenkins plugins to parse the results and show them is a trivial matter if we have the script and the jenkins plugins needed.

A weekly run of it might be even enough, so we don't have to slow down the whole, already slow, jenkins jobs to run performance tests on it as well, or?

[tisto]

@fredvd @gforcada the Jenkinsfile is here:

https://github.com/plone/plone.restapi/blob/master/Jenkinsfile

Here are the install instructions for jMeter and the necessary jMeter plugins:

Install Instructions jMeter

Download jmeter 5.5:

sudo wget https://dlcdn.apache.org//jmeter/binaries/apache-jmeter-5.5.tgz
sudo tar xfvz apache-jmeter-5.5.tgz
sudo mv apache-jmeter-5.5 jmeter

Install CMDrunner:

cd /opt/jmeter/lib
sudo curl -O https://repo1.maven.org/maven2/kg/apc/cmdrunner/2.2.1/cmdrunner-2.2.1.jar

Install jmeter plugins manager

cd ext/
sudo curl -O https://repo1.maven.org/maven2/kg/apc/jmeter-plugins-manager/1.6/jmeter-plugins-manager-1.6.jar

Install bzm parallel plugin:

sudo java -jar cmdrunner-2.2.1.jar --tool org.jmeterplugins.repository.PluginManagerCMD install bzm-parallel

Install filterresults/mergeresults plugin:

sudo java -jar cmdrunner-2.2.1.jar --tool org.jmeterplugins.repository.PluginManagerCMD install jpgc-filterresults
sudo java -jar cmdrunner-2.2.1.jar --tool org.jmeterplugins.repository.PluginManagerCMD install jpgc-mergeresults

Make sure jmeter can be run by regular user:

sudo chown -R kitconcept:root jmeter

Source: https://sarkershantonu.github.io/2021/01/06/install-jmeter-plugins-cli/

--

I just managed to get the performance tests running on our server again. Still, it would be better to have this on Plone infra. I am not aware that you could run performance tests on GHA.

Another option would be that we just connect one of the kitconcept nodes to the Plone infrastructure. Though that wouldn't be a long term solution I guess.

[gforcada]

Oh wow, that's quite some server setup that needs to be done.

We can do that on one server and add a label to it so we only have to configure one server 🤔

[fredvd]

Ah interesting, that’s what I had planned to test with our Jenkins set up but life and other community priorities :-( We are now using central job files from the jenkins 1 config bit not uet Jenkinsfiles in the repos with the v2 pipelines.

@gforcada I didn’t find time or forgot to ask you more about the Ansible setup for the server and the workers in Innsbruck. Or did I ? 😳 We could add the setup for the performance tests for restapi using Ansible on the workers. But are the ansible files up to date and in use? I want to practice a bit more with it and also have our setup reproducible.

[gforcada]

We did sort of talk a little bit about it, but not in depth by any means 😓

The situation is far from ideal, they are mostly used but not completely, and anyway trying to manage jenkins itself via ansible is a not really good idea, managing the server packages and other random configuration files, or services (like mr.roboto) is still a good idea though 😄

Let's see if I manage to attend Beethoven sprint 🤞🏾 🍀

[tisto]

@gforcada @fredvd if setting up Jenkins is too much work, we could just go with running Locust tests for now. The downside is that we Locust/Jenkins does not allow to render a graph over time that allows us to see performance regressions over time. Anyways, if @gforcada would be able to attend Beethoven Sprint that would be great and we could discuss the matter in detail facet to face. :)

[gforcada]

@tisto I'm looking forward to come, yes, if possible with my two colleagues 🎉 @sneridagh pointed me to a colleague of yours that still has not replied to me 😕

I would very much like to get a final yes/no decision if we still fit within the sprint head count 😄 mostly for planning reasons ✈️ 🚄

[gforcada]

Slightly corrected instructions:

wget https://dlcdn.apache.org//jmeter/binaries/apache-jmeter-5.5.tgz
tar xfvz apache-jmeter-5.5.tgz
mv apache-jmeter-5.5 jmeter

cd /opt/jmeter/lib
curl -O https://repo1.maven.org/maven2/kg/apc/cmdrunner/2.2.1/cmdrunner-2.2.1.jar

cd /opt/jmeter/lib/ext
curl -O https://repo1.maven.org/maven2/kg/apc/jmeter-plugins-manager/1.6/jmeter-plugins-manager-1.6.jar

cd /opt/jmeter/lib
java -jar cmdrunner-2.2.1.jar --tool org.jmeterplugins.repository.PluginManagerCMD install bzm-parallel
java -jar cmdrunner-2.2.1.jar --tool org.jmeterplugins.repository.PluginManagerCMD install jpgc-filterresults
java -jar cmdrunner-2.2.1.jar --tool org.jmeterplugins.repository.PluginManagerCMD install jpgc-mergeresults

cd /opt
chown -R jenkins:root jmeter

[gforcada]

It's running: https://jenkins.plone.org/job/restapi-performance/

It's taking quite a long time to run though 🤔