Testing on the customer's environment

[dkrut]

Hi!

I'm going to use JmeterDSL in a customer environment. Can you advise what is the best way to do this, please?
It seems that transferring the java project isn't the best idea) What solution are you using?

[rabelenda]

Hello!

Sounds like a really interesting question, but I think more context is required for giving a correct answer.

In general it depends on what you want to do and what are the limitations in the environment.

What you mean by "transferring de java project"?

Maybe giving an example as what is a basic and rough scenario/steps on how you see you could solve the problem (even though you already know is not a good one) would help us identify how to help you and improve it. Having your comments why such solution would be not effective, could also help us identify what is the main problem you are trying to solve.

Hope to hear more on this soon.

Regards

[rabelenda]

Ok, now I understand better what you are trying to achieve. You want to provide the client an artifact to run the tests.

Very interesting topic. In general the DSL is meant to be run from CI environment, or dev environment, or using a Distributed or remote engine (like BZ & Octoperf).

I think both solutions that your are thinking are correct options for the scenario you are trying to solve.

Alternatives

Uber Jar

The first alternative you lists sounds to be the safest one, only needs some tuning in maven. You can check here where I uploaded an example of it. It basically includes:

• junit-platform-launcher dependency to be able to run junit tests from jar
• main method in PerformanceTest class to be able to run junit tests from jar
• assembly plugin configuration to build the uber jar with dependencies.

Note: I tried using maven shade but it does not include test dependencies (requires changing them to runtime scope) when using shadeTestJar.

You can build the jar with mvn clean package -DskipTests.

This solution requires uploading a big jar (in my case it generated a 73MB jar) every time you change something in the testplan, but I think is the safest one since already includes all required dependencies (no potential problems on upgrade of dependencies, or missing some plugin jar or resource).

Another option is using maven jar plugin to generate test artifact, and copy all dependencies, and every time the test plan changes only upload test jar. We might distribute a pre compiled jar with the DSL for these scenarios if more users are in similar situation, but it sounds like this is a particular requirement.

saveToJmx

When using the second alternative you listed you would need to be more careful, since you also need to install jmeter on customer environment, with required test resources and jars/plugins. Additionally, if you use any lambdas in test plan (jsr223 sampler, pre and post processors or http samplers) the jmx will need the associated code.

Jbang

A third option could be using jbang adding proper execution logic and dependencies to java file, install jbang on customer environment and copy just the java class (and any resources required) to customer environment. This would require kind of duplicating dependencies if you already have them in maven. Here is an example of a performance test with jbang dependencies and execution logic:

///usr/bin/env jbang "$0" "$@" ; exit $?
/*
These commented lines make the class executable if you have jbang installed by making file
executable (eg: chmod +x ./PerformanceTest.java) and just executing it with ./PerformanceTest.java
*/
//DEPS org.assertj:assertj-core:3.22.0
//DEPS org.junit.jupiter:junit-jupiter-engine:5.8.2
//DEPS org.junit.platform:junit-platform-launcher:1.8.2
//DEPS us.abstracta.jmeter:jmeter-java-dsl:${project.version}

import static org.assertj.core.api.Assertions.assertThat;
import static us.abstracta.jmeter.javadsl.JmeterDsl.*;

import java.io.IOException;
import java.io.PrintWriter;
import org.apache.http.entity.ContentType;
import org.junit.jupiter.api.Test;
import org.junit.platform.engine.discovery.DiscoverySelectors;
import org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder;
import org.junit.platform.launcher.core.LauncherFactory;
import org.junit.platform.launcher.listeners.SummaryGeneratingListener;
import org.junit.platform.launcher.listeners.TestExecutionSummary;
import us.abstracta.jmeter.javadsl.core.TestPlanStats;

public class PerformanceTest {

  @Test
  public void test() throws IOException {
    TestPlanStats stats = testPlan(
        threadGroup(1, 3,
          httpSampler("http://localhost")
            .post("{\"var\":\"val\"}", ContentType.APPLICATION_JSON),
          httpSampler("http://localhost")
        )
      )
      .run();
    assertThat(stats.overall().errorsCount()).isEqualTo(0);
  }

  /*
   This method is only included to make test class self executable. You can remove it when
   executing tests with maven, gradle or some other tool.
   */
  public static void main(String[] args) {
    SummaryGeneratingListener summaryListener = new SummaryGeneratingListener();
    LauncherFactory.create()
        .execute(LauncherDiscoveryRequestBuilder.request()
                .selectors(DiscoverySelectors.selectClass(PerformanceTest.class))
                .build(),
            summaryListener);
    TestExecutionSummary summary = summaryListener.getSummary();
    summary.printFailuresTo(new PrintWriter(System.out));
    System.exit(summary.getTotalFailureCount() > 0 ? 1 : 0);
  }

}

clone + maven

A final option is installing maven in customer environment and cloning/copying the test project to run it with maven.

Pending questions

• Can you integrate the execution on a CI/CD pipeline for the client and run it there? With this you can just provide the repository to the client, with a pipeline configuration for the CI tool, and just run the tests with maven in the CI.

[dkrut]

Thanks to the answer!
To provide the repository to the client is not easy that seems.
I have another suggestion to solve this issue. To pack the project into a docker image with java and maven preinstalled. Then the customer needs only installed docker to run tests. What do you think?

[rabelenda]

It sounds interesting, would you be willing to submit a PR with that?

[dkrut]

I think it doesn't apply to JmeterDSL... It is a question to run tests. I'm trying to use this docker image. Seems it's very convenient

[Rutkovski]

Ok, now I understand better what you are trying to achieve. You want to provide the client an artifact to run the tests.

Very interesting topic. In general the DSL is meant to be run from CI environment, or dev environment, or using a Distributed or remote engine (like BZ & Octoperf).

I think both solutions that your are thinking are correct options for the scenario you are trying to solve.

Alternatives

Uber Jar

The first alternative you lists sounds to be the safest one, only needs some tuning in maven. You can check here where I uploaded an example of it. It basically includes:

• junit-platform-launcher dependency to be able to run junit tests from jar
• main method in PerformanceTest class to be able to run junit tests from jar
• assembly plugin configuration to build the uber jar with dependencies.

Note: I tried using maven shade but it does not include test dependencies (requires changing them to runtime scope) when using shadeTestJar.

You can build the jar with mvn clean package -DskipTests.

This solution requires uploading a big jar (in my case it generated a 73MB jar) every time you change something in the testplan, but I think is the safest one since already includes all required dependencies (no potential problems on upgrade of dependencies, or missing some plugin jar or resource).

Another option is using maven jar plugin to generate test artifact, and copy all dependencies, and every time the test plan changes only upload test jar. We might distribute a pre compiled jar with the DSL for these scenarios if more users are in similar situation, but it sounds like this is a particular requirement.

saveToJmx

When using the second alternative you listed you would need to be more careful, since you also need to install jmeter on customer environment, with required test resources and jars/plugins. Additionally, if you use any lambdas in test plan (jsr223 sampler, pre and post processors or http samplers) the jmx will need the associated code.

Jbang

A third option could be using jbang adding proper execution logic and dependencies to java file, install jbang on customer environment and copy just the java class (and any resources required) to customer environment. This would require kind of duplicating dependencies if you already have them in maven. Here is an example of a performance test with jbang dependencies and execution logic:

///usr/bin/env jbang "$0" "$@" ; exit $?
/*
These commented lines make the class executable if you have jbang installed by making file
executable (eg: chmod +x ./PerformanceTest.java) and just executing it with ./PerformanceTest.java
*/
//DEPS org.assertj:assertj-core:3.22.0
//DEPS org.junit.jupiter:junit-jupiter-engine:5.8.2
//DEPS org.junit.platform:junit-platform-launcher:1.8.2
//DEPS us.abstracta.jmeter:jmeter-java-dsl:${project.version}

import static org.assertj.core.api.Assertions.assertThat;
import static us.abstracta.jmeter.javadsl.JmeterDsl.*;

import java.io.IOException;
import java.io.PrintWriter;
import org.apache.http.entity.ContentType;
import org.junit.jupiter.api.Test;
import org.junit.platform.engine.discovery.DiscoverySelectors;
import org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder;
import org.junit.platform.launcher.core.LauncherFactory;
import org.junit.platform.launcher.listeners.SummaryGeneratingListener;
import org.junit.platform.launcher.listeners.TestExecutionSummary;
import us.abstracta.jmeter.javadsl.core.TestPlanStats;

public class PerformanceTest {

  @Test
  public void test() throws IOException {
    TestPlanStats stats = testPlan(
        threadGroup(1, 3,
          httpSampler("http://localhost")
            .post("{\"var\":\"val\"}", ContentType.APPLICATION_JSON),
          httpSampler("http://localhost")
        )
      )
      .run();
    assertThat(stats.overall().errorsCount()).isEqualTo(0);
  }

  /*
   This method is only included to make test class self executable. You can remove it when
   executing tests with maven, gradle or some other tool.
   */
  public static void main(String[] args) {
    SummaryGeneratingListener summaryListener = new SummaryGeneratingListener();
    LauncherFactory.create()
        .execute(LauncherDiscoveryRequestBuilder.request()
                .selectors(DiscoverySelectors.selectClass(PerformanceTest.class))
                .build(),
            summaryListener);
    TestExecutionSummary summary = summaryListener.getSummary();
    summary.printFailuresTo(new PrintWriter(System.out));
    System.exit(summary.getTotalFailureCount() > 0 ? 1 : 0);
  }

}

clone + maven

A final option is installing maven in customer environment and cloning/copying the test project to run it with maven.

Pending questions

• Can you integrate the execution on a CI/CD pipeline for the client and run it there? With this you can just provide the repository to the client, with a pipeline configuration for the CI tool, and just run the tests with maven in the CI.

At first, thank you for this answer! It really helped me
Could you help one more time, please, with using resources in case if we use uber jar?
I mean how can I use method csvDataSet () from DSL?
If I make a jar, and put String with the path to file or TestResource into csvDataSet () in, doesn't find file into a jar.