SSBSE-Challenge-2020-application

This is the replication package of our study for SSBSE 2020 challenge

Table of Content:

• Model Generation
• Test Generation
• Mutation Score
  • Run PIT
  • Collect mutation scores
  • Collect killed mutants
• Data Analysis
  • Coverage analysis
  • killed mutants analysis

Model Generation

for model generation go to model-generation directory:

cd model-generation

Then run the following script:

bash docker-run.sh

This bash file makes a docker container and runs the model generation. The output of this execution will be saved in model/gson. The execution log of the model generation will save in model-generation/logs.

Test Generation

For running the test generation, first, go to the test-generation directory.

cd test-generation

The models generated by model generation should be located in analysis-result/gson. We have already put the models in this directory.

For test generation, run docker-run.

for default evosuite:

bash docker-run.sh <number-of-runs> <number-of-parallel-processes> 0 0

for evosuite + model seeding:

bash docker-run.sh -m -r  <number-of-runs> <number-of-parallel-processes> <p-Object-pool> <seed-clone> 

To replicate our study, run this script with the following parameters.

for default evosuite:

bash docker-run.sh 10 12 0 0

for evosuite + model seeding:

bash docker-run.sh -m -r 10 12 0.3 0.5

The generated test cases will be stored in the generated_tests directory. The CSV file containing the fitness function values is saved in the results directory. Also, the execution logs of EvoSuite instances will be stored in the logs directory.

! The script does not run EvoSuite if the CSV file in results contains a row describing the results of a particular EvoSuite run. Since we already put the results achieved by EvoSuite and EvoSuite + model seeding in these CSV files, bu running the scripts mentioned above, it does not run anything. First, you need to remove the rows in results CSV files to replicate the test generation.

Mutation Score

Run PIT

Run docker-mutation-score.sh bash file to apply PIT on generated test suites.

for applying PIT on generated tests by default evosuite:

bash docker-mutation-score.sh <number-of-runs> <number-of-parallel-processes> 0 0

for applying PIT on generated tests by evosuite + model seeding:

bash docker-mutation-score.sh -m -r <number-of-runs> <number-of-parallel-processes> <p-Object-pool> <seed-clone>

To replicate our study, run this script with the following parameters.

default evosuite:

bash docker-mutation-score.sh 10 12 0.3 0.5

evosuite + model seeding:

bash docker-mutation-score.sh -m -r 10 12 0.3 0.5

After pit execution, the PIT reports will be saved in pitest/out.

! The script does not run PIT if we already have a report for a test case. So, for replicating the PIT execution phase, you need to remove the existing PIT reports from the pitest/out directory.

Collect mutation scores

To make a CSV file of mutation scores, run the following script in the container:

for default evosuite:

bash pitest/scripts/bash collect_mutation_scores.sh 10 classes.csv 0.3 0.5

for evosuite + model seeding:

bash pitest/scripts/bash collect_mutation_scores.sh -m -r 10 classes.csv 0.3 0.5

The CSV will be stored in pitest/csv.

Collect killed mutants

To perform the mutant analysis, you need to make large CSV files containing all of the mutants in the JsonReader class. For make these CSVs, run the following scripts:

for default EvoSuite:

bash pitest/scripts/bash/collect_all_mutants.sh 10 classes.csv 0.3 0.5 TRUE

for evosuite + model seeding:

bash pitest/scripts/bash/collect_all_mutants.sh -m 10 classes.csv 0.3 0.5 TRUE

The CSV files will be stored in pitest/killed_mutants.

Data Analysis

For data analysis, go to analysis directory:

cd analysis

Coverage analysis

To calculate the coverages of test suites generated by pure EvoSuite or evosuite+ model seeding, run the following R script:

RScript RScripts/analysis.r 

The summary will be printed, and also, the plots will be stored in the analysis/images directory.

killed mutants analysis

For calculating the numbers used in the ven diagram, run the following R script:

RScript RScripts/mutants.R