Handle a potential corner case

[kaiyuanw]

Context

Update the code to handle the following case:

Imagine a test class import a src class but does not cover any lines
in the src class. Assume that the test class only has a test method
testExecNothing() and this method is executed before another test
method testExecSomething() in another test class. Assume the test
method testExecSomething() covers some lines in the src class. In the
original implementation, testExecNothing() shares the same hitArray as
testExecSomething() and this may lead to incorrect computation of the
suspiciousness scores. For example, if testExecSomething() fails,
then all lines covered by testExecSomething() will be treated as they
are executed twice as many times as they actually are because
testExecNothing()'s hitArray is updated by testExecSomething().

Check lists

• Unit tests
• Test pass
• Coding style (indentation, etc)

Additional comments

Please add any information of interest here below.

[jose]

Hi @kaiyuanw,

First of all, thanks for your pull request.

Is this corner case still an issue if you use the latest version of GZoltar? Version 1.7.2 introduces a custom test runner which executes each unit test method in isolation to collect precise code coverage of static fields and it also seems to address the corner case you described.

Please find below the source code for the corner case you described and some bash commands to verify whether GZoltar reports accurate coverage. As far I can see, the corner case is no longer an issue.

Foo.java

public class Foo {
  public boolean bar(int x) {
    if (x == 37) { // BUG, it should be 'x == 42'
      return true;
    }
    return false;
  }
}

TestA.java

import org.junit.Test;

// "Imagine a test class import a src class but does not cover any lines
// in the src class. Assume that the test class only has a test method
// testExecNothing() and this method is executed before another test
// method testExecSomething() in another test class."
public class TestA {

  private Foo foo;

  @Test
  public void testExecNothing() {
    // NO-OP, 0 coverage
  }
}

TestB.java

import org.junit.Test;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertTrue;

// "Assume the test method testExecSomething() covers some lines in
// the src class."
public class TestB {
  @Test
  public void testExecSomething() {
    Foo foo = new Foo();
    assertFalse(foo.bar(0));
    assertTrue(foo.bar(42));
  }
}

Perform fault-localization

# Setup
$ mkdir -p /tmp/gz_pr_7
$ cd /tmp/gz_pr_7
$ wget https://github.com/GZoltar/gzoltar/releases/download/v1.7.2/gzoltar-1.7.2.201905090602.zip
$ unzip gzoltar-1.7.2.201905090602.zip
$ export GZOLTAR_CLI_JAR=/tmp/gz_pr_7/lib/gzoltarcli.jar
$ export GZOLTAR_AGENT_JAR=/tmp/gz_pr_7/lib/gzoltaragent.jar
# TODO: create the following files Foo.java, TestA.java, and TestB.java
# and copy the Java code above to each file

# Compile class under test and two test classes
$ javac -cp .:/home/$USER/.m2/repository/junit/junit/4.11/junit-4.11.jar \
            Foo.java TestA.java TestB.java

# Prepare list of unit test methods to run
$ echo "JUNIT,TestA#testExecNothing" > unit_tests_file.txt
$ echo "JUNIT,TestB#testExecSomething" >> unit_tests_file.txt

# Run each unit test method in isolation
$ java -javaagent:$GZOLTAR_AGENT_JAR=includes="Foo",excludes="TestA:TestB" \
        -cp .:/home/$USER/.m2/repository/junit/junit/4.11/junit-4.11.jar:$GZOLTAR_CLI_JAR \
        com.gzoltar.cli.Main runTestMethods \
          --testMethods unit_tests_file.txt \
          --collectCoverage

# Create fault-localization report
$ java -cp .:$GZOLTAR_CLI_JAR \
      com.gzoltar.cli.Main faultLocalizationReport \
        --buildLocation . \
        --dataFile gzoltar.ser \
        --outputDirectory .

# List executed test cases (excluding the ones without any coverage)
$ cat sfl/txt/tests.csv
name,outcome,runtime,stacktrace
TestB#testExecSomething,FAIL,11597694,java.lang.AssertionError at ...

# List instrumented lines of code
$ cat sfl/txt/spectra.csv
name
$Foo#Foo():1
$Foo#bar(int):3
$Foo#bar(int):4
$Foo#bar(int):6

# List coverage matrix
$ cat sfl/txt/matrix.txt
1 1 0 1 -

TestB#testExecSomething took 11597694 nanoseconds to run, it failed, and it covered lines 1, 3 and 6 of class Foo. As TestA#testExecNothing does not exercise any line of code of class Foo, it is not reported by GZoltar.

--
Best,
Jose