HowTo: Investigate a failure

This page describes the current workflow, at time of writing, for investigating a bad result from an experiment.

Note	We ideally want to automate as much of this as possible eventually.

Finding the failure

Failures get saved to $OUTDIR/saved/$MACHINE/$STATUS/$YEAR/$MONTH/$DAY/$HOUR_$MINUTE_$SECOND, where $OUTDIR is the value of paths.out_dir in the tester configuration. This directory format means that the most recent failure of a particular kind can be found by taking the ASCIIbetically highest choice for each directory under $STATUS.

There isn’t yet an easy way to tell from the dashboard alone which failures correspond to which directories; if you need more granularity than just 'the latest failure of a particular type on a particular machine', consult $OUTDIR/results.log.

Checking the plan

Inside the failure directory, plan.json.gz contains a (compressed) plan file that describes the failed testcase, which compilers failed, and how. We can use act-tester-analyse to get human-readable summaries of this information.

To get a summary of every compilation in the plan that failed:

act-tester-analyse -S plan.json.gz

Other flags can add more information: for example, -C adds compiler summaries, while -P adds plan stage timing.

Replaying the invoke stage

To check for reproducibility, you can replay the invoke stage of the plan with act-tester-invoke -f plan.json.gz; -f tells the tester that it should force a repeat of the stage.

Bisecting the test case

Note	This requires both act and act-bash.

Note	Eventually, there should be an automated workflow for bisecting using plan information.

When using act-tester with the act fuzzer, the resulting testcases are often too unwieldy to investigate. To trim down the testcase, one can bisect it: run the tester in a stripped-down mode on prefixes of the fuzzer trace until the minimal prefix that exhibits the faulty behaviour occurs. To do this, use the bisect script from act-bash:

tar xf $NAME-OF-FAILING-TESTCASE.tar.gz
cd $NAME-OF-FAILING-TESTCASE
$PATH-TO-ACT-BASH/bisect -m $MACHINE -c $COMPILER [-O $OPT-LEVEL] [-M $MACHINE-OPT] fuzz.trace orig.litmus

This currently outputs the minified trace as ./trace.mini and a reconstituted litmus file as ./trace.mini.litmus. This is subject to change.