⌛ kruonis ⌛

A tool to perform benchmarks on TypeScript.

Kruonis is an embodiment of the Lithuanian goddess of time, as essentially the tool measures the time it takes for code to run.

tl;dr

A Benchmark is a set of Tests.

When running a Benchmark, each Test is run several times (test cycle).

Kruonis summarizes the statistiscs of each cycle's performance among all cycles.

Note: we use performance-now to measure performance.

Usage example

First, import kruonis' main classes:

import { Benchmark, Test } from "kruonis";

Then, let's create a benchmark:

let benchmark = new Benchmark();

Additionally, kruonis lets you pass your benchmark properties as an object to the constructor, such as:

benchmark = new Benchmark({ 'maxCycles': 50, 'name': 'Benchmark', 'maxTime': 15 });

The possible properties are available here.

We can also define events for the benchmark class, such as:

benchmark
    .on('onBegin', (benchmark: Benchmark) => {
        // Code to run on the beginning of this benchmark
        // Example:
        console.log("Beginning of the benchmark")
    })
    .on('onTestBegin', (benchmark: Benchmark, test: Test) => {
        // Code to run on the end of the benchmark (on the end of all tests)
        // Example:
        console.log("Running test: " + test.name)
    })
    .on('onTestEnd', (benchmark: Benchmark) => {
        // Code to run on the end of the benchmark (on the end of all tests)
        // Example:
        console.log("The stats of the test that just ran are: " + test.getStats())
    })
    .on('onEnd', (benchmark: Benchmark) => {
        // Code to run on the end of the benchmark (on the end of all tests)
        // Example:
        console.log("Ended benchmark")
    });

A benchmark consists of a set of tests. Therefore, we can add tests to a benchmark. Each Test can also take events. For example:

// Example object for test
let testArray: number[];

benchmark
    .add(
        new Test('exampleTest1', () => {
            // Measure code performance of what goes here
            // Example:
            for (let i = 0; i < testArray.length; ++i)
                testArray[i] *= testArray[i];
        })
        .on('onBegin', (test: Test) => {
            // Code to execute on before starting the cycle loop
            // Example:
            testArray = [1, 2, 3, 4, 5, 6];
        })
        .on('onCycleBegin', (test: Test) => {
            // Code to execute before each cycle
            // Example:
            console.log("Starting cycle");
        })
        .on('onCycleEnd', (test: Test) => {
            // Code to execute after each cycle ran
            // Example:
            testArray = [1, 2, 3, 4, 5, 6];
        })
        .on('onEnd', (test: Test) => {
            // Test to execute after all cycles
            // The Stats object with the cycle performances is now populated
            // Example:
            console.log("Finished running all cycles");
        })
    )
    .add(
        // Add another test ...
    );

After adding all tests, we can then run them using:

benchmark.run();

After running the benchmark we can obtain the statistics of each ran test in several ways:

1. Using the return array of the behcnmark.run() method. Example:

const results: Array<[string, Stats]> = benchmark.run();
for(let result of results) {
    console.log("Test name: " + result[0]);
    console.log("Test stats: " + result[1]);
}

2. Adding an event listener of the onTestEnd event to the benchmark.

benchmark.on('onTestEnd', (benchmark: Benchmark, test: Test) => {
    console.log("Test name: " + test.name);
    console.log("Test stats: " + test.getStats());
}). run();

3. Adding an event listener of the onEnd event to the benchmark.

benchmark.on('onEnd', (benchmark: Benchmark) => {
    for(let result of benchmark.getResults()) {
        console.log("Test name: " + result[0]);
        console.log("Test stats: " + result[1]);
    }
}). run();

4. Adding an identical event listener to all tests' onEnd events.

benchmark
    .add(
        new Test('exampleTest1', () => {
            // Code
        })
        .on('onEnd', (test: Test) => {
            console.log("Test name: " + test.name);
            console.log("Test stats: " + test.getStats());
        })
    .add(
        // Similar for other tests
    ).run();
}

The Statistics outputted can be consulted here. An example of a Stats object is:

{
    // The mean run time of the test
    'mean': 11.4,
    // The standard deviation
    'std': 3.352610922848042,
    // The number of ran cycles
    'count': 10,
    // The maximum time it took to run the test code
    'max': 18,
    // The maximum time it took to run the test code
    'min': 6
}

How does it work?

The logic behind the benchmark.run() method (and the order in which the events are run) is:

# Benchmark scope
Benchmark.onBegin()

for each test of tests:
    Benchmark.onTestBegin()

    # Test scope
    test.onBegin()

    while(number of cycles < minCycles and
          spent time on ran cycles < maxTime and
          number of ran cycles < maxCycles)

        # Cycle scope
        test.onCycleBegin()

        runTestCode()

        test.onCycleEnd()

    test.onEnd()
    # Ended test

    Benchmark.onTestEnd()

Benchmark.onEnd()

Acknowledgments

This work is financed by the ERDF – European Regional Development Fund through the Operational Programme for Competitiveness and Internationalisation - COMPETE 2020 Programme and by National Funds through the Portuguese funding agency, FCT – Fundação para a Ciência e a Tecnologia within project PTDC/CCI-INF/32636/2017 (POCI-01-0145-FEDER-032636).