Code

Add incremental benchmarking scripts#44

Code

Open

jerhard wants to merge 87 commits intomastergoblint/bench:masterfrom incremental_benchmarking_scriptsgoblint/bench:incremental_benchmarking_scriptsCopy head branch name to clipboard

+2,087

Commits

Commits on Nov 23, 2022

• Move scripts for incremental benchmarking from the analyzer to the bench repo.

  Show description for ccc2a6a
  

  jerhard

  and

  stilscher

  committed
  ccc2a6a

• Change incremental benchmarking scripts for running with sqlite

  jerhard

  committed
  73cebe1

• Update path for conf

  jerhard

  committed
  c9e5566

• Add build_compdb_sqlite.sh

  jerhard

  committed
  e138867

• Update configuration for sqlite

  jerhard

  committed
  7bd5d92

• efficiency.py: Make it configurable which project to benchmark incremental goblint on.

  jerhard

  committed
  dffaeac

• Make precision take paramters for which project to run on

  jerhard

  committed
  764372e

• Do not use dataclasses for compatibility with python 3.6

  jerhard

  committed
  c773156

• Improve error message

  jerhard

  committed
  7009d93

• Change annotation of type for compatibility with Python 3.6

  jerhard

  committed
  11e92bf

• Add configuration for incremental postsolver to sqlite benchmarking config.

  jerhard

  committed
  2b7eafd

• Utils.analyze_commit: Fix handling of case that file list is passed.

  jerhard

  committed
  1500752

• Print analyze command and commit hash in config.out created by incremental benchmarking script.

  jerhard

  committed
  79c940f

• Add -v option to analyze commands in efficiency script.

  jerhard

  committed
  a248b6b

• Merge branch 'master' into incremental_benchmarking_scripts

  jerhard

  committed
  075194b

Commits on Nov 28, 2022

• Add figlet as project.

  Show description for 6d1a67d
  jerhard

  committed
  6d1a67d

Commits on Nov 29, 2022

• Adapt regular expression to extract runtime from output.

  jerhard

  committed
  595de4c

• incremental benchmarking: Use Makefile instead of compiledb for figlet.

  jerhard

  committed
  8e5888f

• Add biuld_compdb_figlet.sh

  jerhard

  committed
  8bcc304

• Add preliminary setup for chrony incremental benchmarks.

  Show description for 97ea54e
  jerhard

  committed
  97ea54e

• Reset time range of considered commits for zstd.

  Show description for 587a2dd
  jerhard

  committed
  587a2dd

• Update build script for chrony.

  jerhard

  committed
  0345ce7

• Fix project configuration for chrony to not provide files (but use compiledb).

  jerhard

  committed
  c943f9f

• Merge branch 'incremental_benchmarking_scripts_chrony' into incremental_benchmarking_scripts

  jerhard

  committed
  5fc065e

Commits on Nov 30, 2022

• Use github mirror for chrony..

  Show description for 2639efb
  jerhard

  committed
  2639efb

Commits on Dec 9, 2022

• Plot.py: Create cummulative graphs again, combining results from 3 configs in one graph.

  Show description for 1eb1a82
  jerhard

  committed
  1eb1a82

• Plot.py: Iterate over folders of form result_efficiency_[project], for given projects.

  jerhard

  committed
  3d7620a

• Change cummulative_distr_compare2 to again only create comparisons between 2 configs.

  jerhard

  committed
  33b687f

• Logarithmic plots: choose y-min depending on minimum of plotted data (rather than a constant).

  jerhard

  committed
  1d1fc2a

• Plot 3-way comparision again logarithmically.

  jerhard

  committed
  00944ab

Commits on Dec 12, 2022

• Avoid artefact in cummulative graph.

  jerhard

  committed
  5273984

• Plot incremental analysis with incremental postsolver separately.

  jerhard

  committed
  5336085

• Rename function cummulative_distr_all3 -> cummulative_distr_all4

  jerhard

  committed
  36bf6b3

• Change output for .pdf files to .pgf.

  jerhard

  committed
  800db88

• Change figsize for cummulative_distr functions.

  jerhard

  committed
  3beedd8

Commits on Dec 14, 2022

• Revert "Change figsize for cummulative_distr functions."

  Show description for 3c67b91
  jerhard

  committed
  3c67b91

• Revert "Change output for .pdf files to .pgf."

  Show description for 162e743
  jerhard

  committed
  162e743

Commits on Dec 15, 2022

• Plot: Create precision plots again.

  jerhard

  committed
  57025e3

• Remove printstats from argumeents for compare_runs.

  jerhard

  committed
  fdb7ffb

• Define analyzed branch per repository; needed for precision.py.

  jerhard

  committed
  e946610

• Extract analysis and solving time in efficiency script.

  jerhard

  committed
  9d03382

Commits on Dec 16, 2022

• Fix efficiency plot.

  jerhard

  committed
  203d9dc

• add boxplot generation as alternative for precision graph

  stilscher

  committed
  9f9a4e2

• fix naming

  stilscher

  committed
  67ff095

• fix ylim

  stilscher

  committed
  f358d13

Commits on Dec 17, 2022

• Extract walltime instead of CPU time.

  Show description for 1c6f018
  jerhard

  committed
  1c6f018

• Peform non-incremental run on child commit as well.

  jerhard

  committed
  621e84d

Commits on Dec 18, 2022

• Add script to run all efficiency scripts.

  jerhard

  committed
  7eba011

Commits on Dec 19, 2022

• Change line style for efficiency plots, text for configuration description.

  jerhard

  committed
  d19d7cb

Commits on Dec 20, 2022

• Add run_precision script.

  jerhard

  committed
  fa658ff

Commits on Dec 21, 2022

• Use chrony configuration for chrony.

  jerhard

  committed
  24d7a9b

Commits on Dec 23, 2022

• reset current time to static value for chrony

  stilscher

  committed
  bfc219a

Commits on Dec 25, 2022

• Precision: Run figlet and zstd.

  jerhard

  committed
  294cf9a

Commits on Dec 28, 2022

• Change ymax for precision graphs: 0.3

  jerhard

  committed
  6ad3b20

Commits on Dec 30, 2022

• Change text for different configurations to enumeration.

  jerhard

  committed
  696506f

• Merge branch 'master' into incremental_benchmarking_scripts

  jerhard

  committed
  d4a2d8e

• Update size of fonts for precision graph.

  jerhard

  committed
  e671d9c

Commits on Dec 31, 2022

• Precision plot: Add ticks at every commit.

  jerhard

  committed
  6333415

Commits on Jun 15, 2023

• Move confs for figlet, chrony, zstd into bench repo, adapt scripts for it.

  jerhard

  committed
  b3838d3

• Relax constraint on numpy, to make constraints satisfiable, add compiledb as requirement.

  jerhard

  committed
  e3db195

• Refactor collect_data to reduce redundancy.

  jerhard

  committed
  a6d27d9

• Refactor collect_data further to reduce redundancy.

  jerhard

  committed
  670cf3e

Commits on Jun 16, 2023

• Collec changed/added/removed functions from child (incremental) config.

  jerhard

  committed
  376a176

• Adapt match for changed/added/removed functions for changed goblint output.

  jerhard

  committed
  9b67c07

• Extract CPU times for total runtimes, analysis and solving times.

  jerhard

  committed
  4f24c89

• Collect_data: Check whether result directory exists before iterating over it.

  Show description for e4dc7fe
  jerhard

  committed
  e4dc7fe

Commits on Jun 28, 2023

• additionally plot efficiency bar plots

  stilscher

  committed
  7e6e8a7

Commits on Jul 11, 2023

• Interactive confs: set incremental.detect-renames to false.

  jerhard

  committed
  ddaabd7

• Extract incremental analysis of child commit in precision script.

  jerhard

  committed
  e169eff

Commits on Jul 12, 2023

• Exclude figlet from precision run.

  jerhard

  committed
  249ebf5

• Precision script: fix config file look-up.

  jerhard

  committed
  de4208b

Commits on Jul 13, 2023

• Incremental efficiency script: Do not limit commits to 50 relevant LoC changes.

  jerhard

  committed
  bbb3e7d

Commits on Jul 20, 2023

• Precision: run all configurations.

  jerhard

  committed
  793496c

• Fix parameters for increm rel with incr. postsolver.

  jerhard

  committed
  2e50ca1

• Incremental precision script: perform merge results for all configs.

  jerhard

  committed
  5c9d7e1

• Interactive precision: iterate over configurations in outer instead of inner loop.

  jerhard

  committed
  c7d9e15

• Use float instead of np.float as np.float has been removed.

  jerhard

  committed
  54e6359

• Plot separate efficiency graphs for <= 50 loc changes and > 50 loc changes.

  jerhard

  committed
  92af047

• Efficiency plots: add efficiency_bar_plots again.

  jerhard

  committed
  c020267

• Interactive efficiency: plot cpu_time instead of wall time.

  jerhard

  committed
  ca79b69

Commits on Jul 23, 2023

• Add printouts for debugging.

  jerhard

  committed
  086955b

• Remove spurious --enable from parameters passed in precision script.

  jerhard

  committed
  02d9efc

• Output precision results into different files, extend plot script to produce plots for each of them.

  jerhard

  committed
  8657739

• Add underscore in file names of results produced by precision script and the precision plots.

  jerhard

  committed
  26fc662

Commits on Jul 25, 2023

• Precision script: explicitely pass load-dir/save-dir for incremental data.

  jerhard

  committed
  20571d3

Commits on Jul 28, 2023

• Remove detect-renames field for compatibility with sttt-2022 goblint.

  jerhard

  committed
  41d5558

Commits on Jul 30, 2023

• Adapt script to extract changed functions from goblint versino sttt-2022.

  jerhard

  committed
  6158b17