Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add incremental benchmarking scripts #44

Open
wants to merge 87 commits into
base: master
Choose a base branch
from
Open

Add incremental benchmarking scripts #44

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
ccc2a6a
Move scripts for incremental benchmarking from the analyzer to the be…
jerhard Oct 5, 2022
73cebe1
Change incremental benchmarking scripts for running with sqlite
jerhard Oct 6, 2022
c9e5566
Update path for conf
jerhard Oct 6, 2022
e138867
Add build_compdb_sqlite.sh
jerhard Oct 6, 2022
7bd5d92
Update configuration for sqlite
jerhard Oct 6, 2022
dffaeac
efficiency.py: Make it configurable which project to benchmark increm…
jerhard Oct 6, 2022
764372e
Make precision take paramters for which project to run on
jerhard Oct 6, 2022
c773156
Do not use dataclasses for compatibility with python 3.6
jerhard Oct 6, 2022
7009d93
Improve error message
jerhard Oct 6, 2022
11e92bf
Change annotation of type for compatibility with Python 3.6
jerhard Oct 6, 2022
2b7eafd
Add configuration for incremental postsolver to sqlite benchmarking c…
jerhard Oct 6, 2022
1500752
Utils.analyze_commit: Fix handling of case that file list is passed.
jerhard Oct 6, 2022
79c940f
Print analyze command and commit hash in config.out created by increm…
jerhard Oct 7, 2022
a248b6b
Add -v option to analyze commands in efficiency script.
jerhard Oct 7, 2022
075194b
Merge branch 'master' into incremental_benchmarking_scripts
jerhard Nov 23, 2022
6d1a67d
Add figlet as project.
jerhard Nov 28, 2022
595de4c
Adapt regular expression to extract runtime from output.
jerhard Nov 29, 2022
8e5888f
incremental benchmarking: Use Makefile instead of compiledb for figlet.
jerhard Nov 29, 2022
8bcc304
Add biuld_compdb_figlet.sh
jerhard Nov 29, 2022
97ea54e
Add preliminary setup for chrony incremental benchmarks.
jerhard Nov 29, 2022
587a2dd
Reset time range of considered commits for zstd.
jerhard Nov 29, 2022
0345ce7
Update build script for chrony.
jerhard Nov 29, 2022
c943f9f
Fix project configuration for chrony to not provide files (but use co…
jerhard Nov 29, 2022
5fc065e
Merge branch 'incremental_benchmarking_scripts_chrony' into increment…
jerhard Nov 29, 2022
2639efb
Use github mirror for chrony..
jerhard Nov 30, 2022
1eb1a82
Plot.py: Create cummulative graphs again, combining results from 3 co…
jerhard Dec 9, 2022
3d7620a
Plot.py: Iterate over folders of form result_efficiency_[project], fo…
jerhard Dec 9, 2022
33b687f
Change cummulative_distr_compare2 to again only create comparisons be…
jerhard Dec 9, 2022
1d1fc2a
Logarithmic plots: choose y-min depending on minimum of plotted data …
jerhard Dec 9, 2022
00944ab
Plot 3-way comparision again logarithmically.
jerhard Dec 9, 2022
5273984
Avoid artefact in cummulative graph.
jerhard Dec 12, 2022
5336085
Plot incremental analysis with incremental postsolver separately.
jerhard Dec 12, 2022
36bf6b3
Rename function cummulative_distr_all3 -> cummulative_distr_all4
jerhard Dec 12, 2022
800db88
Change output for .pdf files to .pgf.
jerhard Dec 12, 2022
3beedd8
Change figsize for cummulative_distr functions.
jerhard Dec 12, 2022
3c67b91
Revert "Change figsize for cummulative_distr functions."
jerhard Dec 14, 2022
162e743
Revert "Change output for .pdf files to .pgf."
jerhard Dec 14, 2022
57025e3
Plot: Create precision plots again.
jerhard Dec 15, 2022
fdb7ffb
Remove printstats from argumeents for compare_runs.
jerhard Dec 15, 2022
e946610
Define analyzed branch per repository; needed for precision.py.
jerhard Dec 15, 2022
9d03382
Extract analysis and solving time in efficiency script.
jerhard Dec 15, 2022
203d9dc
Fix efficiency plot.
jerhard Dec 16, 2022
9f9a4e2
add boxplot generation as alternative for precision graph
stilscher Dec 16, 2022
67ff095
fix naming
stilscher Dec 16, 2022
f358d13
fix ylim
stilscher Dec 16, 2022
1c6f018
Extract walltime instead of CPU time.
jerhard Dec 17, 2022
621e84d
Peform non-incremental run on child commit as well.
jerhard Dec 17, 2022
7eba011
Add script to run all efficiency scripts.
jerhard Dec 18, 2022
d19d7cb
Change line style for efficiency plots, text for configuration descri…
jerhard Dec 19, 2022
fa658ff
Add run_precision script.
jerhard Dec 20, 2022
24d7a9b
Use chrony configuration for chrony.
jerhard Dec 21, 2022
bfc219a
reset current time to static value for chrony
stilscher Dec 23, 2022
294cf9a
Precision: Run figlet and zstd.
jerhard Dec 25, 2022
6ad3b20
Change ymax for precision graphs: 0.3
jerhard Dec 28, 2022
696506f
Change text for different configurations to enumeration.
jerhard Dec 30, 2022
d4a2d8e
Merge branch 'master' into incremental_benchmarking_scripts
jerhard Dec 30, 2022
e671d9c
Update size of fonts for precision graph.
jerhard Dec 30, 2022
6333415
Precision plot: Add ticks at every commit.
jerhard Dec 31, 2022
b3838d3
Move confs for figlet, chrony, zstd into bench repo, adapt scripts fo…
jerhard Jun 15, 2023
e3db195
Relax constraint on numpy, to make constraints satisfiable, add compi…
jerhard Jun 15, 2023
a6d27d9
Refactor collect_data to reduce redundancy.
jerhard Jun 15, 2023
670cf3e
Refactor collect_data further to reduce redundancy.
jerhard Jun 15, 2023
376a176
Collec changed/added/removed functions from child (incremental) config.
jerhard Jun 16, 2023
9b67c07
Adapt match for changed/added/removed functions for changed goblint o…
jerhard Jun 16, 2023
4f24c89
Extract CPU times for total runtimes, analysis and solving times.
jerhard Jun 16, 2023
e4dc7fe
Collect_data: Check whether result directory exists before iterating …
jerhard Jun 16, 2023
7e6e8a7
additionally plot efficiency bar plots
stilscher Jun 28, 2023
ddaabd7
Interactive confs: set incremental.detect-renames to false.
jerhard Jul 11, 2023
e169eff
Extract incremental analysis of child commit in precision script.
jerhard Jul 11, 2023
249ebf5
Exclude figlet from precision run.
jerhard Jul 12, 2023
de4208b
Precision script: fix config file look-up.
jerhard Jul 12, 2023
bbb3e7d
Incremental efficiency script: Do not limit commits to 50 relevant Lo…
jerhard Jul 13, 2023
793496c
Precision: run all configurations.
jerhard Jul 20, 2023
2e50ca1
Fix parameters for increm rel with incr. postsolver.
jerhard Jul 20, 2023
5c9d7e1
Incremental precision script: perform merge results for all configs.
jerhard Jul 20, 2023
c7d9e15
Interactive precision: iterate over configurations in outer instead o…
jerhard Jul 20, 2023
54e6359
Use float instead of np.float as np.float has been removed.
jerhard Jul 20, 2023
92af047
Plot separate efficiency graphs for <= 50 loc changes and > 50 loc ch…
jerhard Jul 20, 2023
c020267
Efficiency plots: add efficiency_bar_plots again.
jerhard Jul 20, 2023
ca79b69
Interactive efficiency: plot cpu_time instead of wall time.
jerhard Jul 20, 2023
086955b
Add printouts for debugging.
jerhard Jul 23, 2023
02d9efc
Remove spurious --enable from parameters passed in precision script.
jerhard Jul 23, 2023
8657739
Output precision results into different files, extend plot script to …
jerhard Jul 23, 2023
26fc662
Add underscore in file names of results produced by precision script …
jerhard Jul 23, 2023
20571d3
Precision script: explicitely pass load-dir/save-dir for incremental …
jerhard Jul 25, 2023
41d5558
Remove detect-renames field for compatibility with sttt-2022 goblint.
jerhard Jul 28, 2023
6158b17
Adapt script to extract changed functions from goblint versino sttt-2…
jerhard Jul 30, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
114 changes: 114 additions & 0 deletions scripts/incremental/benchmarking/conf/chrony-incrpostsolver.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
{
"ana": {
"activated": [
"expRelation",
"base",
"threadid",
"threadflag",
"threadreturn",
"escape",
"mutexEvents",
"mutex",
"access",
"mallocWrapper",
"mhp",
"symb_locks",
"var_eq",
"mallocFresh",
"race"
],
"ctx_insens": [
"var_eq"
],
"base": {
"privatization": "none",
"context": {
"non-ptr": false
}
},
"thread": {
"domain": "plain",
"include-node": false
},
"race": {
"free": false
},
"dead-code": {
"lines": true
},
"int": {
"interval": true,
"def_exc": true
},
"malloc": {
"wrappers": [
"Malloc",
"Realloc",
"Malloc2",
"Realloc2",
"ARR_CreateInstance",
"realloc_array",
"ARR_GetNewElement"
]
}
},
"sem": {
"unknown_function": {
"spawn": false,
"invalidate": {
"globals": false,
"args": false
}
}
},
"solvers": {
"td3": {
"restart": {
"wpoint": {
"enabled": false
}
}
}
},
"exp": {
"earlyglobs": true
},
"cil": {
"merge": {
"inlines": false
}
},
"dbg": {
"timing": {
"enabled": true
}
},
"warn": {
"assert": false,
"behavior": false,
"integer": false,
"cast": false,
"race": true,
"deadcode": true,
"analyzer": false,
"unsound": true,
"imprecise": false,
"unknown": false,
"error": false,
"warning": true,
"info": false,
"debug": false,
"success": true
},
"incremental": {
"postsolver": {
"enabled": true
},
"restart": {
"sided": {
"enabled": false
},
"write-only": true
}
}
}
114 changes: 114 additions & 0 deletions scripts/incremental/benchmarking/conf/chrony.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
{
"ana": {
"activated": [
"expRelation",
"base",
"threadid",
"threadflag",
"threadreturn",
"escape",
"mutexEvents",
"mutex",
"access",
"mallocWrapper",
"mhp",
"symb_locks",
"var_eq",
"mallocFresh",
"race"
],
"ctx_insens": [
"var_eq"
],
"base": {
"privatization": "none",
"context": {
"non-ptr": false
}
},
"thread": {
"domain": "plain",
"include-node": false
},
"race": {
"free": false
},
"dead-code": {
"lines": true
},
"int": {
"interval": true,
"def_exc": true
},
"malloc": {
"wrappers": [
"Malloc",
"Realloc",
"Malloc2",
"Realloc2",
"ARR_CreateInstance",
"realloc_array",
"ARR_GetNewElement"
]
}
},
"sem": {
"unknown_function": {
"spawn": false,
"invalidate": {
"globals": false,
"args": false
}
}
},
"solvers": {
"td3": {
"restart": {
"wpoint": {
"enabled": false
}
}
}
},
"exp": {
"earlyglobs": true
},
"cil": {
"merge": {
"inlines": false
}
},
"dbg": {
"timing": {
"enabled": true
}
},
"warn": {
"assert": false,
"behavior": false,
"integer": false,
"cast": false,
"race": true,
"deadcode": true,
"analyzer": false,
"unsound": true,
"imprecise": false,
"unknown": false,
"error": false,
"warning": true,
"info": false,
"debug": false,
"success": true
},
"incremental": {
"postsolver": {
"enabled": false
},
"restart": {
"sided": {
"enabled": false
},
"write-only": true
}
}
}
103 changes: 103 additions & 0 deletions scripts/incremental/benchmarking/conf/figlet-incrpostsolver.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
{
"ana": {
"activated": [
"expRelation",
"base",
"threadid",
"threadflag",
"threadreturn",
"escape",
"mutexEvents",
"mutex",
"access",
"mallocWrapper",
"mhp",
"symb_locks",
"var_eq",
"mallocFresh",
"race"
],
"ctx_insens": [
"var_eq"
],
"base": {
"privatization": "none",
"context": {
"non-ptr": false
}
},
"thread": {
"domain": "plain",
"include-node": false
},
"race": {
"free": false
},
"dead-code": {
"lines": true
},
"int": {
"interval": true,
"def_exc": true
}
},
"sem": {
"unknown_function": {
"spawn": false,
"invalidate": {
"globals": false,
"args": false
}
}
},
"solvers": {
"td3": {
"restart": {
"wpoint": {
"enabled": false
}
}
}
},
"exp": {
"earlyglobs": true
},
"cil": {
"merge": {
"inlines": false
}
},
"dbg": {
"timing": {
"enabled": true
}
},
"warn": {
"assert": false,
"behavior": false,
"integer": false,
"cast": false,
"race": true,
"deadcode": true,
"analyzer": false,
"unsound": true,
"imprecise": false,
"unknown": false,
"error": false,
"warning": true,
"info": false,
"debug": false,
"success": true
},
"incremental": {
"postsolver": {
"enabled": true
},
"restart": {
"sided": {
"enabled": false
},
"write-only": true
}
}
}
Loading