Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Auto-generate benchmarks with genthat, and run them #22

Open
wants to merge 82 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
82 commits
Select commit Hold shift + click to select a range
bad7b51
Tooling for benchmarks generated by genthat.
vogr Aug 3, 2021
dee0ff7
Keep some generated tests (from yaml package) in the repo.
vogr Aug 3, 2021
23033e7
Quote entry name for genthat benchmarks in rebench.conf.
vogr Aug 3, 2021
798fb9b
Add script to download and install R packages.
vogr Aug 3, 2021
ec641dc
Download and install in a single script.
vogr Aug 3, 2021
cafbeb8
Commit packages.txt file
vogr Aug 3, 2021
617f4c1
Specify repo when downloading devtools
vogr Aug 3, 2021
0d47711
Do not depend on genthat for setup_R_packages.sh
vogr Aug 3, 2021
eee62cd
Add genthat-CRAN in the suits in rebench.conf
vogr Aug 3, 2021
640490b
Add prodlim tests in the repo.
vogr Aug 3, 2021
b15a4b3
Improve script to install packages: only use stdlib functions.
vogr Aug 5, 2021
079d6f0
Remove freeze.sh script: no needed since we use MRAN.
vogr Aug 5, 2021
a2bc055
intall_pkgs.sh: make saving the source optionnal.
vogr Aug 5, 2021
ac3bbb7
Script to extract testcases from the installed packages.
vogr Aug 5, 2021
079420b
Add parameter to change number of inner iterations.
vogr Aug 5, 2021
45d74f2
Fix check in number of arguments.
vogr Aug 5, 2021
5108f1d
Add README.md explaining how to extract testcases using the Dockerfile.
vogr Aug 5, 2021
54c8b5c
Install dependancies for extract_testcases.
vogr Aug 5, 2021
6d843d7
Add Dockerfile to automate test extraction.
vogr Aug 5, 2021
93647dc
Add main entrypoint script for Dockerfile
vogr Aug 5, 2021
536b228
Typos in README and run.sh
vogr Aug 5, 2021
973e519
Split installation of genthat and of pkgs in two scripts.
vogr Aug 5, 2021
11ede08
Typo in README.md
vogr Aug 5, 2021
94bdf27
Also pin CRAN snapshot for installation of genthat.
vogr Aug 5, 2021
c6046ca
Do not try to extract source to null directory.
vogr Aug 5, 2021
793ed35
Optionnaly, do not specify install directory
vogr Aug 5, 2021
b58a7a3
Do not put class name to lower in genthat benchmarks
vogr Aug 5, 2021
b5d7d32
Count number of inner-iterations necessary to reach 200ms of runtime …
vogr Aug 5, 2021
3f7518d
Correctly generate the benchmarks: only keep function that can be run by
vogr Aug 5, 2021
14240c9
Read number of inner-iterations from the file previously generated
vogr Aug 5, 2021
81513b9
Correctly read nb iteration from dictionnary.
vogr Aug 5, 2021
7a39153
Keep number of inner-iteration saved in the repo.
vogr Aug 5, 2021
8b35f25
Shorter names fir inner-it determination scripts.
vogr Aug 5, 2021
dd2f454
Cleanup extract_testcases script.
vogr Aug 5, 2021
2dc80f3
Use file specifying number of inner it for each benchmark when
vogr Aug 5, 2021
7c2b52c
Check result for each inner iteration (out of the timed loop)
vogr Aug 6, 2021
1929d1b
Wrap function_to_run() result in a list() to prevent NULL results from
vogr Aug 6, 2021
79b6de3
Remove old yaml tests.
vogr Aug 6, 2021
17a2bbc
More documentation.
vogr Aug 6, 2021
24de287
Save retv and check it as a separate step (ie not directly in genthat…
vogr Aug 6, 2021
cc2d980
Add the recording and checking steps to the container main command
vogr Aug 6, 2021
3c04605
Remove old versions of generated prodlim benchmarks
vogr Aug 6, 2021
68e6971
Record_retv: run a new R session for every testcase
vogr Aug 9, 2021
273142d
Add helper scripts.
vogr Aug 10, 2021
52fd1c8
Make record_retv parallel
vogr Aug 10, 2021
5e7b310
Add comment about parallelism in extract_testcases
vogr Aug 10, 2021
32848e0
Make check against_recorded_retv parallel
vogr Aug 10, 2021
0b83398
Change working directory in parallel loop for record_retv
vogr Aug 10, 2021
c402938
Set WD in parallel loop for extract_testcases.
vogr Aug 10, 2021
c67d826
Do not use next in parallel loop
vogr Aug 10, 2021
4a72ed0
List callr as dep
vogr Aug 10, 2021
98a92c9
Complete README with manual instructions.
vogr Aug 10, 2021
2c9c7bc
Correctly save seed in record_retv
vogr Aug 10, 2021
e7ec0c6
Typo
vogr Aug 10, 2021
645f500
Use parallel record_retv in instructions.
vogr Aug 10, 2021
da79d8a
Correctly save seed.
vogr Aug 10, 2021
f039551
Add selection of generated tests with their retv (filtered to keep only
vogr Aug 10, 2021
943b5a4
Use same pattern for extension added to failed tests.
vogr Aug 10, 2021
5799897
Add comment about isolation in min_nb_iter
vogr Aug 10, 2021
f4780e3
Update usage of min_nb_iter.R
vogr Aug 10, 2021
f589d2e
Don't load doParallel in scripts, use it through namespace.
vogr Aug 10, 2021
4dfdbc1
Move capture.output outside of the hot loop in min_inner_it computation.
vogr Aug 10, 2021
a999691
min_nb_iter: source in a user-created environment
vogr Aug 10, 2021
4ea7194
Update n_inner_it command in README
vogr Aug 10, 2021
bccf9e8
Commit n_inner_it for the 2000 first testcases.
vogr Aug 10, 2021
5e6cb03
Update packages.txt
vogr Aug 10, 2021
68ecea1
Remove mistakenly added line in packages.txt
vogr Aug 10, 2021
e6d0241
Keep only the extracted testcases from the 100 first packages.
vogr Aug 10, 2021
89ac82d
Document scripts in inner_it
vogr Aug 10, 2021
c99ff8d
Genthat is needed when running the benchmarks.
vogr Aug 10, 2021
fc0ee9f
Use the right result variable in the debug message.
vogr Aug 10, 2021
fcdec7e
Quote benchmark name in rebench command
vogr Aug 11, 2021
387e911
Normalize path of extfile and Rfile in genthat-CRAN harness
vogr Aug 11, 2021
6e9bfa5
Move the generated, but currently unused, genthat-benchmark files out
vogr Aug 13, 2021
c2a88bb
Make the genthat files compatible with the new harness.
vogr Aug 13, 2021
e0430e5
Make the genthat harness compatible with the new genthat files.
vogr Aug 13, 2021
deaae4f
Remove the archived tests
vogr Aug 13, 2021
e3b7745
Keep only a few tests.
vogr Aug 13, 2021
af50f50
Record retv is now integrated in genthat.
vogr Aug 13, 2021
c3ddcbf
Use the PRL-PRG branch for the modified genthat, instead of the one
vogr Aug 13, 2021
ff7fc06
Use correct retv name in logging in harness.
vogr Aug 13, 2021
12573c1
Update the extfile so that they match the expected format.
vogr Aug 13, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions Benchmarks/genthat-CRAN/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# Genthat testcases

These benchmarks are automatically extracted from CRAN packages using genthat by the scripts in `Setup/genthat`.

Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@


genthat_extracted_call <- function() {
APCanalysis:::EERpenalties(n = 8, k = 5, m = 5, eer = 0.25, reps = 15000)
}




Binary file not shown.
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@


genthat_extracted_call <- function() {
APCanalysis:::FDRpenalties(n = 8, k = 5, m = 5, fdr = 0.05, reps = 12000)
}




Binary file not shown.
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@


genthat_extracted_call <- function() {
APCanalysis:::IERpenalties(n = 8, k = 5, m = 5, ier = 0.01, reps = 15000)
}




Binary file not shown.
Binary file not shown.
11 changes: 11 additions & 0 deletions Benchmarks/genthat-CRAN/generated/APCanalysis/apc/test-1.R
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@


genthat_extracted_call <- function() {
PB12response <- APCanalysis::PB12response
PB12matrix <- APCanalysis::PB12matrix
APCanalysis:::apc(y = PB12response, x = PB12matrix, maxsize = 6, level = 0.2, method = 2, reps = 10000)
}




Binary file not shown.
Binary file not shown.
18 changes: 18 additions & 0 deletions Benchmarks/genthat-CRAN/generated/APCanalysis/plot.apc/test-1.R
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@


genthat_extracted_call <- function() {
my.apc <- structure(list(Results = structure(list(Model = structure(c(7L, 3L, 1L, 6L, 2L, 4L, 5L), .Label = c("+ B",
"+ D", "+ H", "+ I", "+ J", "+ K", "intercept"), class = "factor"), Size = 0:6, RSS = c(2851.2042, 1533.449, 997.8466,
491.4964, 160.6414, 8.3445, 4.0604), APC = c(7.9555, 7.9483, 8.2006, 8.2455, 7.9602, 5.9386, 6.2103), Active = structure(c(2L,
2L, 2L, 2L, 2L, 2L, 1L), .Label = c(" ", "* "), class = "factor")), class = "data.frame", row.names = c("", "xH",
"xB", "xK", "xD", "xI", "xJ")), Penalties = c(0, 0.613, 1.295, 2.048, 2.881, 3.817, 4.809), Level = 0.2, ErrorType = "EER",
k = 11L, maxsize = 6, apc = structure(5.93860463652268, names = "xI"), Ests = structure(c(4.8275, 10.4791666666667,
6.68083333333333, 6.49583333333334, -5.25083333333334, -3.5625, -0.597499999999997, 0.377500000000002, 0.312500000000001,
0.2725, -0.135833333333332, -0.0741666666666676), names = c("(Intercept)", "xH", "xB", "xK", "xD", "xI", "xJ",
"xC", "xF", "xG", "xA", "xE")), ActEffs = "B, D, H, I, K", NonActEffs = "A, C, E, F, G, J"), class = "apc")
APCanalysis:::plot.apc(x = my.apc)
}




Binary file not shown.
18 changes: 18 additions & 0 deletions Benchmarks/genthat-CRAN/generated/APCanalysis/summary.apc/test-1.R
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@


genthat_extracted_call <- function() {
my.apc <- structure(list(Results = structure(list(Model = structure(c(7L, 3L, 1L, 6L, 2L, 4L, 5L), .Label = c("+ B",
"+ D", "+ H", "+ I", "+ J", "+ K", "intercept"), class = "factor"), Size = 0:6, RSS = c(2851.2042, 1533.449, 997.8466,
491.4964, 160.6414, 8.3445, 4.0604), APC = c(7.9555, 7.9483, 8.2006, 8.2455, 7.9602, 5.9386, 6.2103), Active = structure(c(2L,
2L, 2L, 2L, 2L, 2L, 1L), .Label = c(" ", "* "), class = "factor")), class = "data.frame", row.names = c("", "xH",
"xB", "xK", "xD", "xI", "xJ")), Penalties = c(0, 0.613, 1.295, 2.048, 2.881, 3.817, 4.809), Level = 0.2, ErrorType = "EER",
k = 11L, maxsize = 6, apc = structure(5.93860463652268, names = "xI"), Ests = structure(c(4.8275, 10.4791666666667,
6.68083333333333, 6.49583333333334, -5.25083333333334, -3.5625, -0.597499999999997, 0.377500000000002, 0.312500000000001,
0.2725, -0.135833333333332, -0.0741666666666676), names = c("(Intercept)", "xH", "xB", "xK", "xD", "xI", "xJ",
"xC", "xF", "xG", "xA", "xE")), ActEffs = "B, D, H, I, K", NonActEffs = "A, C, E, F, G, J"), class = "apc")
APCanalysis:::summary.apc(object = my.apc)
}




Binary file not shown.
10 changes: 10 additions & 0 deletions Benchmarks/genthat-CRAN/generated/BASIX/BASIX.combnapply/test-1.R
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@


genthat_extracted_call <- function() {
vec <- c(1, 3, 5, 7, 9, 10)
BASIX:::BASIX.combnapply(vec = vec)
}




Binary file not shown.
11 changes: 11 additions & 0 deletions Benchmarks/genthat-CRAN/generated/BASIX/BASIX.equal/test-1.R
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@


genthat_extracted_call <- function() {
b <- c(1, 9, 3, 4, 5, 6, 7, 8, 9, 10)
a <- 1:10
BASIX:::BASIX.equal(a = a, b = b)
}




Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@


genthat_extracted_call <- function() {
vec <- c(1, 3, 5, 7, 9, 10)
to <- 8
from <- 5
BASIX:::BASIX.find.interval(vec = vec, from = from, to = to, start = 1)
}




Binary file not shown.
11 changes: 11 additions & 0 deletions Benchmarks/genthat-CRAN/generated/BASIX/BASIX.match/test-1.R
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@


genthat_extracted_call <- function() {
vec <- c(2, 3, 4, 10, 11)
elements <- c(1, 2, 10)
BASIX:::BASIX.match(elements = elements, vec = vec)
}




Binary file not shown.
10 changes: 10 additions & 0 deletions Benchmarks/genthat-CRAN/generated/BASIX/BASIX.table/test-1.R
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@


genthat_extracted_call <- function() {
mat <- structure(c(1, 0, 0, 0, 0, 0, 0, 0, 0), .Dim = c(3L, 3L))
BASIX:::BASIX.table(matrix = mat)
}




Binary file not shown.
10 changes: 10 additions & 0 deletions Benchmarks/genthat-CRAN/generated/BASIX/BASIX.unique/test-1.R
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@


genthat_extracted_call <- function() {
mat <- structure(c(1, 0, 0, 0, 0, 0, 0, 0, 0), .Dim = c(3L, 3L))
BASIX:::BASIX.unique(matrix = mat)
}




Binary file not shown.
984 changes: 984 additions & 0 deletions Benchmarks/genthat-CRAN/generated/abc/abc/test-2.R

Large diffs are not rendered by default.

Binary file not shown.
630 changes: 630 additions & 0 deletions Benchmarks/genthat-CRAN/generated/abc/expected.deviance/test-1.R

Large diffs are not rendered by default.

Binary file not shown.
207 changes: 207 additions & 0 deletions Benchmarks/genthat-CRAN/generated/abc/getmode/test-100.R

Large diffs are not rendered by default.

Binary file not shown.
1,624 changes: 1,624 additions & 0 deletions Benchmarks/genthat-CRAN/generated/abc/normalise/test-100.R

Large diffs are not rendered by default.

Binary file not shown.
12 changes: 12 additions & 0 deletions Benchmarks/genthat-CRAN/generated/abc/plot.gfit/test-1.R
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@


genthat_extracted_call <- function() {
res.gfit.bott <- structure(list(dist.sim = c(1.94792260837597, 2.48685540768078, 2.20072789722867, 1.90543387838366,
2.02150262739967, 1.77656019685176, 1.91338570248844, 1.82429571532952, 2.64783696107181, 2.71029891434803), dist.obs = 1.99776885122106),
class = "gfit")
abc:::plot.gfit(x = res.gfit.bott, main = "Histogram under H0")
}




Binary file not shown.
Loading