updating with final week results

README.md

@@ -13,6 +13,7 @@ The initial release just included the final results files that are reviewed afte
 Since then, the following information has been added:
 
 - (2021-12-13) Descriptions and links to the specific reference files
+- (2022-02-08) Descriptions and links to the benchmark truth sets (VCF and BED when available)
 
 We rarely change the reference files we are testing with.
 However, aligners and callers are added/updated semi-frequently depending on the significance of the change and reported performance by the developers/researchers.
@@ -37,3 +38,7 @@ Examples of things already on the TODO radar:
 - Rules for aligners and callers
 - Rules for evaluation
 - ~~Description/links to specific reference files~~
+
+## Release cadence
+Starting with Release 2022-02-11, we intend to restrict the formal releases to _at most_ once a week.
+New or potentially partial results may appear throughout the work week, but we intend to summarize these with a single formal release each week.

results/20220211/small_summary_20220211.csv

@@ -0,0 +1,72 @@
+prep,reference,aligner,caller,N,Precision,Sensitivity,F-measure,vc_run_time
+PCR-free-illumina,clinical,sentieon-201808.07,strelka-2.9.10,8,0.992325,0.9919875,0.992175,-1.0
+PCR-free-illumina,hg38_GIAB_masked,dragmap-1.2.1,clair3-0.1-r9,8,0.9976375000000001,0.992425,0.995025,58347.462325
+PCR-free-illumina,hg38_GIAB_masked,dragmap-1.2.1,deepvariant-1.2.0,8,0.9989375,0.9928125000000001,0.9958750000000001,74635.7289875
+PCR-free-illumina,hg38_GIAB_masked,dragmap-1.2.1,dnascope-0.5-202010.04-PO,8,0.99845,0.9924249999999999,0.995425,-1.0
+PCR-free-illumina,hg38_GIAB_masked,dragmap-1.2.1,dnascope-1.0-202112.01-PO,8,0.9984375,0.994225,0.9963124999999999,-1.0
+PCR-free-illumina,hg38_GIAB_masked,dragmap-1.2.1,octopus-0.7.4,8,0.9962,0.9899249999999999,0.9930625,45715.272287499996
+PCR-free-illumina,hg38_GIAB_masked,dragmap-1.2.1,sentieon_gatk4-202010.04,8,0.9912749999999999,0.9924,0.9918499999999999,11849.36745
+PCR-free-illumina,hg38_GIAB_masked,dragmap-1.2.1,strelka-2.9.10,8,0.9893125,0.9921500000000001,0.9907499999999999,9014.550025
+PCR-free-illumina,hg38_GIAB_masked,sentieon-202010.04-recal,clair3-0.1-r9,8,0.9973875,0.9925999999999999,0.9949875,63847.848675
+PCR-free-illumina,hg38_GIAB_masked,sentieon-202010.04-recal,deepvariant-1.2.0,8,0.99865,0.993475,0.9960500000000001,73980.897375
+PCR-free-illumina,hg38_GIAB_masked,sentieon-202010.04-recal,dnascope-0.5-202010.04-PO,8,0.9983124999999999,0.9912375,0.99475,-1.0
+PCR-free-illumina,hg38_GIAB_masked,sentieon-202010.04-recal,dnascope-1.0-202112.01-PO,8,0.9985625,0.9931625,0.9958500000000001,-1.0
+PCR-free-illumina,hg38_GIAB_masked,sentieon-202010.04-recal,octopus-0.7.4,8,0.994675,0.9902124999999999,0.99245,49050.7662
+PCR-free-illumina,hg38_GIAB_masked,sentieon-202010.04-recal,sentieon_gatk4-202010.04,8,0.988775,0.9931125000000001,0.9909375,11090.670699999999
+PCR-free-illumina,hg38_GIAB_masked,sentieon-202010.04-recal,strelka-2.9.10,8,0.990525,0.9929749999999999,0.9917500000000001,10790.0723875
+PCR-free-illumina,hg38_GIAB_masked,sentieon-202010.04,clair3-0.1-r9,8,0.9973624999999999,0.9935875000000001,0.9954625,61328.3111125
+PCR-free-illumina,hg38_GIAB_masked,sentieon-202010.04,deepvariant-1.2.0,8,0.9988374999999999,0.9937374999999999,0.9962625,81445.1432625
+PCR-free-illumina,hg38_GIAB_masked,sentieon-202010.04,dnascope-0.5-202010.04-PO,8,0.99795,0.9927375,0.9953375,-1.0
+PCR-free-illumina,hg38_GIAB_masked,sentieon-202010.04,dnascope-1.0-202112.01-PO,8,0.9982875,0.9944,0.9963375,-1.0
+PCR-free-illumina,hg38_GIAB_masked,sentieon-202010.04,octopus-0.7.4,8,0.9945875000000001,0.9902625,0.9924125,47852.1594125
+PCR-free-illumina,hg38_GIAB_masked,sentieon-202010.04,sentieon_gatk4-202010.04,8,0.986475,0.9933749999999999,0.9899249999999999,13900.056799999998
+PCR-free-illumina,hg38_GIAB_masked,sentieon-202010.04,strelka-2.9.10,8,0.9869375,0.99315,0.9900374999999999,9290.641974999999
+PCR-free-illumina,hg38_GIAB_masked,snap-2.0.0,clair3-0.1-r9,8,0.996575,0.9928,0.994675,60180.919675
+PCR-free-illumina,hg38_GIAB_masked,snap-2.0.0,deepvariant-1.2.0,8,0.998425,0.9936750000000001,0.9960375,67940.365925
+PCR-free-illumina,hg38_GIAB_masked,snap-2.0.0,dnascope-0.5-202010.04-PO,8,0.9972875000000001,0.9932125,0.995275,-1.0
+PCR-free-illumina,hg38_GIAB_masked,snap-2.0.0,dnascope-1.0-202112.01-PO,8,0.9975875000000001,0.9942375,0.9959250000000001,-1.0
+PCR-free-illumina,hg38_GIAB_masked,snap-2.0.0,octopus-0.7.4,8,0.9938,0.9906625,0.9922249999999999,47899.824974999996
+PCR-free-illumina,hg38_GIAB_masked,snap-2.0.0,sentieon_gatk4-202010.04,8,0.9848875,0.99385,0.98935,13891.268712500001
+PCR-free-illumina,hg38_GIAB_masked,snap-2.0.0,strelka-2.9.10,8,0.9861,0.9934,0.9897375,9228.167775
+PCR-free-illumina,hg38_asm5_alt,dragmap-1.2.1,clair3-0.1-r9,8,0.9975499999999999,0.9705,0.9838250000000001,58263.8051875
+PCR-free-illumina,hg38_asm5_alt,dragmap-1.2.1,deepvariant-1.2.0,8,0.998625,0.9708749999999999,0.98455,72233.8049375
+PCR-free-illumina,hg38_asm5_alt,dragmap-1.2.1,dnascope-0.5-202010.04-PO,8,0.9982875,0.9709749999999999,0.9844125,-1.0
+PCR-free-illumina,hg38_asm5_alt,dragmap-1.2.1,dnascope-1.0-202112.01-PO,8,0.9978,0.9735375000000001,0.9855125,-1.0
+PCR-free-illumina,hg38_asm5_alt,dragmap-1.2.1,octopus-0.7.4,8,0.996875,0.9680875,0.9822875,41904.761075
+PCR-free-illumina,hg38_asm5_alt,dragmap-1.2.1,sentieon_gatk4-202010.04,8,0.9935375,0.9701500000000001,0.9817125,10794.6066
+PCR-free-illumina,hg38_asm5_alt,dragmap-1.2.1,strelka-2.9.10,8,0.9901875,0.969925,0.9799625000000001,8291.4376875
+PCR-free-illumina,hg38_asm5_alt,parabricks-3.5.0,germline,1,0.9909,0.9944,0.9927,-1.0
+PCR-free-illumina,hg38_asm5_alt,parabricks-3.6.0,deepvariant,1,0.9986,0.9945,0.9965,-1.0
+PCR-free-illumina,hg38_asm5_alt,sentieon-202010.04-recal,clair3-0.1-r9,8,0.9977750000000001,0.991625,0.9946999999999999,62775.0722125
+PCR-free-illumina,hg38_asm5_alt,sentieon-202010.04-recal,deepvariant-1.2.0,8,0.998625,0.9925499999999999,0.9955875000000001,72320.34945000001
+PCR-free-illumina,hg38_asm5_alt,sentieon-202010.04-recal,dnascope-0.5-202010.04-PO,8,0.9983,0.990675,0.9944625,-1.0
+PCR-free-illumina,hg38_asm5_alt,sentieon-202010.04-recal,dnascope-1.0-202112.01-PO,8,0.9985999999999999,0.9925875,0.995575,-1.0
+PCR-free-illumina,hg38_asm5_alt,sentieon-202010.04-recal,octopus-0.7.4,8,0.995975,0.9892875,0.9926125,45915.7730125
+PCR-free-illumina,hg38_asm5_alt,sentieon-202010.04-recal,sentieon_gatk4-202010.04,8,0.9922,0.9921625000000001,0.9921625,10009.022425000001
+PCR-free-illumina,hg38_asm5_alt,sentieon-202010.04-recal,strelka-2.9.10,8,0.992325,0.9919875,0.992175,10465.407799999999
+PCR-free-illumina,hg38_asm5_alt,sentieon-202010.04,clair3-0.1-r9,8,0.9977,0.9925999999999999,0.9951125000000001,61921.3732
+PCR-free-illumina,hg38_asm5_alt,sentieon-202010.04,deepvariant-1.2.0,8,0.9988125,0.9927,0.995725,74472.74207499999
+PCR-free-illumina,hg38_asm5_alt,sentieon-202010.04,dnascope-0.5-202010.04-PO,8,0.9979875,0.992075,0.9950375,-1.0
+PCR-free-illumina,hg38_asm5_alt,sentieon-202010.04,dnascope-1.0-202112.01-PO,8,0.9983375000000001,0.993775,0.9960500000000001,-1.0
+PCR-free-illumina,hg38_asm5_alt,sentieon-202010.04,octopus-0.7.4,8,0.9958374999999999,0.9893000000000001,0.99255,46081.922099999996
+PCR-free-illumina,hg38_asm5_alt,sentieon-202010.04,sentieon_gatk4-202010.04,8,0.9899,0.9924375,0.9911625,12472.8063125
+PCR-free-illumina,hg38_asm5_alt,sentieon-202010.04,strelka-2.9.10,8,0.9890125000000001,0.9921374999999999,0.990575,9537.153
+PCR-free-illumina,hg38_asm5_alt,snap-2.0.0,clair3-0.1-r9,8,0.9967250000000001,0.9927,0.9946875,56418.9348375
+PCR-free-illumina,hg38_asm5_alt,snap-2.0.0,deepvariant-1.2.0,8,0.9984,0.9936499999999999,0.996,63108.32505000001
+PCR-free-illumina,hg38_asm5_alt,snap-2.0.0,dnascope-0.5-202010.04-PO,8,0.9973000000000001,0.9932124999999999,0.9952625,-1.0
+PCR-free-illumina,hg38_asm5_alt,snap-2.0.0,dnascope-1.0-202112.01-PO,8,0.9976375,0.9941375000000001,0.9958750000000001,-1.0
+PCR-free-illumina,hg38_asm5_alt,snap-2.0.0,octopus-0.7.4,8,0.9943875,0.9905125,0.9924625,45841.266975
+PCR-free-illumina,hg38_asm5_alt,snap-2.0.0,sentieon_gatk4-202010.04,8,0.986675,0.993725,0.9901875,14853.2475625
+PCR-free-illumina,hg38_asm5_alt,snap-2.0.0,strelka-2.9.10,8,0.9868125,0.99325,0.9900125,9125.236850000001
+PacBio-CCS-high,hg38_GIAB_masked,minimap2-2.23,clair3-0.1-r9,7,0.9979999999999999,0.9986857142857143,0.9983285714285713,22684.995342857143
+PacBio-CCS-high,hg38_GIAB_masked,minimap2-2.23,deepvariant-1.2.0,7,0.9976142857142858,0.9975571428571429,0.9975857142857143,61487.09210000001
+PacBio-CCS-high,hg38_GIAB_masked,minimap2-2.23,dnascope_hifi-0.4-202010.04,7,0.9986285714285714,0.9986285714285715,0.9986285714285714,40171.3795
+PacBio-CCS-high,hg38_GIAB_masked,minimap2-2.23,peppermargindv-r0.7,7,0.9982714285714286,0.9984142857142857,0.9983428571428572,56577.98644285714
+PacBio-CCS-high,hg38_GIAB_masked,pbmm2-1.7.0,clair3-0.1-r9,7,0.9983857142857142,0.9986857142857143,0.9985428571428573,18095.315285714285
+PacBio-CCS-high,hg38_GIAB_masked,pbmm2-1.7.0,dnascope_hifi-0.4-202010.04,7,0.9987285714285715,0.9987428571428573,0.9987428571428572,41551.35887142857
+PacBio-CCS-high,hg38_GIAB_masked,pbmm2-1.7.0,peppermargindv-r0.7,7,0.9985571428571428,0.9983428571428572,0.9984571428571429,47794.900299999994
+PacBio-CCS-high,hg38_GIAB_masked,sentieon_mm2-202010.04,clair3-0.1-r9,7,0.9979857142857141,0.9986857142857143,0.9983428571428572,21963.831142857143
+PacBio-CCS-high,hg38_GIAB_masked,sentieon_mm2-202010.04,deepvariant-1.2.0,7,0.9976142857142858,0.9975571428571429,0.9975857142857143,65419.06621428572
+PacBio-CCS-high,hg38_GIAB_masked,sentieon_mm2-202010.04,dnascope_hifi-0.4-202010.04,7,0.9986428571428572,0.9986285714285715,0.9986285714285714,42779.48097142857
+PacBio-CCS-high,hg38_GIAB_masked,sentieon_mm2-202010.04,peppermargindv-r0.7,7,0.9982714285714286,0.9984142857142857,0.9983428571428572,47890.72687142857
+PacBio-CCS-high,hg38_noalt,sentieon_mm2-202010.04,dnascope_hifi-0.4-202010.04,7,0.9986428571428572,0.9986285714285715,0.9986142857142857,39470.55577142857