CHANGELOG

2025-01-10: PughLab pipeline-suite (version 0.9.14);
- MAJOR update to genotype_gvcfs.pl:
	- replaced germline correlation functions (very memory intensive) with bcftools gtcheck (compares genotypes for all germline/somatic variants and provides number of match/mismatch for each sample pair)
	- updated pughlab_pipeline_auto_report.pl to find new concordance file
	- updated plot_qc_metrics.R to reflect concordance (proportion of matching genotypes) rather than correlation
	- added create panel of normals within panelCN.mops.pl
- MINOR updates:
	- added bwa:n_cpus and panelCN.mops:pon parameters to tool config files
	- changed job dependencies within bwa.pl
	- changed order of loaded modules within novobreak.pl

2024-11-21: PughLab pipeline-suite (version 0.9.13);
- fixed bug in PGA calculation (collect_sequenza_output.R); previous only considered gains/amps
- MINOR updates throughout:
	- updated tool versions across pipelines (updated defaults in example config files)
	- harmonized parameter names across pipelines (ie, intervals_bed to targets_bed)
	- removed tool version defaults from individual steps (ie, module load perl must now be module load perl/<version>)
	- for tools that require matched normals, added checks and will now exit successfully if no normals are available

2024-10-23: PughLab pipeline-suite (version 0.9.12);
- fixed/improved samtools sort command (memory/thread arguments) in bwa.pl
- increased runtime for select variants step in annotate_germline.pl
- added clinvar (pathogenic) filter to annotate_germline.pl
- increased default limits for --limitOutSJcollapsed and --limitSjdbInsertNsj in fusioncatcher.pl
- added file checks to avoid re-running previously completed jobs in case of pipeline resume in
	- mutect2.pl, varscan.pl, pindel.pl, vardict_wgs.pl
- fixed cleanup step (for each patient, will only run if any jobs were submitted) in
	- manta.pl, pindel.pl
- fixed cleanup step for pughlab_fragmentomics_pipeline.pl
- improved extraction of sample ID from filename in collect_ascat_output.R
- standardized allele formatting (specifically for somaticsniper) in collect_snv_output.R
- minor updates to report functions including:
	- write_targetseq_methods.pl (minor updates for msi, svict)
	- plot_qc_metrics.R (fixed issue finding sample ID from contamination data)
	- plot_seqz_cna_summary.R (fixed issue for plot limits for incomplete (NA) data)
	- format_ensemble_mutations.R (fixed issue of duplicate entries in final variant set)

2024-07-22: PughLab pipeline-suite (version 0.9.11);
- changed input file extension from bcf to vcf in collect_delly_output.R
- added conditional variable assignments for RNA-Seq only runs (manta version and aligner reference)
- added stringsAsFactors = FALSE in combine_key_metrics.R
- fixed mavis command parameters in pughlab_rnaseq_pipeline.pl
- minor bug fixes to methyldackel.pl
- minor updates to accomodate RNA-Seq only runs in collect_mavis_output.R
- minor updates regarding missing data in collect_sequenza_output.R
- updates to cleanup step for pindel.pl and vardict_wgs.pl

2024-06-13: PughLab pipeline-suite (version 0.9.10);
- MAJOR update to pughlab_fragmentomics_pipeline.pl:
	- now accepts a list of site files for use with Griffin
	- see fragmentomics_pipeline_config.yaml for how to specify custom sites
	- end motif function now outputs 5' and 3' end motifs separately
	- allow specification of tool to use for deduplication steps (picard or sambamba)
	- updated n_cpus paramters for consistency (some n_cpu, now all n_cpus)
- MAJOR updates to ichorCNA:
	- added functionality for targeted sequencing (exome or target panels)
	- allow user to create or supply a custom panel of normals for ichorCNA
	- removed WGS requirement from utilities.pl
- update to bwa to allow specification of memory used for sort step (sets max memory per thread parameter, assuming 4 threads, based on memory available)
- update to make contest optional
- updates to methylDackel to indicate whether or not to test all cytosines or CpG only
- updates to strelka to resume failed instances (rather than restart from scratch)
- fixed final filtering step in delly to use per-sample filter (in addition to cohort filter)
- added error handling for single sample runs in collect_rnaseqc_output.R
- minor changes/code cleanup
	- collect_rsem_output.R and collect_agena_output.R
	- varscan.pl, svict.pl, star.pl, run_sequenza_with_optimal_gamma.pl (minor updates)
	- novobreak.pl, mavis.pl, contest.pl (cleanup)
- removed hardcoded paths and replaced with options found in tool config:
	- pindel.pl, svict.pl, mutect2.pl, mutect.pl, haplotypecaller.pl, genotype_gvcfs.pl, gatk.pl, ascat.pl
- updates to final reports:
	- added output folder for QC summary
	- removed hardcoded paths for MutSigCV
	- added error handling for tumour-only runs (format_ensemble_mutations.R)
	- added mavis to RNA-Seq methods

2024-05-02: PughLab pipeline-suite (version 0.9.9);
- added methyldackel to pughlab_emseq_pipeline.pl
	- updated methyldackel command (removed filter steps which are better incorporated into downstream analyses)
	- added n_cpus for methyldackel extract (methyldackel.pl and emseq_pipeline_config.yaml)
- updates to RNA-Seq report:
	- added Arriba virus counts to final report
	- minor updates to rna-based fusion summary
	- minor updates to rna-based snv summary
- minor updates:
	- fixed error in cleanup command in arriba.pl, pindel.pl, varscan.pl
	- improved error checking and data formatting in collect_arriba_output.R
	- minor formatting changes for consistency in collect_fusioncatcher_output.R and collect_star-fusion_output.R (in-frame to inframe)
	- added checks for output files to avoid re-running completed steps (mutect.pl [create-panel-of-normals])
	- minor update to vcf-concat and sort commands in mutect2.pl [when merging split-by-chr runs]
	- removed manta requirement for strelka.pl with create-panel-of-normals setting

2024-03-27: PughLab pipeline-suite (version 0.9.8);
- updated bwameth version/path (now available as a module on H4H)
- minor updates to example config files:
	- added manta step (removed from strelka) to dna pipeline config
	- added default version for bwa_meth + increased some mem/time parameters
- minor updates throughout scripts:
	- annotate_germline.pl (added sort steps to prevent out of order errors in vcf2maf)
	- collect_snv_output.R (added filtering for germline variants to remove duplicates)
	- bwa.pl (bwameth version/path)
	- mutect2.pl (removed non-existant file from cleanup)
	- novobreak.pl (changed n_cpu to n_cpus for consistency)
	- methyldackel.pl (mbias output file names)
	- strelka.pl (code cleanup; removed manta code)
	- trim_adapters.pl (fixed output file names for trim_galore)
- minor updates to report functions (added error checking and conditions)

2024-03-06: PughLab pipeline-suite (version 0.9.7);
- NEW: initial add of methyldackel.pl to run mbias and extract functions on EM-Seq
- pughlab_fragmentomics_pipeline.pl and fragmentomics_pipeline_config.yaml: added option to specify either number of reads or overall scale factor for downsample step
- minor update to error checks for EM-Seq pipeline (bait/target no longer required [ie, for WG])
- get_sequencing_metrics.pl: added HS metrics for all exome/targeted sequencing panels
- collect_rsem_output.R: removed 'var' column from cbioportal output file
- minor updates to summarize/report functions:
	- added package cosmicsig for standardization and annotation of mutation signatures (apply_cosmic_mutation_signatures.R)
	- added minor error checks (calculate_tmb.R, get_hrdetect_signatures.R, plot_expression_summary.R)
	- other minor bug fixes (plot_recurrent_mutations.R, plot_rna_snv_summary.R, plot_sv_summary.R, pughlab_pipeline_auto_report.pl)
	- added HS metrics for exome or targeted panels (plot_qc_metrics.R)
	- added data_config argument to extract sample information (plot_qc_metrics.R, pughlab_pipeline_auto_report.pl)
	- minor updates to generate_report.pl
- updates to example config files:
	- added/moved signature specification for cosmic sbs to summarize step (dna_pipeline_config)
	- initial add of methyldackel version/parameters (emseq_pipeline_config)


2024-02-15: PughLab pipeline-suite (version 0.9.6);
- NEW: added code to trim adapters (used for EM-Seq pipeline only)
- NEW: added convert_bam_to_cram.pl
- MISC UPDATES:
	- collect_fastqc_metrics.pl: added hpc_group specification
	- bwa, gatk and star: simplified formatting of output yaml
	- bwa and star: cleaned up flowcell usage
	- strelka: removed manta as this is now run separately (manta.pl)
	- pindel: added default insert size used for dryrun instead of failing when input files are not available yet
	- novobreak: updated parameter configuration and workflow (wip); added checks for final output files to avoid re-running early steps
	- varscan: added conditions for PoN creation; added more files to cleanup
	- vardict: updated conditions for PoN creation
	- star: added bam index for Aligned.sortedByCoord.out.bam
	- fragmentomics end-motif motif_get_contexts.R: split output for 5' and 3' motifs
	- minor updates to logging and status checks

2023-11-30: PughLab pipeline-suite (version 0.9.5);
- UPDATE: added ERDS (germine CNV caller) to pughlab_dnaseq_germline_pipeline.pl 
- added griffin output to collect_fragmentomics_output.R
- added cleanup step to pughlab_fragmentomics_pipeline.pl
- added example config for emseq pipeline (emseq_pipeline_config.yaml)
- minor updates to ERDS (cleanup and collection steps)

2023-11-29.2: PughLab pipeline-suite (version 0.9.4);
- NEW: initial addition of EM-Seq pipeline (wip)
        - pughlab_emseq_pipeline.pl to run the full EM-Seq pipeline (wip)
        - bwa.pl: incorporated bwameth into existing pipeline
        - get_sequencing_metrics.pl: added hs and gc bias metrics
- UPDATE: added flowcell argument to adjust markduplicates command (https://knowledge.illumina.com/instrumentation/general/instrumentation-general-faq-list/000007786):
        - bwa.pl
        - star.pl
        - utilities.pl (will force to 'random' if undefined in tool configs)
        - example dna and rna pipeline config yamls

2023-11-29: PughLab pipeline-suite (version 0.9.3);
- UPDATE: added griffin nucleosome profiling to fragmentomics pipeline; currently only validated with Griffin v0.1.0; example sites.yaml files can be found in scripts/fragmentomics/griffin/site_configs
	- pughlab_fragmentomics_pipeline.pl
	- fragmentomics_pipeline_config.yaml (specify paths to sites.yaml files and step parameters)
	- added reference files for griffin
- mavis.pl: updated filename for germline Delly input (bcf to vcf)
- gatk_gcnv.pl: minor updates; merged collect_gatk_gcnv_output.R into collect_gatk_cnv_output.R
- annotate_germline.pl: added sort command to keep_positions.txt
- collect_fragmentomics_output.R: write insert size histogram data to file
- report: minor updates throughout

2023-10-12: PughLab pipeline-suite (version 0.9.2);
- fastq yaml format has been standardized between dna and rna experiments
- fastqc: integrated into dna/rna pipelines
- get_sequencing_metrics: update to ensure matched normal pileup is included in dependencies for tumour contamination estimates
- delly: added final filter step for germline-only runs
- mavis: updates for compatibility with germline-only runs; added max_jobs to config files
- annotate_germline: added CPSR pathogenicity score to output MAFs
- report: assorted updates to handle germline-only projects

2023-10-07: PughLab pipeline-suite (version 0.9.1);
- mavis.pl: added arguments to customize memory/time for mavis jobs
- collect_delly_output.R: added check to identify which FORMAT tag contains CN information
- create_fastq_yaml.pl: added flexibility to input filenames
- NEW: create_data_yaml__tgl.R; can generate fastq or bam config yaml files when provided with directory containing these files; assumes filenames are in standard TGL format
- collect_sequencing_metrics/delly/manta: minor updates
- mutect/mutect2/strelka/varscan: updates to error checks and additional file cleanup for germline cases and/or PoN generation
- assorted updates for germline-only runs (utilities/example configs)

2023-09-26: PughLab pipeline-suite (version 0.9.0);
- NEW: initial addition of pughlab_fragmentomics_pipeline.pl; this pipeline examines fragmentation patterns within plasma WGS BAM files utilizing code developed by Derek Wong

2023-09-21: PughLab pipeline-suite (version 0.8.4);
- NEW: added manta.pl to allow for SV calling without subsequent variant calling steps (Strelka)
- NEW: initial addition of germline-only pipeline
- pindel: added minimum depth to filter step
- runIchorCNA: fixed a bug that prevented testing of alternate ploidies
- collect_snv_output: added germline filter for pindel calls
- format_ensemble_mutations: add VAF for each tool and clarified filters used (N tools; M2/VAF; within patient evidence)

2023-07-17: PughLab pipeline-suite (version 0.8.3);
- delly: updates to germline SV pipeline (spec. renaming merged files to be more reflective of contents)
- mavis: added germline SV calls from Delly/Manta to pipeline
- mavis: updated formatting of output files for cBioportal
- coverage: added per-interval coverage metrics for targeted-panel sequencing
- gatk_cnv: bug fix for target intervals in final collection step
- arriba: added option to increase n_cpus for arriba
- rsem: removed optional bam output
- vardict_wgs: intervals now run on a chromosome level to reduce the overall number of jobs submitted
- summary/report: fixed an error in calculate_tmb; other minor updates throughout

2023-02-27: PughLab pipeline-suite (version 0.8.2);
- NEW: data folder containing example intervals files (for WGS:VarDict) and Agena SNP positions (for genotype_gvcfs)
- pindel: skip patients without tumour samples (ie, normal-only); in merge step (for WGS or other split runs), skip/delete empty files as this creates issues downstream; for WGS (or other split runs), skip searching for tandem duplications (this step is VERY slow)
- delly: forces merge and subsequent steps to run if additional sample(s) added
- genotype_gvcfs: added function to look at Agena SNPs (will compare VAFs across the cohort)
- get_coverage and count_callable_bases: fixed an issue for callable bases that excluded patients with a single sample (ie, tumour only)
- improved readability of pipeline-suite output logs for genotype_gvcfs.pl
- fixed file called by pughlab_pipeline_auto_report.pl to plot recurrent snvs
- fixed typo for escaped characters in apply_cosmic_mutation_signatures.R
- minor update to handle missing data in summarize_mutation_data.R
- minor other updates/bug fixes to report functions

2023-01-06: PughLab pipeline-suite (version 0.8.1)
- improved readability of pipeline-suite output logs
- minor other updates/bug fixes to report functions

2022-12-21: PughLab pipeline-suite (version 0.8.0)
- MAJOR CHANGE (pughlab_pipeline_auto_report.pl): complete overhaul of summarize/report steps (complete reorganization [split summarize/report]; extra error checking; incorporated all new tools)
- NEW: plot_ascat_summary.R for summary plots for ASCAT output
- NEW: get_chord_signatures.R and get_hrdetect_signatures.R for HRD signatures (CHORD and HRDetect)
- NEW: calculate_tmb.R to calculate tumour mutation burden
- MINOR CHANGE: format_ensemble_mutations.R: added basic coverage/VAF filter to ensemble output
- MINOR CHANGE: apply_oncokb_filter_and_summarize.R: added set_tumour_only option to treat variants with low n_depth as tumour-only
- MINOR CHANGE: split plot_snv_summary.R into summarize_mutation_data.R (mutation-type frequency) and plot_recurrent_mutations.R (recurrence heatmap)
- MINOR CHANGE: plot_sv_summary.R: added better SV filters and circos plots
- annotate_germline: fixed a bug; now checks for final output prior to running intermediate steps
- ascat: added/fixed output for purity/ploidy estimates
- delly: added n_cpus option; removed delete intermediates step and added automatic re-run when new samples are added
- gatk_cnv: removed delete intermediates step and added automatic re-run when new samples are added to the PoN
- ichor_cna: added error checks to prevent previously complete steps from re-running
- collect_sequenza_output.R: added new output file useful for HRDetect
- minor other updates/bug fixes

2022-11-09: PughLab pipeline-suite (version 0.7.3)
- NEW: ascat.pl; run ASCAT on tumour/normal pairs
- QC: added Picard CollectInsertSizeMetrics
- Pindel: changed source for insert size to QC output file (CollectInsertSizeMetrics)
- updates to IchorCNA output collection
- minor other updates/bug fixes

2022-10-14: PughLab pipeline-suite (version 0.7.2)
- bwa: added n_cpus argument for bwa step
- bwa: added sambamba option for merge/markdup (picard merge/markdup fails with large WGS files)
- pindel: added minimum insert size (pindel fails if insert size provided is shorter than expected read length)
- delly: added multi-threading capability, which was added in recent delly versions (1.1.3+)
- haplotypeCaller: will run split by chromosome if input is WGS and running on slurm (uses array ability)
- VCF2MAF: added arguments to customize buffer size and number of forks used for VEP (can be specified in config.yaml)
- minor other updates/bug fixes

2022-08-15: PughLab pipeline-suite (version 0.7.1)
- NEW: arriba.pl; added Arriba for fusion detection in RNA-Seq

2022-07-22: PughLab pipeline-suite (version 0.7.0)
- bwa: added file used to check for completion of alignment step prior to sorting
- pindel: moved intermediate directory to sample directory (from patient directory) to avoid overwritting necessary files
- ichorCNA: fixed a bug/typo in the ichorCNA command (re: chromosomes to run)
- gatk_cnv: minor updates to account for different seq_types (targeted/exome/wgs)
- delly: added final data collection step for Delly (SV and CN calls per segment)
- mavis: minor bug fix in data collection step
- run_panelCN_mops: updates to gene annotation in data collection step
- NEW: erds_gcnv.pl; ERDS for WGS (g)CNV detection
- NEW: gatk_gcnv.pl; initial add GATK:gCNV for germline CNV detection

2022-06-13: PughLab pipeline-suite (version 0.6.8)
- updates to germline filtering
- updates to collection/annotation of mavis output
- updates to collection/annotation of sequencing metrics (ensures provided sample IDs are used rather than tags from BAM header)
- updates to method for counting callable bases (to count total/all genic/only coding bases)
- updates to collect_snv_data to filter ambiguous germline events
- sequenza: capped max CN at 9 (from default 20); other minor updates for clarity
- vardict: minor fix re:customizable java mem
- report: added methods for MSI
- report: minor change to ensemble code to accept maf (deals with header)
- report: minor change to snv_tool_overlap to cap N tool combinations shown

2022-03-29: PughLab pipeline-suite (version 0.6.7)
- updates to annotate_germline to consider recalibrated variants as candidates
- updates to pindel to ensure failed segments are not carried downstream
- for pindel, removed SV tracking for WGS to improve run times (removed pindel option from mavis for wgs)
- updates to collect_job_stats (all tools) for pbs compatibility
- updates throughout for use with vcf2maf module (alternative to providing path)
- updates to get_coverage.pl to split final data collection into two parts (coverage and callable bases)
- updates to count_callable_bases.R to find callable bases for different intervals (total,target,exon,cds)

2022-03-23: PughLab pipeline-suite (version 0.6.6)
- bug fixes for msi-sensor
- minor update for pbs

2022-03-18: PughLab pipeline-suite (version 0.6.5)
- added custom version/memory/time requirements for msi sensor
- updated method for annotated CNA segments for Sequenza and GATK_CNV output
- vardict (wgs): added additional check step to prevent completed steps from re-running on resume
- vardict: added argument to customize java mem (increasing mem previously had no effect on jvm)
- initial add of compatibility for PBS (job scheduling system)
- moved sub _get_files to utilities.pl (removed from all other locations)
- minor bug/efficiency fixes: gatk_cnv, get_sequencing_metrics, haplotype_caller, mutect, mutect2
- minor updates to report functions

2022-02-15: PughLab pipeline-suite (version 0.6.4)
- gatk: final check status now looks at correct job ID
- cleaned up check for/report RNA support (mavis output)
- fixed a few typos in the panelCNmops functions
- gatk_cnv: fixed a bug in cleanup step
- novobreak: removed duplicated code block
- genotype_gvcfs: added force overwrite for bgzip
- report/qc: extracts patient/sample info from qc directories instead of bwa (dna)
 
2021-12-10: PughLab pipeline-suite (version 0.6.3)
- added mavis to rnaseq pipeline (with additions to report functions as well)
- bug fix in convert_fusioncatcher.R
- added to cleanup step for novobreak
- minor updates to subset/vcf2maf steps in genotype_gvcfs.pl
- minor updates to report/plot functions

2021-11-30: PughLab pipeline-suite (version 0.6.2)
- moved common function (check_final_status) to utilities
- minor updates to report functions
- update to convert_fusioncatcher to catch empty input files
- pindel update to fix typo that caused files to be overwritten (merge output step)
- minor updates to mavis dependencies to avoid overloading the queue
- split novobreak process into 2 steps to reduce resource requirements

2021-11-05: PughLab pipeline-suite (version 0.6.1)
- many updates to summarize/report functions!
- added panelcn.mops output summary to report
- removed quotes from final output file from format_ensemble_mutations.R [required for cbioportal and mutsig]
- added germline SViCT data, and optional memory parameter, to Mavis runs
- BUG fix in mavis; was skipping to next sample if any SV tool had no data
- BUG fix in delly; would exclude variants if GT = ref in ANY sample (filtering performed BEFORE sample subset)

2021-10-29: PughLab pipeline-suite (version 0.6.0)
- initial add of panelCN.mops for targeted-seq data
- major update to genotype_gvcfs; added hard filtering and annotation options
- added more information to output log for mavis
- added simple filters (recurrence; short INDELs) to mavis output for cbioportal
- minor updates and bug fixes

2021-10-22: PughLab pipeline-suite (version 0.5.9)
- mavis output: cleaned up output and added new cbio sv format
- added optional vcf2maf step to genotype_gvcfs
- added option to customize recurrence threshold (number of samples) for PoN creation
- mavis: added step to extract key drawings and compress all intermediate files
- added read quality filter to pindel for high-confidence calls
- minor bug fixes to report scripts
- updated report to work with single sample data

2021-10-08: PughLab pipeline-suite (version 0.5.8)
- fixed bug in convert_svict that removed ALL rows
- removed unnecessary warnings in mavis for tumour-only samples
- fixed issue in novobreak for finding required files
- updates to summarize and report functions
- update: added 'group' option for job submissions

2021-09-30: PughLab pipeline-suite (version 0.5.7)
- turned all tools off by default
- added extra layer of dependencies for pughlab_dnaseq_pipeline.pl --variant_calling to reduce failure due to job overload
- fixes to convert_novobreak and convert_pindel for compatibility with mavis
- updated paths for panel of normals for mutect and mutect2
- added checks to make panel of normals (mutect, mutect2, strelka, vardict, varscan)
- added checks to run final combine step (where necessary)
- split mutect2 to run per-chromosome (somatic and artefact-detection)
- simplified novobreak (removed split by chr-pairs); this requires more time/memory, but is much less error prone
- minor edits/bug fixes throughout

2021-09-22: PughLab pipeline-suite (version 0.5.6)
- added single-end alignment option to bwa.pl
- initial add of ichorCNA
- updated error checking criteria for pindel, novobreak, svict and ichorCNA

2021-09-20: PughLab pipeline-suite (version 0.5.5)
- minor updates to pindel pipeline to improve run filtering
- minor fix to novobreak (missing n_cpus args)
- minor fix to pughlab_dnaseq_pipeline.pl (typo)

2021-09-17: PughLab pipeline-suite (version 0.5.4)
- changed 'create_report' argument to 'summarize' to more accurately reflect function
- added additional info to logs for use with debugging
- updates to pindel and novobreak (wip; trying to improve mem/time/space requirements)
- minor updates to plotting functions for final report 

2021-08-24: PughLab pipeline-suite (version 0.5.3)
- initial add of Pindel
- updates to SViCT pipeline
- addition/update to mavis pipeline (convert scripts + processing new tools)

2021-07-27: PughLab pipeline-suite (version 0.5.2)
- minor updates to report code (plotting and ensemble caller)
- adding mutation signatures to final report

2021-07-15: PughLab pipeline-suite (version 0.5.1)
- function to convert novobreak output for mavis
- initial add of SViCT

2021-07-08: PughLab pipeline-suite (version 0.5.0)
- initial add of this log
- updated readme [moved most information to the wiki]
- initial addition of novobreak
- fixed method for finding cwd
- minor updates to documentation and final report