Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Develop #13

Merged
merged 68 commits into from
Nov 28, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
edee46d
Added version print statements to all tasks.
williamrowell May 20, 2023
74d14fa
CommandShellCheck, SC2046 Quote this to prevent word splitting.
williamrowell May 20, 2023
bc86fa6
update wdl-ci config file after successful tests
github-actions[bot] May 20, 2023
c631102
Fix htsbox version command.
williamrowell May 20, 2023
28d0011
update wdl-ci config file after successful tests
github-actions[bot] May 20, 2023
052365f
update submodule
williamrowell May 24, 2023
fcbd54c
Create LICENSE
williamrowell Jun 2, 2023
de2e218
Set max retries to 3 for HPC backend configuration
hkeward Jun 6, 2023
0ba71f4
Format docs; add backend-specific docs
hkeward Jun 27, 2023
e1b4074
Update paths for AWS reference files
hkeward Jun 27, 2023
c7714fe
Allow user to override container registry
hkeward Jul 4, 2023
5d1299b
Avoid underestimating disk size based on first element of array.
williamrowell Aug 11, 2023
cb9f17f
update wdl-ci config file after successful tests
github-actions[bot] Aug 11, 2023
6ea580a
Merge pull request #7 from PacificBiosciences/bugfix/disk-size-arrays
williamrowell Aug 14, 2023
9c6c96e
Calculate total bases input for each parent to set yak params on the fly
gconcepcion Aug 16, 2023
c9f7092
less than not greater than
gconcepcion Aug 16, 2023
e873f20
Document submitting the workflow directly to Cromwell using cURL
hkeward Aug 23, 2023
902f926
Add dependencies.zip to gitignore
hkeward Aug 23, 2023
f1e76ea
Merge pull request #8 from PacificBiosciences/docs/cromwell-curl-command
hkeward Aug 23, 2023
56833d4
Adding multi-reference alignment option
gconcepcion Sep 22, 2023
a40d2a2
add yak bloom filter condition
gconcepcion Sep 22, 2023
fcf9ffe
fix coverage
gconcepcion Sep 22, 2023
d462924
determine yak settings for both parents rather than independently
gconcepcion Sep 25, 2023
af88a08
fix tests and remove some debug comments I missed
gconcepcion Sep 25, 2023
137d6a9
update wdl-ci config file after successful tests
github-actions[bot] Sep 25, 2023
6c02327
update wdl-ci config file after successful tests
github-actions[bot] Sep 25, 2023
49acf5c
Take a stab at estimating depth based on filesize.
williamrowell Sep 28, 2023
f24e1e6
changed flag name and fixed my flipped logic
williamrowell Sep 28, 2023
42d0d84
Adjusted tests.
williamrowell Sep 28, 2023
c8a9b8d
Memory is an int.
williamrowell Sep 28, 2023
4273515
update wdl-ci config file after successful tests
github-actions[bot] Sep 28, 2023
2dc2760
added warning comment about estimating depth by filesize
williamrowell Sep 28, 2023
ebe6afe
Explicitly pass default `yak count` and `hifiasm` params.
williamrowell Sep 29, 2023
eee833f
Merge pull request #10 from PacificBiosciences/wrowell/yak-suggestions
gconcepcion Oct 2, 2023
05312fe
update README to reflect array of reference inputs as well as arrays …
gconcepcion Oct 26, 2023
de41388
Merge pull request #9 from PacificBiosciences/gconcepcion/yak-changes
gconcepcion Oct 26, 2023
8a84fa6
disclaimer and dockstore.yml
gconcepcion Oct 26, 2023
8cf587a
Merge branch 'develop' into feature/print_versions
gconcepcion Oct 26, 2023
894094f
update wdl-ci config file after successful tests
github-actions[bot] Oct 26, 2023
b9f0bca
clean up README
gconcepcion Oct 26, 2023
8853083
Merge pull request #6 from PacificBiosciences/feature/print_versions
williamrowell Oct 27, 2023
461a6f8
update input jsons
gconcepcion Oct 31, 2023
8a90cd3
add -L3.1g for NGx calculation default for human genomes
gconcepcion Nov 1, 2023
deba05b
substitute paftools for htsbox
gconcepcion Nov 3, 2023
8b9a61d
clean up comments
gconcepcion Nov 3, 2023
150b4a4
now that we are aligning haps separately, there is a single assembly …
gconcepcion Nov 7, 2023
58bdf47
add double quotes and fix tests
gconcepcion Nov 15, 2023
de6bf21
update wdl-ci config file after successful tests
github-actions[bot] Nov 15, 2023
cc5afcb
update wdl-ci json
gconcepcion Nov 15, 2023
fe8b97f
update wdl-common
gconcepcion Nov 15, 2023
b5b121e
update tests for new wdl-common
gconcepcion Nov 15, 2023
f2b175f
fix tests
gconcepcion Nov 15, 2023
2471441
update wdl-ci config file after successful tests
github-actions[bot] Nov 15, 2023
c434e7d
attempt to fix tests
gconcepcion Nov 16, 2023
979ab48
update test locaton and remove tests from non-humanassembly tasks
gconcepcion Nov 16, 2023
22c3fc2
update wdl config to reflect local HPC test running
gconcepcion Nov 17, 2023
421fcc7
temporarily disable tests while troubleshooting
gconcepcion Nov 20, 2023
013c77f
fix tests for CoA
gconcepcion Nov 20, 2023
51a53aa
finally fixed?
gconcepcion Nov 20, 2023
d1b2510
Use latest version of wdl-ci
hkeward Nov 21, 2023
a88f0c2
do not save haplotype bams and output trio merged bam
gconcepcion Nov 21, 2023
37b6ffb
update wdl-ci config file after successful tests
github-actions[bot] Nov 21, 2023
24cfc8d
shorten chm13 name
gconcepcion Nov 27, 2023
cdcbc59
update backends documentation
gconcepcion Nov 27, 2023
12c7f42
variant calling for trio asm
gconcepcion Nov 28, 2023
389d8ff
cleanup README
gconcepcion Nov 28, 2023
13ef8d7
paftools print version and minor doc fixes before merge
gconcepcion Nov 28, 2023
b87d85b
...
gconcepcion Nov 28, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -191,13 +191,13 @@ The Docker image used by a particular step of the workflow can be identified by

| Image | Major tool versions | Links |
| :- | :- | :- |
| align_hifiasm | <ul><li>[minimap2 2.17](https://github.com/lh3/minimap2/releases/tag/v2.17)</li><li>[samtools 1.14](https://github.com/samtools/samtools/releases/tag/1.14)</li></ul> | [Dockerfile](https://github.com/PacificBiosciences/wdl-dockerfiles/tree/987efde4d614a292fbfe9f3cf146b63005ad6a8a/docker/align_hifiasm) |
| bcftools | <ul><li>[bcftools 1.14](https://github.com/samtools/bcftools/releases/tag/1.14)</li></ul> | [Dockerfile](https://github.com/PacificBiosciences/wdl-dockerfiles/tree/987efde4d614a292fbfe9f3cf146b63005ad6a8a/docker/bcftools) |
| gfatools | <ul><li>[gfatools 0.4](https://github.com/lh3/gfatools/releases/tag/v0.4)</li><li>[htslib 1.14](https://github.com/samtools/htslib/releases/tag/1.14)</li><li>[k8 0.2.5](https://github.com/attractivechaos/k8/releases/tag/0.2.5)</li><li>[caln50 01091f2](https://github.com/lh3/calN50/tree/01091f25bc24e17fbf0da3407ea24aa448c489ae)</li></ul> | [Dockerfile](https://github.com/PacificBiosciences/wdl-dockerfiles/tree/987efde4d614a292fbfe9f3cf146b63005ad6a8a/docker/gfatools) |
| hifiasm | <ul><li>[hifiasm 0.19.4](https://github.com/chhylp123/hifiasm/releases/tag/0.19.4)</li></ul> | [Dockerfile](https://github.com/PacificBiosciences/wdl-dockerfiles/tree/987efde4d614a292fbfe9f3cf146b63005ad6a8a/docker/hifiasm) |
| htslib | <ul><li>[htslib 1.14](https://github.com/samtools/htslib/releases/tag/1.14)</li></ul> | [Dockerfile](https://github.com/PacificBiosciences/wdl-dockerfiles/tree/987efde4d614a292fbfe9f3cf146b63005ad6a8a/docker/htslib) |
| paftools | <ul><li>[minimap2 2.17](https://github.com/lh3/minimap2/releases/tag/v2.17)</li><li>[samtools 1.14](https://github.com/samtools/samtools/releases/tag/1.14)</li></ul> | [Dockerfile](https://github.com/PacificBiosciences/wdl-dockerfiles/tree/987efde4d614a292fbfe9f3cf146b63005ad6a8a/docker/align_hifiasm) |
| align_hifiasm | <ul><li>[minimap2 2.17](https://github.com/lh3/minimap2/releases/tag/v2.17)</li><li>[samtools 1.14](https://github.com/samtools/samtools/releases/tag/1.14)</li></ul> | [Dockerfile](https://github.com/PacificBiosciences/wdl-dockerfiles/tree/3560fcc5a84e044067cea9c9a7669cfc2659178e/docker/align_hifiasm) |
| bcftools | <ul><li>[bcftools 1.14](https://github.com/samtools/bcftools/releases/tag/1.14)</li></ul> | [Dockerfile](https://github.com/PacificBiosciences/wdl-dockerfiles/tree/3560fcc5a84e044067cea9c9a7669cfc2659178e/docker/bcftools) |
| gfatools | <ul><li>[gfatools 0.4](https://github.com/lh3/gfatools/releases/tag/v0.4)</li><li>[htslib 1.14](https://github.com/samtools/htslib/releases/tag/1.14)</li><li>[k8 0.2.5](https://github.com/attractivechaos/k8/releases/tag/0.2.5)</li><li>[caln50 01091f2](https://github.com/lh3/calN50/tree/01091f25bc24e17fbf0da3407ea24aa448c489ae)</li></ul> | [Dockerfile](https://github.com/PacificBiosciences/wdl-dockerfiles/tree/3560fcc5a84e044067cea9c9a7669cfc2659178e/docker/gfatools) |
| hifiasm | <ul><li>[hifiasm 0.19.4](https://github.com/chhylp123/hifiasm/releases/tag/0.19.4)</li></ul> | [Dockerfile](https://github.com/PacificBiosciences/wdl-dockerfiles/tree/3560fcc5a84e044067cea9c9a7669cfc2659178e/docker/hifiasm) |
| htslib | <ul><li>[htslib 1.14](https://github.com/samtools/htslib/releases/tag/1.14)</li></ul> | [Dockerfile](https://github.com/PacificBiosciences/wdl-dockerfiles/tree/3560fcc5a84e044067cea9c9a7669cfc2659178e/docker/htslib) |
| paftools | <ul<li>[paftools 2.26-r1182-dirty](https://github.com/lh3/minimap2/blob/bc588c0eeb26426d0d90a93fb0877358a389c515/misc/paftools.js)</li></ul> | [Dockerfile](https://github.com/PacificBiosciences/wdl-dockerfiles/tree/3560fcc5a84e044067cea9c9a7669cfc2659178e/docker/align_hifiasm) |
| parse-cohort | <ul><li>python 3.8.10; custom scripts</li></ul> | [Dockerfile](https://github.com/PacificBiosciences/wdl-dockerfiles/tree/
987efde4d614a292fbfe9f3cf146b63005ad6a8a/docker/parse-cohort) |
| samtools | <ul><li>[samtools 1.14](https://github.com/samtools/samtools/releases/tag/1.14)</li></ul> | [Dockerfile](https://github.com/PacificBiosciences/wdl-dockerfiles/tree/987efde4d614a292fbfe9f3cf146b63005ad6a8a/docker/samtools) |
| yak | <ul><li>[yak 0.1](https://github.com/lh3/yak/releases/tag/v0.1)</li></ul> | [Dockerfile](https://github.com/PacificBiosciences/wdl-dockerfiles/tree/987efde4d614a292fbfe9f3cf146b63005ad6a8a/docker/yak) |
3560fcc5a84e044067cea9c9a7669cfc2659178e/docker/parse-cohort) |
gconcepcion marked this conversation as resolved.
Show resolved Hide resolved
| samtools | <ul><li>[samtools 1.14](https://github.com/samtools/samtools/releases/tag/1.14)</li></ul> | [Dockerfile](https://github.com/PacificBiosciences/wdl-dockerfiles/tree/3560fcc5a84e044067cea9c9a7669cfc2659178e/docker/samtools) |
| yak | <ul><li>[yak 0.1](https://github.com/lh3/yak/releases/tag/v0.1)</li></ul> | [Dockerfile](https://github.com/PacificBiosciences/wdl-dockerfiles/tree/3560fcc5a84e044067cea9c9a7669cfc2659178e/docker/yak) |
2 changes: 2 additions & 0 deletions workflows/assemble_genome/assemble_genome.wdl
Original file line number Diff line number Diff line change
Expand Up @@ -342,6 +342,8 @@ task paftools {
command <<<
set -euo pipefail

k8 /opt/minimap2-2.17/misc/paftools.js version

samtools view -h ~{bam} | \
k8 /opt/minimap2-2.17/misc/paftools.js sam2paf - | \
sort -k6,6 -k8,8n | \
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,6 @@ workflow de_novo_assembly_sample {
Array[File] zipped_assembly_fastas = assemble_genome.zipped_assembly_fastas
Array[File] assembly_stats = assemble_genome.assembly_stats
Array[IndexData] merged_bams = assemble_genome.merged_bams
#Array[IndexData] asm_bams = assemble_genome.asm_bams

Array[IndexData] paftools_vcfs = paftools_vcf
Array[File] paftools_vcf_stats = bcftools_stats.stats
Expand Down
4 changes: 2 additions & 2 deletions workflows/main.wdl
Original file line number Diff line number Diff line change
Expand Up @@ -86,8 +86,8 @@ workflow de_novo_assembly {

parameter_meta {
cohort: {help: "Sample information for the cohort"}
reference: {help: "Reference genome data"}
backend: {help: "Backend where the workflow will be executed ['GCP', 'Azure', 'AWS']"}
reference: {help: "Array of Reference genome data"}
gconcepcion marked this conversation as resolved.
Show resolved Hide resolved
backend: {help: "Backend where the workflow will be executed ['GCP', 'Azure', 'AWS', 'HPC']"}
zones: {help: "Zones where compute will take place; required if backend is set to 'AWS' or 'GCP'"}
gconcepcion marked this conversation as resolved.
Show resolved Hide resolved
aws_spot_queue_arn: {help: "Queue ARN for the spot batch queue; required if backend is set to 'AWS'"}
aws_on_demand_queue_arn: {help: "Queue ARN for the on demand batch queue; required if backend is set to 'AWS'"}
Expand Down