Next Release

.gitmodules

@@ -1,3 +1,3 @@
 [submodule "workflows/wdl-common"]
 	path = workflows/wdl-common
-	url = https://github.com/PacificBiosciences/wdl-common
+	url = ../wdl-common.git

LICENSE

@@ -1,4 +1,4 @@
-Copyright (c) 2023, Pacific Biosciences of California, Inc.
+Copyright (c) 2024, Pacific Biosciences of California, Inc.
 
 All rights reserved.
 

README.md

@@ -115,7 +115,7 @@ A cohort can include one or more samples. Samples need not be related, but if yo
 
 | Type | Name | Description | Notes |
 | :- | :- | :- | :- |
-| String | cohort_id | A unique name for the cohort; used to name outputs | |
+| String | cohort_id | A unique name for the cohort; used to name outputs | Alphanumeric characters, underscore (`_`), and dash (`-`) are allowed. |
 | Array[[Sample](#sample)] | samples | The set of samples for the cohort. At least one sample must be defined. | |
 | Array[String] | phenotypes | [Human Phenotype Ontology (HPO) phenotypes](https://hpo.jax.org/app/) associated with the cohort. If no particular phenotypes are desired, the root HPO term, `"HP:0000001"`, can be used. | |
 
@@ -125,12 +125,12 @@ Sample information for each sample in the workflow run.
 
 | Type | Name | Description | Notes |
 | :- | :- | :- | :- |
-| String | sample_id | A unique name for the sample; used to name outputs | |
+| String | sample_id | A unique name for the sample; used to name outputs | Alphanumeric characters, underscore (`_`), and dash (`-`) are allowed. |
 | Array[[IndexData](https://github.com/PacificBiosciences/wdl-common/blob/main/wdl/structs.wdl)] | movie_bams | The set of unaligned movie BAMs associated with this sample | |
 | String? | sex | Sample sex | ["MALE", "FEMALE", `null`]. If the sex field is missing or `null`, sex will be set to unknown. Used to set the expected sex chromosome karyotype for TRGT and HiFiCNV. |
 | Boolean | affected | Is this sample affected by the phenotype? | \[`true`, `false`\] |
-| String? | father_id | Paternal `sample_id` | |
-| String? | mother_id | Maternal `sample_id` | |
+| String? | father_id | Paternal `sample_id` | Alphanumeric characters, underscore (`_`), and dash (`-`) are allowed. |
+| String? | mother_id | Maternal `sample_id` | Alphanumeric characters, underscore (`_`), and dash (`-`) are allowed. |
 
 ## [ReferenceData](workflows/humanwgs_structs.wdl)
 
@@ -172,8 +172,8 @@ These files are hosted publicly in each of the cloud backends; see `backends/${b
 
 | Type | Name | Description | Notes |
 | :- | :- | :- | :- |
-| String? | deepvariant_version | Version of deepvariant to use \["1.5.0"\] | |
-| [DeepVariantModel](https://github.com/PacificBiosciences/wdl-common/blob/main/wdl/structs.wdl)? | deepvariant_model | Optional alternate DeepVariant model file to use | |
+| String? | deepvariant_version | Version of deepvariant to use \["1.6.0"\] | This release is only compatible with DeepVariant [1.6](https://github.com/google/deepvariant/releases/tag/v1.6.0). |
+ | File? | custom_deepvariant_model_tar | Optional alternate DeepVariant [custom model](https://github.com/PacificBiosciences/wdl-common/blob/8d868818b62345fdb64ab104238d406d65334b3e/wdl/workflows/deepvariant/README.md) to use | |
 | Int? | pbsv_call_mem_gb | Optionally set RAM (GB) for pbsv_call during cohort analysis | |
 | Int? | glnexus_mem_gb | Optionally set RAM (GB) for GLnexus during cohort analysis | |
 | Boolean? | run_tertiary_analysis | Run the optional tertiary analysis steps \[`false`\] | \[`true`, `false`\] |
@@ -214,7 +214,7 @@ These files will be output for each sample defined in the cohort.
 | Array[Array[File]] | cpg_pileup_bigwigs | 5mCpG site methylation probability pileups generated by pb-CpG-tools | |
 | Array[File] | paraphase_output | Output generated by [Paraphase](https://github.com/PacificBiosciences/paraphase) | |
 | Array[[IndexData](https://github.com/PacificBiosciences/wdl-common/blob/main/wdl/structs.wdl)] | paraphase_realigned_bam | Realigned BAM for selected medically relevant genes in segmental duplications (with index), generated by Paraphase | |
-| Array[Array[File]] | paraphase_vcfs | Phased Variant calls for selected medically relevant genes in segmental duplications, generated by Paraphase | |
+| Array[File] | paraphase_vcfs | Tarball of phased variant calls for selected medically relevant genes in segmental duplications, generated by Paraphase | |
 | Array[[IndexData](https://github.com/PacificBiosciences/wdl-common/blob/main/wdl/structs.wdl)] | hificnv_vcfs | VCF output containing copy number variant calls for the sample from [HiFiCNV](https://github.com/PacificBiosciences/HiFiCNV) | |
 | Array[File] | hificnv_copynum_bedgraphs | Copy number values calculated for each region | |
 | Array[File] | hificnv_depth_bws | Bigwig file containing the depth measurements from HiFiCNV | |
@@ -256,21 +256,21 @@ The Docker image used by a particular step of the workflow can be identified by
 | Image | Major tool versions | Links |
 | :- | :- | :- |
 | bcftools | <ul><li>[bcftools 1.14](https://github.com/samtools/bcftools/releases/tag/1.14)</li></ul> | [Dockerfile](https://github.com/PacificBiosciences/wdl-dockerfiles/tree/3560fcc5a84e044067cea9c9a7669cfc2659178e/docker/bcftools) |
-| deepvariant | User-defined; default is version [1.5.0](https://github.com/google/deepvariant/releases/tag/v1.5.0) | [DeepVariant GitHub](https://github.com/google/deepvariant) |
+| deepvariant | User-defined; default is version [1.6.0](https://github.com/google/deepvariant/releases/tag/v1.6.0) | [DeepVariant GitHub](https://github.com/google/deepvariant) |
 | glnexus | <ul><li>[glnexus v1.4.3](https://github.com/dnanexus-rnd/GLnexus/releases/tag/v1.4.3)</li></ul> | [GLnexus GitHub](https://github.com/dnanexus-rnd/GLnexus) |
 | hificnv | <ul><li>[HiFiCNV v0.1.7](https://github.com/PacificBiosciences/HiFiCNV/releases/tag/v0.1.7)</li><li>[bcftools 1.16](https://github.com/samtools/bcftools/releases/tag/1.16)</li></ul> | [Dockerfile](https://github.com/PacificBiosciences/wdl-dockerfiles/tree/0b0fbe939648087e9fdea4497ae08dc76538ebf0/docker/hificnv) |
 | hiphase | <ul><li>[HiPhase 1.0.0](https://github.com/PacificBiosciences/HiPhase/releases/tag/v1.0.0)</li><li>[samtools 1.18](https://github.com/samtools/samtools/releases/tag/1.18)</li><li>[bcftools 1.18](https://github.com/samtools/bcftools/releases/tag/1.18)</li></ul> | [Dockerfile](https://github.com/PacificBiosciences/wdl-dockerfiles/tree/d26db6204409dfeff56e169cdba0cc14bc272f15/docker/hiphase) |
 | htslib | <ul><li>[htslib 1.14](https://github.com/samtools/htslib/releases/tag/1.14)</li></ul> | [Dockerfile](https://github.com/PacificBiosciences/wdl-dockerfiles/tree/3560fcc5a84e044067cea9c9a7669cfc2659178e/docker/htslib) |
 | mosdepth | <ul><li>[mosdepth 0.2.9](https://github.com/brentp/mosdepth/releases/tag/v0.2.9)</li></ul> | [Dockerfile](https://github.com/PacificBiosciences/wdl-dockerfiles/tree/3560fcc5a84e044067cea9c9a7669cfc2659178e/docker/mosdepth) |
-| paraphase | <ul><li>[minimap2 2.17](https://github.com/lh3/minimap2/releases/tag/v2.17)</li><li>[samtools 1.14](https://github.com/samtools/samtools/releases/tag/1.14)</li><li>[paraphase 2.2.3](https://github.com/PacificBiosciences/paraphase/releases/tag/v2.2.3)</li></ul> | [Dockerfile](https://github.com/PacificBiosciences/wdl-dockerfiles/tree/3560fcc5a84e044067cea9c9a7669cfc2659178e/docker/paraphase) |
+| paraphase | <ul><li>[minimap2 2.26](https://github.com/lh3/minimap2/releases/tag/v2.26)</li><li>[samtools 1.18](https://github.com/samtools/samtools/releases/tag/1.18)</li><li>[paraphase 3.0.0](https://github.com/PacificBiosciences/paraphase)</li></ul> | [Dockerfile](https://github.com/PacificBiosciences/wdl-dockerfiles/tree/4f510e5f434cc138577853f56558b90e059fd770/docker/paraphase) |
 | pb-cpg-tools | <ul><li>[pb-CpG-tools v2.3.2](https://github.com/PacificBiosciences/pb-CpG-tools/releases/tag/v2.3.2)</li></ul> | [Dockerfile](https://github.com/PacificBiosciences/wdl-dockerfiles/tree/7481837d3b0f539adf4f64209a65cf28eebf3dba/docker/pb-cpg-tools) |
-| pbmm2 | <ul><li>[pbmm2 1.10.0](https://github.com/PacificBiosciences/pbmm2/releases/tag/v1.10.0)</li><li>[datamash 1.1.0](https://ftp.gnu.org/gnu/datamash/)</li><li>[pysam 0.16.0.1](https://github.com/pysam-developers/pysam/releases/tag/v0.16.0.1)</li></ul> | [Dockerfile](https://github.com/PacificBiosciences/wdl-dockerfiles/tree/3560fcc5a84e044067cea9c9a7669cfc2659178e/docker/pbmm2) |
+| pbmm2 | <ul><li>[pbmm2 1.13.1](https://github.com/PacificBiosciences/pbmm2/releases/tag/v1.13.1)</li><li>[datamash 1.1.0](https://ftp.gnu.org/gnu/datamash/)</li><li>[pysam 0.16.0.1](https://github.com/pysam-developers/pysam/releases/tag/v0.16.0.1)</li></ul> | [Dockerfile](https://github.com/PacificBiosciences/wdl-dockerfiles/tree/6d3d818dce6f97454ac29f4ed96637e450a19f63/docker/pbmm2) |
 | pbsv | <ul><li>[pbsv 2.9.0](https://github.com/PacificBiosciences/pbsv/releases/tag/v2.9.0)</li><li>[htslib 1.14](https://github.com/samtools/htslib/releases/tag/1.14)</li></ul> | [Dockerfile](https://github.com/PacificBiosciences/wdl-dockerfiles/tree/f9e33a757e6d8cb15696ac930a2efd0fd7a885d8/docker/pbsv) |
 | pyyaml | <ul><li>[pyyaml 5.3.1](https://github.com/yaml/pyyaml/releases/tag/5.3.1)</li></ul> | [Dockerfile](https://github.com/PacificBiosciences/wdl-dockerfiles/tree/f72e862bca2f209b9909e6043ef0197975762f27/docker/pyyaml) |
 | samtools | <ul><li>[samtools 1.14](https://github.com/samtools/samtools/releases/tag/1.14)</li></ul> | [Dockerfile](https://github.com/PacificBiosciences/wdl-dockerfiles/tree/3560fcc5a84e044067cea9c9a7669cfc2659178e/docker/samtools) |
 | slivar | <ul><li>[slivar 0.2.2](https://github.com/brentp/slivar/releases/tag/v0.2.2)</li><li>[bcftools 1.14](https://github.com/samtools/bcftools/releases/tag/1.14)</li><li>[vcfpy 0.13.3](https://github.com/bihealth/vcfpy/releases/tag/v0.13.3)</li><li>[pysam 0.19.1](https://github.com/pysam-developers/pysam/releases/tag/v0.19.1)</li></ul> | [Dockerfile](https://github.com/PacificBiosciences/wdl-dockerfiles/tree/3560fcc5a84e044067cea9c9a7669cfc2659178e/docker/slivar) |
 | svpack | <ul><li>[svpack 36180ae6](https://github.com/PacificBiosciences/svpack/tree/a82598ebc4013bf32e70295b83b380ada6302c4a)</li><li>[htslib 1.18](https://github.com/samtools/htslib/releases/tag/1.18)</li><li>[pysam 0.21.0](https://github.com/pysam-developers/pysam/releases/tag/v0.21.0)</li> | [Dockerfile](https://github.com/PacificBiosciences/wdl-dockerfiles/tree/8edbc516abc0ff43ac279b48018003923721b054/docker/svpack) |
-| trgt | <ul><li>[trgt 0.5.0](https://github.com/PacificBiosciences/trgt/releases/tag/v0.5.0)</li><li>[samtools 1.18](https://github.com/samtools/samtools/releases/tag/1.18)</li><li>[bcftools 1.18](https://github.com/samtools/bcftools/releases/tag/1.18)</li><li>[pysam 0.21.0](https://github.com/pysam-developers/pysam/releases/tag/v0.21.0)</li></ul> | [Dockerfile](https://github.com/PacificBiosciences/wdl-dockerfiles/tree/d2a45e0213ac3fa631a51a48757c442d3ed550b6/docker/trgt) |
+| trgt | <ul><li>[trgt 0.8.0](https://github.com/PacificBiosciences/trgt/releases/tag/v0.7.0)</li><li>[samtools 1.18](https://github.com/samtools/samtools/releases/tag/1.18)</li><li>[bcftools 1.18](https://github.com/samtools/bcftools/releases/tag/1.18)</li><li>[pysam 0.21.0](https://github.com/pysam-developers/pysam/releases/tag/v0.21.0)</li></ul> | [Dockerfile](https://github.com/PacificBiosciences/wdl-dockerfiles/tree/3bbc033f8f942b10a304b4fa907957a789c73ef7/docker/trgt) |
 
 ---
 

backends/aws/inputs.aws.json

@@ -64,7 +64,7 @@
     "clinvar_lookup": "s3://dnastack-resources/dataset/slivar/clinvar_gene_desc.20221214T183140.txt"
   },
   "humanwgs.deepvariant_version": "String (optional)",
-  "humanwgs.deepvariant_model": "WomCompositeType {\n model -> WomCompositeType {\n data -> File\ndata_index -> File \n}\nmetadata -> File \n}? (optional)",
+  "humanwgs.custom_deepvariant_model_tar": "File (optional)",
   "humanwgs.pbsv_call_mem_gb": "Int (optional, default = if N<=3: 64 else 96)",
   "humanwgs.glnexus_mem_gb": "Int (optional, default = 30)",
   "humanwgs.run_tertiary_analysis": "Boolean (optional, default = false)",

backends/azure/inputs.azure.json

@@ -64,7 +64,7 @@
     "clinvar_lookup": "/datasetpbrarediseases/dataset/slivar/clinvar_gene_desc.20221214T183140.txt"
   },
   "humanwgs.deepvariant_version": "String (optional)",
-  "humanwgs.deepvariant_model": "WomCompositeType {\n model -> WomCompositeType {\n data -> File\ndata_index -> File \n}\nmetadata -> File \n}? (optional)",
+  "humanwgs.custom_deepvariant_model_tar": "File (optional)",
   "humanwgs.pbsv_call_mem_gb": "Int (optional, default = if N<=3: 64 else 96)",
   "humanwgs.glnexus_mem_gb": "Int (optional, default = 30)",
   "humanwgs.run_tertiary_analysis": "Boolean (optional, default = false)",

backends/gcp/inputs.gcp.json

@@ -64,7 +64,7 @@
     "clinvar_lookup": "gs://pacbio-wdl/dataset/slivar/clinvar_gene_desc.20221214T183140.txt"
   },
   "humanwgs.deepvariant_version": "String (optional)",
-  "humanwgs.deepvariant_model": "WomCompositeType {\n model -> WomCompositeType {\n data -> File\ndata_index -> File \n}\nmetadata -> File \n}? (optional)",
+  "humanwgs.custom_deepvariant_model_tar": "File (optional)",
   "humanwgs.pbsv_call_mem_gb": "Int (optional, default = if N<=3: 64 else 96)",
   "humanwgs.glnexus_mem_gb": "Int (optional, default = 30)",
   "humanwgs.run_tertiary_analysis": "Boolean (optional, default = false)",

backends/hpc/README.md

@@ -41,7 +41,7 @@ Reference data is hosted on Zenodo at [10.5281/zenodo.8415406](https://zenodo.or
 
 ```bash
 # download the reference data bundle
-wget https://zenodo.org/record/8415406/files/wdl-humanwgs.v1.0.2.resources.tgz
+wget https://zenodo.org/records/8415406/files/wdl-humanwgs.v1.0.2.resource.tgz
 
 # extract the reference data bundle and rename as dataset
 tar -xzf wdl-humanwgs.v1.0.2.resources.tgz && mv static_resources dataset

backends/hpc/inputs.hpc.json

@@ -64,7 +64,7 @@
     "clinvar_lookup": "<local_path_prefix>/dataset/slivar/clinvar_gene_desc.20221214T183140.txt"
   },
   "humanwgs.deepvariant_version": "String (optional)",
-  "humanwgs.deepvariant_model": "WomCompositeType {\n model -> WomCompositeType {\n data -> File\ndata_index -> File \n}\nmetadata -> File \n}? (optional)",
+  "humanwgs.custom_deepvariant_model_tar": "File (optional)",
   "humanwgs.pbsv_call_mem_gb": "Int (optional, default = if N<=3: 64 else 96)",
   "humanwgs.glnexus_mem_gb": "Int (optional, default = 30)",
   "humanwgs.run_tertiary_analysis": "Boolean (optional, default = false)",

wdl-ci.config.json

@@ -25,7 +25,7 @@
       "tasks": {
         "pbmm2_align": {
           "key": "pbmm2_align",
-          "digest": "3r4icze5zkps7m6xoruzvnfzk2fp4gqd",
+          "digest": "ntqgonovwrgxezaewjfw7oec4t34yvhp",
           "tests": [
             {
               "inputs": {
@@ -142,7 +142,7 @@
         },
         "trgt": {
           "key": "trgt",
-          "digest": "ylzep5nroxhzjff43gkc6fs25ydor7dd",
+          "digest": "fyt3gqmt5tfykbls33kbr62nw3d4rvhj",
           "tests": [
             {
               "inputs": {
@@ -178,7 +178,7 @@
         },
         "coverage_dropouts": {
           "key": "coverage_dropouts",
-          "digest": "3el45hg36hlyx5cswr3dkvqfg644cvbn",
+          "digest": "iac5c3nzugilbarb4xsat37n2j5hjov4",
           "tests": [
             {
               "inputs": {
@@ -245,7 +245,7 @@
         },
         "paraphase": {
           "key": "paraphase",
-          "digest": "gzktyxvdrw73el5khnudlpu23x34lbxv",
+          "digest": "xme5pugrmhvnys5vas6jfagm65eac4hz",
           "tests": [
             {
               "inputs": {
@@ -570,7 +570,7 @@
       "tasks": {
         "deepvariant_make_examples": {
           "key": "deepvariant_make_examples",
-          "digest": "35kzpf37semcoxs7frzvhjrc4zvwoyan",
+          "digest": "3jfeho4suf23enopqj23cx6ygviftqny",
           "tests": [
             {
               "inputs": {
@@ -598,7 +598,7 @@
                 "task_start_index": 0,
                 "tasks_per_shard": 8,
                 "total_deepvariant_tasks": 64,
-                "deepvariant_version": "1.5.0",
+                "deepvariant_version": "1.6.0",
                 "runtime_attributes": "${default_runtime_attributes}"
               },
               "output_tests": {
@@ -620,7 +620,7 @@
         },
         "deepvariant_call_variants": {
           "key": "deepvariant_call_variants",
-          "digest": "a6ksi3haiz5pye7p64c67zeeauit7gqf",
+          "digest": "rywffaewvhwakdysxejsrqbz7bqwzt2o",
           "tests": [
             {
               "inputs": {
@@ -637,12 +637,12 @@
                   "${resources_file_path}/deepvariant/${sample_id}.56.example_tfrecords.tar.gz"
                 ],
                 "total_deepvariant_tasks": 64,
-                "deepvariant_version": "1.5.0",
+                "deepvariant_version": "1.6.0",
                 "runtime_attributes": "${default_runtime_attributes}"
               },
               "output_tests": {
-                "tfrecord": {
-                  "value": "${resources_file_path}/deepvariant/${sample_id}.GRCh38.call_variants_output.tfrecord.gz",
+                "tfrecords_tar": {
+                  "value": "${resources_file_path}/deepvariant/${sample_id}.GRCh38.call_variants_output.tar.gz",
                   "test_tasks": [
                     "compare_file_basename",
                     "check_gzip"
@@ -654,12 +654,12 @@
         },
         "deepvariant_postprocess_variants": {
           "key": "deepvariant_postprocess_variants",
-          "digest": "afmkoy2hy4lcyolms7n6lgzuzudr7uxx",
+          "digest": "ey7zqpajaeesvsg372rehhjmkpqld2qx",
           "tests": [
             {
               "inputs": {
                 "sample_id": "${sample_id}",
-                "tfrecord": "${resources_file_path}/deepvariant/${sample_id}.GRCh38.call_variants_output.tfrecord.gz",
+                "tfrecords_tar": "${resources_file_path}/deepvariant/${sample_id}.GRCh38.call_variants_output.tar.gz",
                 "nonvariant_site_tfrecord_tars": [
                   "${resources_file_path}/deepvariant/${sample_id}.0.nonvariant_site_tfrecords.tar.gz",
                   "${resources_file_path}/deepvariant/${sample_id}.8.nonvariant_site_tfrecords.tar.gz",
@@ -674,7 +674,7 @@
                 "reference_index": "${datasets_file_path}/GRCh38/human_GRCh38_no_alt_analysis_set.fasta.fai",
                 "reference_name": "GRCh38",
                 "total_deepvariant_tasks": 64,
-                "deepvariant_version": "1.5.0",
+                "deepvariant_version": "1.6.0",
                 "runtime_attributes": "${default_runtime_attributes}"
               },
               "output_tests": {
@@ -1114,8 +1114,13 @@
   "engines": {
     "c3213beb-97bc-4adc-9bd8-cdb3ac83b398": {
       "key": "c3213beb-97bc-4adc-9bd8-cdb3ac83b398",
-      "enabled": true,
+      "enabled": false,
       "name": "PacBio CoA"
+    },
+    "abc123": {
+      "key": "abc123",
+      "enabled": true,
+      "name": "pacbio-hpc"
     }
   },
   "test_params": {
@@ -1142,6 +1147,11 @@
         "input_file_path": "/coac74908838b5dd7/inputs/small_dataset/chr6.p23",
         "resources_file_path": "/coac74908838b5dd7/inputs/wdl-ci/humanwgs",
         "datasets_file_path": "/datasetpbrarediseases/dataset"
+      },
+      "abc123": {
+        "input_file_path": "/pbi/collections/appslabht/cromwell_output/testdata/inputs/chr6.p23",
+        "resources_file_path": "/pbi/collections/appslabht/cromwell_output/testdata/wdl-ci/humanwgs",
+        "datasets_file_path": "/pbi/collections/appslabht/cromwell_output/testdata/datasetpbrarediseases/dataset"
       }
     }
   }

workflows/input_template.json

@@ -42,7 +42,7 @@
     "clinvar_lookup": "File"
   },
   "humanwgs.deepvariant_version": "String? (optional)",
-  "humanwgs.deepvariant_model": "WomCompositeType {\n model -> WomCompositeType {\n data -> File\ndata_index -> File \n}\nmetadata -> File \n}? (optional)",
+  "humanwgs.custom_deepvariant_model_tar": "File (optional)",
   "humanwgs.pbsv_call_mem_gb": "Int? (optional)",
   "humanwgs.glnexus_mem_gb": "Int? (optional)",
   "humanwgs.run_tertiary_analysis": "Boolean? (optional, default = false)",