Current release doesn't work with DeepVariant 1.6.0

[s-batalov]

Describe the bug
I understand that current default version of DeepVariant in the suggested input JSON 1.5.0.
I have replaced it with 1.6.0 because it has now been available for a while.
HiFi-human-WGS-WDL with 1.6.0 now hangs (idles, doing nothing) without quitting in the deepvariant.deepvariant_call_variants phase.
Last line in log:
BackgroundConfigAsyncJobExecutionActor [6784f40cdeepvariant.deepvariant_call_variants:NA:1]: Status change from - to WaitingForReturnCode
Last lines in subtask tree subfolder stderr
tensorflow.python.framework.errors_impl.NotFoundError: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for /opt/models/pacbio/model.ckpt

Details

• Release Version = 1.1.0
• Backend = Cromwell on HPC with Docker actors
• Inputs = singleton sequenced to 30x on Revio
• Failing task(s) = deepvariant.deepvariant_call_variants

Expected behavior
Expected DeepVariant 1.6.0 to be a drop-in replacement by configuring "1.6.0" in the input JSON.

Additional context
Novice user. Used all default parameters as given, only with 1.6.0 for DeepVariant.

[s-batalov]

After replacing 1.6.0 back to 1.5.0 - the workflow appear to work fine. (the result aligned.BAM of course is the same; "samples" for tensorflow are ~ the same, and all stages finished normally

[williamrowell]

Hi Sergey,

We break up DeepVariant into three different tasks to balance cost/time and use resources efficiently. While the run_deepvariant.py command typically doesn't change much from version to version, the parameters for "internals" like make_examples, call_variants and postprocess_variants can change a lot from version to version. There were a few significant changes from DeepVariant v1.5.0 to v1.6.0, including how model files are stored. Unfortunately, that means it's more complicated than just changing the version number.

Our develop branch (#103) has been updated to use DeepVariant v1.6.0, but we aren't ready to merge this into main and cut a new release yet (so use at your own risk). In the documentation for the next release, we'll make it clearer that we typically only support one version of DeepVariant per release, and I'll likely remove the optional deepvariant_version and custom model inputs (they'll still be implemented, just not in the templates or examples) to avoid confusion. These inputs are really useful if I want to test a new unreleased model, but confusing to end users.

Billy

[williamrowell]

Addressing this with better documentation in the next release.

[s-batalov]

Will current release work down-compatible? (e.g. if version is set to "1.4.0"?

• I intend to run HG002-rep1 (and maybe all five exemplar datasets) from PBCLOUD store to see identical results, for validation)

[williamrowell]

There have been API differences for each of the last few releases that make it difficult to maintain backwards compatibility. The next release will only support DeepVariant v1.6.0.

[s-batalov]

Appreciated. I will match the complete WDL version to attempt to reproduce HG002-rep1 result-set.
Thank you.