sra download seems to be broken now - some of the ftp links lead nowhere

[sergpolly]

@Marlies1993 was running a distiller with some SRA-s as an input and the pipeline kept crashing at the sra step...
After closer inspection it appears that some of the links of this form: https://github.com/mirnylab/distiller-nf/blob/01f6f7bbc4b1edfc3634c131f709b08a40164c74/distiller.nf#L176
are broken ...

for example, take SRR027959 from 2009 hic paper:

venevs@vangogh ➜  ~ wget ftp://ftp-trace.ncbi.nlm.nih.gov/sra/sra-instant/reads/ByRun/sra/SRR/SRR027/SRR027959/SRR027959.sra                 
--2019-11-13 18:12:32--  ftp://ftp-trace.ncbi.nlm.nih.gov/sra/sra-instant/reads/ByRun/sra/SRR/SRR027/SRR027959/SRR027959.sra
           => ‘SRR027959.sra.1’
Resolving ftp-trace.ncbi.nlm.nih.gov (ftp-trace.ncbi.nlm.nih.gov)... 130.14.250.10, 2607:f220:41e:250::11
Connecting to ftp-trace.ncbi.nlm.nih.gov (ftp-trace.ncbi.nlm.nih.gov)|130.14.250.10|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD (1) /sra/sra-instant/reads/ByRun/sra/SRR/SRR027/SRR027959 ... 
No such directory ‘sra/sra-instant/reads/ByRun/sra/SRR/SRR027/SRR027959’.

I don't know enough about sra-s and why are we downloading them using wget - anyone ?

@Marlies1993 can comment and provide other examples if needed

[Phlya]

Had the same problem. Here is what SRA say about it.

[meoomen]

Thanks @Phlya! When I checked the presence of the sra links one by one, I also found that some of them were missing from ftp://ftp-trace.ncbi.nlm.nih.gov/sra/sra-instant/reads/ByRun/sra/SRR. When I removed the sra's that weren't present from my project.yml, my distiller project is running fine, but this doesn't solve the problem of course...

[Phlya]

I just downloaded those missing ones manually... Wget download is much faster than the regular sra tools, but maybe in case of this problem distiller should fall back to fastq-dump?

[meoomen]

Yes, I figured I will have to do manual download for now as well. Thanks!

[golobor]

2f259f5 this update will make distiller use fastq-dump if wget fails. Does it look good? Any other fixes we could implement to the downloading process (i.e. try multiple URLs), while we're at it?

…

On Thu, 14 Nov 2019 at 00:58, Marlies Oomen ***@***.***> wrote: Yes, I figured I will have to do manual download for now as well. Thanks! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#145>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAG64CUCJCV6RLSAGD252QLQTSID5ANCNFSM4JNDFOJQ> .

[golobor]

Btw, here is another trick to force using fastq-dump, in project.yml specify input as:
library1:
lane1:
- sra:SRR0123456?start=1

[sergpolly]

Thank you @golobor and @Phlya ! that was uber quick!

Does it look good?

we are about to try it - we'll let you know here

Any other fixes we could implement to the downloading process (i.e. try multiple URLs), while we're at it?

hmmmm - I do it so rarely that I don't really know what to say ... maybe @Phlya have suggestions ? MirnyLab people ?
If anything i would like a reminder why aren't we doing it the nextflow way ? https://www.nextflow.io/docs/edge/channel.html#fromsra https://www.nextflow.io/blog/2019/release-19.03.0-edge.html - is it because we didn't have time to do it - or because there is something wrong with it ?

[golobor]

maybe it's worth switching, anyone is interested in implementing? :) i think, there was a time where it was returning non-gzipped files, but there has been much improvement lately, so we should probably consider switching.

…

On Thu, 14 Nov 2019 at 16:06, Sergey Venev ***@***.***> wrote: Thank you @golobor <https://github.com/golobor> and @Phlya <https://github.com/Phlya> ! that was uber quick! Does it look good? we are about to try it - we'll let you know here Any other fixes we could implement to the downloading process (i.e. try multiple URLs), while we're at it? hmmmm - I do it so rarely that I don't really know what to say ... maybe @Phlya <https://github.com/Phlya> have suggestions ? MirnyLab people ? If anything i would like a reminder why aren't we doing it the nextflow way ? https://www.nextflow.io/docs/edge/channel.html#fromsra https://www.nextflow.io/blog/2019/release-19.03.0-edge.html - is it because we didn't have time to do it - or because there is something wrong with it ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#145>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAG64CTIJKHXJFY3NY4OMGDQTVSORANCNFSM4JNDFOJQ> .

[sergpolly]

worked for me on the original 2009 Hi-C data, - I guess @Marlies1993 would report here once she tries it as well:

Thank you, again!

maybe it's worth switching, anyone is interested in implementing? :)

sounds fun to me - should simplify some of the distiller.nf code - not sure about timeline requirements though ...

[meoomen]

Thanks for fixing this so quickly! All my sra's are downloading and mapping as well.

[golobor]

a more reliable fix: 55b5e6e