Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error during install - dbsnp issue #3699

Open
jchien-ucd opened this issue Feb 15, 2023 · 15 comments
Open

Error during install - dbsnp issue #3699

jchien-ucd opened this issue Feb 15, 2023 · 15 comments

Comments

@jchien-ucd
Copy link

Running GGD recipe: hg38 dbsnp 154-20210112
URL transformed to HTTPS due to an HSTS policy
--2023-02-14 16:01:32-- https://ftp.ncbi.nih.gov/snp/archive/b154/VCF/GCF_000001405.38.gz
Resolving ftp.ncbi.nih.gov (ftp.ncbi.nih.gov)... 130.14.250.12, 165.112.9.228, 2607:f220:41e:250::7, ...
Connecting to ftp.ncbi.nih.gov (ftp.ncbi.nih.gov)|130.14.250.12|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-02-14 16:01:33 ERROR 404: Not Found.

Traceback (most recent call last):
File "/home/jchien/Apps/bcbio/tools/bin/bcbio_nextgen.py", line 228, in
install.upgrade_bcbio(kwargs["args"])
File "/home/jchien/Apps/bcbio/anaconda/lib/python3.7/site-packages/bcbio/install.py", line 109, in upgrade_bcbio
upgrade_bcbio_data(args, REMOTES)
File "/home/jchien/Apps/bcbio/anaconda/lib/python3.7/site-packages/bcbio/install.py", line 361, in upgrade_bcbio_data
args.cores, ["ggd", "s3", "raw"])
File "/home/jchien/Apps/bcbio/anaconda/envs/python2/bin/tmpbcbio-install/cloudbiolinux/cloudbio/biodata/genomes.py", line 354, in install_data_local
_prep_genomes(env, genomes, genome_indexes, ready_approaches, data_filedir)
File "/home/jchien/Apps/bcbio/anaconda/envs/python2/bin/tmpbcbio-install/cloudbiolinux/cloudbio/biodata/genomes.py", line 480, in _prep_genomes
retrieve_fn(env, manager, gid, idx)
File "/home/jchien/Apps/bcbio/anaconda/envs/python2/bin/tmpbcbio-install/cloudbiolinux/cloudbio/biodata/genomes.py", line 875, in _install_with_ggd
ggd.install_recipe(os.getcwd(), env.system_install, recipe_file, gid)
File "/home/jchien/Apps/bcbio/anaconda/envs/python2/bin/tmpbcbio-install/cloudbiolinux/cloudbio/biodata/ggd.py", line 30, in install_recipe
recipe["recipe"]["full"]["recipe_type"], system_install)
File "/home/jchien/Apps/bcbio/anaconda/envs/python2/bin/tmpbcbio-install/cloudbiolinux/cloudbio/biodata/ggd.py", line 62, in _run_recipe
subprocess.check_output(["bash", run_file])
File "/home/jchien/Apps/bcbio/anaconda/lib/python3.7/subprocess.py", line 411, in check_output
**kwargs).stdout
File "/home/jchien/Apps/bcbio/anaconda/lib/python3.7/subprocess.py", line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['bash', '/home/jchien/Apps/bcbio/genomes/Hsapiens/hg38/txtmp/ggd-run.sh']' returned non-zero exit status 8.
@wangpenhok
Copy link

I met this problem, too. The reason is that version 154 is not available from 'ftp.ncbi.nih.gov' anymore. Instead, they offer 156 as the latest version. so you can edit the GGD recipe manually by updating the URL and version.

@jchien-ucd
Copy link
Author

jchien-ucd commented Feb 15, 2023 via email

@jchien-ucd
Copy link
Author

jchien-ucd commented Feb 15, 2023 via email

@lindakohn
Copy link

I encountered this problem too, located and edited all 154 entries to 156 in <path_to-bcbio>/tmpbcbio-install/cloudbiolinux/ggd-recipes//dbsnp.yaml

@naumenko-sa naumenko-sa changed the title Error during install Error during install - dbsnp issue Mar 20, 2023
@naumenko-sa
Copy link
Contributor

thanks for figuring out!
I've updated the recipe:
https://github.com/chapmanb/cloudbiolinux/blob/master/ggd-recipes/hg38/dbsnp.yaml

@lindakohn
Copy link

Thanks Serhiy!

I believe that also https://github.com/chapmanb/cloudbiolinux/blob/master/ggd-recipes/hg19/dbsnp.yaml needs to be changed accordingly

/Linda

@naumenko-sa
Copy link
Contributor

Thanks, Linda!
I've updated the hg19 recipe!
SN

@wangpenhok
Copy link

Thanks Serhiy!

I think the gnomad genome recipe should be updated according to the new field names of gnomad files as well

@naumenko-sa
Copy link
Contributor

Hi Peter @wangpenhok !

I see 3.1 vcf sites are still there?
https://console.cloud.google.com/storage/browser/gcp-public-data--gnomad/release/3.1/vcf/genomes

What is the issue you are encountering?

SN

@wangpenhok
Copy link

wangpenhok commented Mar 20, 2023 •
edited
Loading

Hi Serhiy@naumenko-sa

The vcf sites are okay, but the file downloaded from $gnomad_fields_to_keep_url may be outdated.

As I had mentioned here, [https://github.com/chapmanb/cloudbiolinux/pull/400#issuecomment-1475735769],

The majority of field names of the latest vcf files are not compatible with names in the file gnomad_fields_to_keep,
Here are some headers from the latest vcfs:
##INFO=<ID=nhomalt-sas-XY,Number=A,Type=Integer,Description="Count of homozygous individuals in XY samples of South Asian ancestry"> ##INFO=<ID=AC-fin-XX,Number=A,Type=Integer,Description="Alternate allele count for XX samples of Finnish ancestry"> ##INFO=<ID=AN-fin-XX,Number=1,Type=Integer,Description="Total number of alleles in XX samples of Finnish ancestry"> ##INFO=<ID=AF-fin-XX,Number=A,Type=Float,Description="Alternate allele frequency in XX samples of Finnish ancestry"> ##INFO=<ID=nhomalt-fin-XX,Number=A,Type=Integer,Description="Count of homozygous individuals in XX samples of Finnish ancestry"> ##INFO=<ID=AC-nfe-XX,Number=A,Type=Integer,Description="Alternate allele count for XX samples of Non-Finnish European ancestry"> ##INFO=<ID=AN-nfe-XX,Number=1,Type=Integer,Description="Total number of alleles in XX samples of Non-Finnish European ancestry"> ##INFO=<ID=AF-nfe-XX,Number=A,Type=Float,Description="Alternate allele frequency in XX samples of Non-Finnish European ancestry"> ##INFO=<ID=nhomalt-nfe-XX,Number=A,Type=Integer,Description="Count of homozygous individuals in XX samples of Non-Finnish European ancestry"> ##INFO=<ID=AC-sas,Number=A,Type=Integer,Description="Alternate allele count for samples of South Asian ancestry"> ##INFO=<ID=AN-sas,Number=1,Type=Integer,Description="Total number of alleles in samples of South Asian ancestry"> ##INFO=<ID=AF-sas,Number=A,Type=Float,Description="Alternate allele frequency in samples of South Asian ancestry"> ##INFO=<ID=nhomalt-sas,Number=A,Type=Integer,Description="Count of homozygous individuals in samples of South Asian ancestry"> ##INFO=<ID=AC-oth-XX,Number=A,Type=Integer,Description="Alternate allele count for XX samples of Other ancestry"> ##INFO=<ID=AN-oth-XX,Number=1,Type=Integer,Description="Total number of alleles in XX samples of Other ancestry"> ##INFO=<ID=AF-oth-XX,Number=A,Type=Float,Description="Alternate allele frequency in XX samples of Other ancestry"> ##INFO=<ID=nhomalt-oth-XX,Number=A,Type=Integer,Description="Count of homozygous individuals in XX samples of Other ancestry"> ##INFO=<ID=AC-amr-XX,Number=A,Type=Integer,Description="Alternate allele count for XX samples of Latino ancestry"> ##INFO=<ID=AN-amr-XX,Number=1,Type=Integer,Description="Total number of alleles in XX samples of Latino ancestry"> ##INFO=<ID=AF-amr-XX,Number=A,Type=Float,Description="Alternate allele frequency in XX samples of Latino ancestry"> ##INFO=<ID=nhomalt-amr-XX,Number=A,Type=Integer,Description="Count of homozygous individuals in XX samples of Latino ancestry"> ##INFO=<ID=AC-XX,Number=A,Type=Integer,Description="Alternate allele count for XX samples"> ##INFO=<ID=AN-XX,Number=1,Type=Integer,Description="Total number of alleles in XX samples">

It is obvious that _ has been replaced with -. In addition, male/female seems to be XY/XX now.

I solved this problem by manually update the gnomad_fields_to_keep file, but this could be annoying for those who install bcbio relying on the pipeline.

Would you please update the file gnomad_fields_to_keep into the latest version?
Thanks~

@btesson-lysarc
Copy link

Hello, I think there is still a small issue with the hg38 recipe for dbsnp, the version number should also be updated:
version=GCF_000001405.38 should be version=GCF_000001405.40

Thanks a lot for all the work put into bcbio.

@naumenko-sa
Copy link
Contributor

Thanks @wangpenhok !
I've updated the fields file and the recipe.
I have not changed the recipe version to avoid gnomad updates for users.

@matthdsm I think you compiled that list years ago, could you please review?
Smaller populations are not in Gnomad anymore.

@matthdsm
Copy link
Member

@matthdsm I think you compiled that list years ago, could you please review?

Hi Serhiy,

It seems you already recompiled the list? Do you need me to do something more?

@tt9756067
Copy link

Thanks Serhiy!

I believe that also https://github.com/chapmanb/cloudbiolinux/blob/master/ggd-recipes/hg19/dbsnp.yaml needs to be changed accordingly

/Linda

Hi Serhiy,

I got the same problem but for GRCh37. Would you mind also updating the dbsnp.yaml for GRCh37?

TT

@Roscutts
Copy link

Just to note I have had the same problem now with GCF_000001405.38 should be version=GCF_000001405.40 but I see it has already been noted in the comments

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@btesson-lysarc @Roscutts @matthdsm @naumenko-sa @lindakohn @jchien-ucd @wangpenhok @tt9756067