error about snakemake #70

yoshihiko1218 · 2022-02-27T08:40:24Z

Hi, when I ran "snakemake -j 1", it return "Error: no Snakefile found, tried Snakefile, snakefile, workflow/Snakefile, workflow/snakefile."
Is there anything I might miss or should I run this in a specific location? snakemake was installed correctly.
Thank you!

creisle · 2022-02-28T18:06:32Z

Snakemake expects the default workflow file to be named "Snakefile" and in your current directory unless you pass it an argument to specify otherwise. So in this case it would need to be run in the directory where you cloned this repository. Is that where you ran it from?

yoshihiko1218 · 2022-03-03T08:18:27Z

Thank you! I figured it out. Then in the next step when I run snakemake -j 48, I met following error:

Building DAG of jobs...
Using shell: /bin/bash
Provided cores: 48
Rules claiming more threads will be scaled down.
Singularity containers: ignored
Job stats:
job count min threads max threads

all 1 1 1
all_diseases 1 1 1
all_drugs 1 1 1
load_PMC4232638 1 1 1
load_PMC4468049 1 1 1
load_approvals 1 1 1
load_cancerhotspots 1 1 1
load_cgi 1 1 1
load_civic 1 1 1
load_do 1 1 1
load_docm 1 1 1
load_ensembl 1 1 1
load_local 6 1 1
load_moa 1 1 1
load_ncit 1 1 1
load_oncotree 1 1 1
load_refseq 1 1 1
load_uberon 1 1 1
total 23 1 1

Select jobs to execute...

[Thu Mar 3 16:10:21 2022]
rule load_local:
input: snakemake_data/local/chromosomes.json
output: snakemake_data/local-chromosomes.COMPLETE
log: snakemake_logs/local-chromosomes.logs.txt
jobid: 6
wildcards: local=chromosomes
resources: tmpdir=/tmp

[Thu Mar 3 16:10:21 2022]
rule load_local:
input: snakemake_data/local/evidenceLevels.json
output: snakemake_data/local-evidenceLevels.COMPLETE
log: snakemake_logs/local-evidenceLevels.logs.txt
jobid: 8
wildcards: local=evidenceLevels
resources: tmpdir=/tmp

[Thu Mar 3 16:10:21 2022]
rule load_local:
input: snakemake_data/local/aacr.json
output: snakemake_data/local-aacr.COMPLETE
log: snakemake_logs/local-aacr.logs.txt
jobid: 10
wildcards: local=aacr
resources: tmpdir=/tmp

[Thu Mar 3 16:10:21 2022]
rule load_local:
input: snakemake_data/local/asco.json
output: snakemake_data/local-asco.COMPLETE
log: snakemake_logs/local-asco.logs.txt
jobid: 12
wildcards: local=asco
resources: tmpdir=/tmp

[Thu Mar 3 16:10:21 2022]
rule load_approvals:
output: snakemake_data/fdaApprovals.COMPLETE
log: snakemake_logs/fdaApprovals.logs.txt
jobid: 30
resources: tmpdir=/tmp

[Thu Mar 3 16:10:21 2022]
rule load_local:
input: snakemake_data/local/signatures.json
output: snakemake_data/local-signatures.COMPLETE
log: snakemake_logs/local-signatures.logs.txt
jobid: 4
wildcards: local=signatures
resources: tmpdir=/tmp

[Thu Mar 3 16:10:21 2022]
rule load_local:
input: snakemake_data/local/vocab.json
output: snakemake_data/local-vocab.COMPLETE
log: snakemake_logs/local-vocab.logs.txt
jobid: 2
wildcards: local=vocab
resources: tmpdir=/tmp

[Thu Mar 3 16:10:21 2022]
Error in rule load_local:
jobid: 6
output: snakemake_data/local-chromosomes.COMPLETE
log: snakemake_logs/local-chromosomes.logs.txt (check log file(s) for error message)
shell:
node bin/load.js file ontology snakemake_data/local/chromosomes.json &> snakemake_logs/local-chromosomes.logs.txt; cp snakemake_logs/local-chromosomes.logs.txt snakemake_data/local-chromosomes.COMPLETE
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

[Thu Mar 3 16:10:21 2022]
Error in rule load_local:
jobid: 8
output: snakemake_data/local-evidenceLevels.COMPLETE
log: snakemake_logs/local-evidenceLevels.logs.txt (check log file(s) for error message)
shell:
node bin/load.js file ontology snakemake_data/local/evidenceLevels.json &> snakemake_logs/local-evidenceLevels.logs.txt; cp snakemake_logs/local-evidenceLevels.logs.txt snakemake_data/local-evidenceLevels.COMPLETE
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

[Thu Mar 3 16:10:21 2022]
Error in rule load_local:
jobid: 10
output: snakemake_data/local-aacr.COMPLETE
log: snakemake_logs/local-aacr.logs.txt (check log file(s) for error message)
shell:
node bin/load.js file ontology snakemake_data/local/aacr.json &> snakemake_logs/local-aacr.logs.txt; cp snakemake_logs/local-aacr.logs.txt snakemake_data/local-aacr.COMPLETE
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

[Thu Mar 3 16:10:21 2022]
Error in rule load_local:
jobid: 12
output: snakemake_data/local-asco.COMPLETE
log: snakemake_logs/local-asco.logs.txt (check log file(s) for error message)
shell:
node bin/load.js file ontology snakemake_data/local/asco.json &> snakemake_logs/local-asco.logs.txt; cp snakemake_logs/local-asco.logs.txt snakemake_data/local-asco.COMPLETE
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

[Thu Mar 3 16:10:21 2022]
Error in rule load_approvals:
jobid: 30
output: snakemake_data/fdaApprovals.COMPLETE
log: snakemake_logs/fdaApprovals.logs.txt (check log file(s) for error message)
shell:
node bin/load.js api fdaApprovals &> snakemake_logs/fdaApprovals.logs.txt; cp snakemake_logs/fdaApprovals.logs.txt snakemake_data/fdaApprovals.COMPLETE
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

[Thu Mar 3 16:10:21 2022]
Error in rule load_local:
jobid: 4
output: snakemake_data/local-signatures.COMPLETE
log: snakemake_logs/local-signatures.logs.txt (check log file(s) for error message)
shell:
node bin/load.js file ontology snakemake_data/local/signatures.json &> snakemake_logs/local-signatures.logs.txt; cp snakemake_logs/local-signatures.logs.txt snakemake_data/local-signatures.COMPLETE
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

[Thu Mar 3 16:10:21 2022]
Error in rule load_local:
jobid: 2
output: snakemake_data/local-vocab.COMPLETE
log: snakemake_logs/local-vocab.logs.txt (check log file(s) for error message)
shell:
node bin/load.js file ontology snakemake_data/local/vocab.json &> snakemake_logs/local-vocab.logs.txt; cp snakemake_logs/local-vocab.logs.txt snakemake_data/local-vocab.COMPLETE
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /home/majiayan/software/pori_graphkb_loader/.snakemake/log/2022-03-03T161021.599949.snakemake.log

creisle · 2022-03-03T18:56:25Z

I will need to see the log file(s) for the job that failed to be able to determine the cause of the error. Could you post the contents of one of the failing jobs? ex snakemake_logs/fdaApprovals.logs.txt

yoshihiko1218 · 2022-03-09T03:04:28Z

Hi! the log looks like this:

node:internal/modules/cjs/loader:936
throw err;
^

Error: Cannot find module 'json-cycle'
Require stack:

/home/majiayan/software/pori_graphkb_loader/src/graphkb.js
/home/majiayan/software/pori_graphkb_loader/src/index.js
/home/majiayan/software/pori_graphkb_loader/bin/load.js
at Function.Module._resolveFilename (node:internal/modules/cjs/loader:933:15)
at Function.Module._load (node:internal/modules/cjs/loader:778:27)
at Module.require (node:internal/modules/cjs/loader:999:19)
at require (node:internal/modules/cjs/helpers:102:18)
at Object. (/home/majiayan/software/pori_graphkb_loader/src/graphkb.js:5:12)
at Module._compile (node:internal/modules/cjs/loader:1097:14)
at Object.Module._extensions..js (node:internal/modules/cjs/loader:1151:10)
at Module.load (node:internal/modules/cjs/loader:975:32)
at Function.Module._load (node:internal/modules/cjs/loader:822:12)
at Module.require (node:internal/modules/cjs/loader:999:19) {
code: 'MODULE_NOT_FOUND',
requireStack: [
'/home/majiayan/software/pori_graphkb_loader/src/graphkb.js',
'/home/majiayan/software/pori_graphkb_loader/src/index.js',
'/home/majiayan/software/pori_graphkb_loader/bin/load.js'
]
}

Node.js v17.6.0

creisle · 2022-03-09T04:07:58Z

Are you running it with --use-singularity ? If so are you on the latest version of the loader? This looks similar to the bug we fixed in 6.2.0 for the singularity container where singularity was overriding the working directory of the docker container.

yoshihiko1218 · 2022-03-09T04:28:00Z

My command was "snakemake -j 48". Also, I redownloaded latest version and ran again but got same result.

yoshihiko1218 · 2022-03-09T04:53:03Z

Also, when I try to do npm install, I get error "Cannot read properties of null (reading 'pickAlgorithm')". I did some google research and saw a solution that says I can run "npm install -g [email protected]". However, with this command, I get "11 packages are looking for funding run npm fund for details" and when I run "npm fund". it says "@bcgsc-pori/[email protected]". Is this tool need any application for use?
Sorry for bunch of question... I am new to npm : (

creisle · 2022-03-09T20:01:39Z

My command was "snakemake -j 48". Also, I redownloaded latest version and ran again but got same result.

I have been using node 16 since it tis the latest LTS version. Its possible it is related to the change in node versions. I will try installing node 17 here and see if I can reproduce your error

Also, when I try to do npm install, I get error "Cannot read properties of null (reading 'pickAlgorithm')". I did some google research and saw a solution that says I can run "npm install -g [email protected]". However, with this command, I get "11 packages are looking for funding run npm fund for details" and when I run "npm fund". it says "@bcgsc-pori/[email protected]". Is this tool need any application for use?
Sorry for bunch of question... I am new to npm : (

npm fund is just for use if you want to donate to open source packages that ask for funding. It shouldn't affect the install process

creisle · 2022-03-11T00:09:27Z

I've looked into this a bit and it seems like there are some outstanding issues with node-gyp/node in node 17 still. I've found a number of tickets related to it for different packages but they appear to be outstanding issues still

Would you be able to try node 16, or using the docker container? With node, generally only the even numbered releases are long-term-stable (LTS) releases so we haven't added support for 17 officially and we will probably wait for 18.

If you need to use 17, did you run into any issues during the npm install stage? I suspect it may not have installed correctly

yoshihiko1218 · 2022-03-11T02:41:35Z

Hi, I downgraded node version to stable version. I ran npm install and I think it succeed.
"(venv) (base) root@dell:/home/majiayan/software/pori_graphkb_loader-6.2.0# npm install
npm WARN deprecated [email protected]: Deprecated due to CVE-2021-21366 resolved in 0.5.0
npm WARN deprecated [email protected]: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details.

added 1180 packages in 3m

76 packages are looking for funding
run npm fund for details"

When I reran snakemake, it seems that error message has changed:
"(venv) (base) root@dell:/home/majiayan/software/pori_graphkb_loader-6.2.0# cat snakemake_logs/fdaApprovals.logs.txt
usage: load.js [-h] [-g GRAPHKB] [-u USERNAME] -p PASSWORD [--pubmed PUBMED]
[--errorLogPrefix ERRORLOGPREFIX] [--maxRecords MAXRECORDS]
{api,file,civic,cosmic} ...
load.js: error: the following arguments are required: -p/--password"

Are those database permission-required?

creisle · 2022-03-21T18:25:40Z

@yoshihiko1218 Sorry for the delay! I somehow missed your reply in my notifications. Yes those are the account details for the user that will be loading the data into the database via the API. If you have set up via the demo/dev instructions your username and password should be something like graphkb_importer and secret (note: these are for dev/testing only and users are expected to change them at a later date if they continue using the same instance)

alirizaaribas-ibg · 2022-04-13T13:20:01Z

Hello, I tried to run snakemake. Here are the errors I couldn't find a solution. Can you please help me? My configuration is Ubuntu 20.04, 256GB RAM, 270GB space on disk, Xeon 56core cpu, docker v21.

.snakemake/log/2022-04-13T143705.123185.snakemake.log
Building DAG of jobs...
Using shell: /bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Singularity containers: ignored
Job stats:
job                        count    min threads    max threads
-----------------------  -------  -------------  -------------
all                            1              1              1
all_diseases                   1              1              1
all_drugs                      1              1              1
download_PMC4232638            1              1              1
download_PMC4468049            1              1              1
download_cancerhotspots        1              1              1
download_cgi                   1              1              1
download_ensembl               1              1              1
download_local_data            5              1              1
download_ncit                  1              1              1
download_refseq                1              1              1
download_uberon                1              1              1
load_PMC4232638                1              1              1
load_PMC4468049                1              1              1
load_approvals                 1              1              1
load_cancerhotspots            1              1              1
load_cgi                       1              1              1
load_civic                     1              1              1
load_do                        1              1              1
load_docm                      1              1              1
load_ensembl                   1              1              1
load_local                     6              1              1
load_moa                       1              1              1
load_ncit                      1              1              1
load_oncotree                  1              1              1
load_refseq                    1              1              1
load_uberon                    1              1              1
total                         36              1              1

Select jobs to execute...

[Wed Apr 13 14:37:05 2022]
rule load_local:
    input: snakemake_data/local/vocab.json
    output: snakemake_data/local-vocab.COMPLETE
    log: snakemake_logs/local-vocab.logs.txt
    jobid: 2
    wildcards: local=vocab
    resources: tmpdir=/tmp

[Wed Apr 13 14:37:14 2022]
Finished job 2.
1 of 36 steps (3%) done
Select jobs to execute...

[Wed Apr 13 14:37:14 2022]
rule download_local_data:
    output: snakemake_data/local/signatures.json
    jobid: 5
    wildcards: local=signatures
    resources: tmpdir=/tmp

[Wed Apr 13 14:37:15 2022]
Finished job 5.
2 of 36 steps (6%) done
Select jobs to execute...

[Wed Apr 13 14:37:15 2022]
rule load_local:
    input: snakemake_data/local/signatures.json
    output: snakemake_data/local-signatures.COMPLETE
    log: snakemake_logs/local-signatures.logs.txt
    jobid: 4
    wildcards: local=signatures
    resources: tmpdir=/tmp

[Wed Apr 13 14:37:18 2022]
Finished job 4.
3 of 36 steps (8%) done
Select jobs to execute...

[Wed Apr 13 14:37:18 2022]
rule download_PMC4232638:
    output: snakemake_data/PMC4232638/13059_2014_484_MOESM2_ESM.xlsx
    jobid: 27
    resources: tmpdir=/tmp

[Wed Apr 13 14:37:26 2022]
Finished job 27.
4 of 36 steps (11%) done
Select jobs to execute...

[Wed Apr 13 14:37:26 2022]
rule download_ensembl:
    output: snakemake_data/ensembl/biomart_export.tsv
    jobid: 35
    resources: tmpdir=/tmp

[Wed Apr 13 14:38:21 2022]
Finished job 35.
5 of 36 steps (14%) done
Select jobs to execute...

[Wed Apr 13 14:38:21 2022]
rule download_local_data:
    output: snakemake_data/local/chromosomes.json
    jobid: 7
    wildcards: local=chromosomes
    resources: tmpdir=/tmp

[Wed Apr 13 14:38:22 2022]
Finished job 7.
6 of 36 steps (17%) done
Select jobs to execute...

[Wed Apr 13 14:38:22 2022]
rule load_local:
    input: snakemake_data/local/chromosomes.json
    output: snakemake_data/local-chromosomes.COMPLETE
    log: snakemake_logs/local-chromosomes.logs.txt
    jobid: 6
    wildcards: local=chromosomes
    resources: tmpdir=/tmp

[Wed Apr 13 14:38:23 2022]
Finished job 6.
7 of 36 steps (19%) done
Select jobs to execute...

[Wed Apr 13 14:38:23 2022]
rule load_PMC4232638:
    input: snakemake_data/local-vocab.COMPLETE, snakemake_data/local-signatures.COMPLETE, snakemake_data/local-chromosomes.COMPLETE, snakemake_data/PMC4232638/13059_2014_484_MOESM2_ESM.xlsx
    output: snakemake_data/PMC4232638.COMPLETE
    log: snakemake_logs/PMC4232638.logs.txt
    jobid: 26
    resources: tmpdir=/tmp

[Wed Apr 13 14:40:23 2022]
Finished job 26.
8 of 36 steps (22%) done
Select jobs to execute...

[Wed Apr 13 14:40:23 2022]
rule download_local_data:
    output: snakemake_data/local/aacr.json
    jobid: 11
    wildcards: local=aacr
    resources: tmpdir=/tmp

[Wed Apr 13 14:40:23 2022]
Finished job 11.
9 of 36 steps (25%) done
Select jobs to execute...

[Wed Apr 13 14:40:23 2022]
rule load_local:
    input: snakemake_data/local/aacr.json
    output: snakemake_data/local-aacr.COMPLETE
    log: snakemake_logs/local-aacr.logs.txt
    jobid: 10
    wildcards: local=aacr
    resources: tmpdir=/tmp

[Wed Apr 13 14:40:25 2022]
Finished job 10.
10 of 36 steps (28%) done
Select jobs to execute...

[Wed Apr 13 14:40:25 2022]
rule download_local_data:
    output: snakemake_data/local/evidenceLevels.json
    jobid: 9
    wildcards: local=evidenceLevels
    resources: tmpdir=/tmp

[Wed Apr 13 14:40:26 2022]
Finished job 9.
11 of 36 steps (31%) done
Select jobs to execute...

[Wed Apr 13 14:40:26 2022]
rule load_local:
    input: snakemake_data/local/evidenceLevels.json
    output: snakemake_data/local-evidenceLevels.COMPLETE
    log: snakemake_logs/local-evidenceLevels.logs.txt
    jobid: 8
    wildcards: local=evidenceLevels
    resources: tmpdir=/tmp

[Wed Apr 13 14:40:28 2022]
Finished job 8.
12 of 36 steps (33%) done
Select jobs to execute...

[Wed Apr 13 14:40:28 2022]
rule load_approvals:
    output: snakemake_data/fdaApprovals.COMPLETE
    log: snakemake_logs/fdaApprovals.logs.txt
    jobid: 30
    resources: tmpdir=/tmp

[Wed Apr 13 14:41:20 2022]
Finished job 30.
13 of 36 steps (36%) done
Select jobs to execute...

[Wed Apr 13 14:41:20 2022]
rule download_ncit:
    output: snakemake_data/ncit/Thesaurus.txt
    jobid: 15
    resources: tmpdir=/tmp

[Wed Apr 13 14:41:24 2022]
Finished job 15.
14 of 36 steps (39%) done
Select jobs to execute...

[Wed Apr 13 14:41:24 2022]
rule load_ncit:
    input: snakemake_data/local-vocab.COMPLETE, snakemake_data/ncit/Thesaurus.txt
    output: snakemake_data/ncit.COMPLETE
    log: snakemake_logs/ncit.logs.txt
    jobid: 14
    resources: tmpdir=/tmp

[Wed Apr 13 14:54:54 2022]
Finished job 14.
15 of 36 steps (42%) done
Select jobs to execute...

[Wed Apr 13 14:54:54 2022]
rule load_do:
    input: snakemake_data/ncit.COMPLETE, snakemake_data/do/doid.json
    output: snakemake_data/do.COMPLETE
    log: snakemake_logs/do.logs.txt
    jobid: 16
    resources: tmpdir=/tmp

[Wed Apr 13 14:57:01 2022]
Finished job 16.
16 of 36 steps (44%) done
Select jobs to execute...

[Wed Apr 13 14:57:01 2022]
rule load_docm:
    input: snakemake_data/local-vocab.COMPLETE, snakemake_data/local-signatures.COMPLETE, snakemake_data/local-chromosomes.COMPLETE, snakemake_data/ncit.COMPLETE, snakemake_data/do.COMPLETE
    output: snakemake_data/docm.COMPLETE
    log: snakemake_logs/docm.logs.txt
    jobid: 23
    resources: tmpdir=/tmp

[Wed Apr 13 15:10:26 2022]
Error in rule load_docm:
    jobid: 23
    output: snakemake_data/docm.COMPLETE
    log: snakemake_logs/docm.logs.txt (check log file(s) for error message)
    shell:
        node bin/load.js --username graphkb_importer --password secret --graphkb http://localhost:8080/api api docm &> snakemake_logs/docm.logs.txt; cp snakemake_logs/docm.logs.txt snakemake_data/docm.COMPLETE
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2022-04-13T143705.123185.snakemake.log

Here is the file for running steps:
docker-run.txt

Is there difference to access at 0.0.0.0 or 127.0.0.1 or localhost? Because when I tried at 127.0.0.1, it redirects to localhost and fails. When I tried at localhost, successfully logins. If there was difference, snakemake might be unable to find the localhost.

Finally, I have read about the fda issue, I manually downloaded it from the issue #61 . But couldn't test it without solving the error above.

Docker pori stack is stable:

iprdemo user and client are working:

graphkb client has lots of errors:

Thanks for reading to the end.

creisle self-assigned this Mar 8, 2022

creisle added the awaiting reply pending reply from reporter label Mar 8, 2022

creisle removed the awaiting reply pending reply from reporter label Mar 9, 2022

creisle added the awaiting reply pending reply from reporter label Mar 11, 2022

creisle mentioned this issue Apr 21, 2022

Invalid parameter: redirect_uri bcgsc/pori#49

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

error about snakemake #70

error about snakemake #70

yoshihiko1218 commented Feb 27, 2022

creisle commented Feb 28, 2022

yoshihiko1218 commented Mar 3, 2022

creisle commented Mar 3, 2022 •

edited

Loading

yoshihiko1218 commented Mar 9, 2022

creisle commented Mar 9, 2022

yoshihiko1218 commented Mar 9, 2022

yoshihiko1218 commented Mar 9, 2022

creisle commented Mar 9, 2022

creisle commented Mar 11, 2022 •

edited

Loading

yoshihiko1218 commented Mar 11, 2022

creisle commented Mar 21, 2022

alirizaaribas-ibg commented Apr 13, 2022

error about snakemake #70

error about snakemake #70

Comments

yoshihiko1218 commented Feb 27, 2022

creisle commented Feb 28, 2022

yoshihiko1218 commented Mar 3, 2022

creisle commented Mar 3, 2022 • edited Loading

yoshihiko1218 commented Mar 9, 2022

creisle commented Mar 9, 2022

yoshihiko1218 commented Mar 9, 2022

yoshihiko1218 commented Mar 9, 2022

creisle commented Mar 9, 2022

creisle commented Mar 11, 2022 • edited Loading

yoshihiko1218 commented Mar 11, 2022

creisle commented Mar 21, 2022

alirizaaribas-ibg commented Apr 13, 2022

creisle commented Mar 3, 2022 •

edited

Loading

creisle commented Mar 11, 2022 •

edited

Loading