rvnpl collapse preocess have been remained at "MESSAGE: 4 families with a total of 30 samples will be scanned for 28,488 pre-defined units"

[zhangshouwei309194]

Dear author:
My process kept on going, but no new output. The message is as follows:
MESSAGE: Binary trait detected in [/annogene/cloud/bioinfo/PMO/shouweizhang/Analysis/B_MED-001/rvnpl_association/class2/INPUT/class2.final.ped]
MESSAGE: Checking local resources 5/5 ...
MESSAGE: 31 samples found in [/annogene/cloud/bioinfo/PMO/shouweizhang/Analysis/B_MED-001/rvnpl_association/class2/INPUT/class2.final.vcf.gz]
MESSAGE: 3 families with a total of 30 samples will be scanned for 28,488 pre-defined units

I tried many approaches, but it remains. Next I will introduce them in detail.
First:
I cheched my ped file:
family9 Q101 0 0 1 0
family9 Q103 Q104 Q105 2 2
family9 Q104 Q101 Q120 1 0
family9 Q105 0 0 2 1
family9 Q108 Q101 Q120 2 0
family9 Q109 Q101 Q120 2 2
family9 Q114 0 0 2 0
family9 Q115 0 0 1 1
family9 Q116 Q115 Q114 2 0
family9 Q117 Q115 Q114 2 2
family9 Q120 0 0 2 1
family9 Q78 Q104 Q105 2 2
family9 Q79 Q104 Q105 2 2
jinrong Q208A 0 0 1 0
jinrong Q209A 0 0 2 2
jinrong Q210A Q208A Q209A 2 0
jinrong q211B Q208A Q209A 2 2
jinrong Q212A Q208A Q209A 2 2
jinrong Q213A Q208A Q209A 2 0
jinrong Q214A Q208A Q209A 2 2
jinrong Q305 Q208A Q209A 1 0
jinrong Q306 Q305 Q308 2 2
jinrong Q307 Q305 Q308 1 0
jinrong Q308 0 0 2 1
weijun LN10 Q204A LN19 2 0
weijun LN11 LN20 Q202A 2 2
weijun LN19 0 0 2 0
weijun LN20 0 0 1 1
weijun Q202A Q204A LN19 2 2
weijun Q204A 0 0 1 0
Because I have missing (Uncertain pathogenicity ) phenotype equal to 0 in column six, so i change them to "1". Moreover, "Q114, Q115, Q116, Q117 in family9 have no relation to othe samples, because i delete one sample.( If one sample just have one parent (not both), it will send erros and exit ). As follows:
family9 Q78 Q104 Q105 2 2
family9 Q79 Q104 Q105 2 2
family9 Q101 0 0 1 1
family9 Q103 Q104 Q105 2 2
family9 Q104 Q101 Q120 1 1
family9 Q105 0 0 2 1
family9 Q108 Q101 Q120 2 1
family9 Q109 Q101 Q120 2 2
family9 Q120 0 0 2 1
family10 Q114 0 0 2 1
family10 Q115 0 0 1 1
family10 Q116 Q115 Q114 2 1
family10 Q117 Q115 Q114 2 2
jinrong Q208A 0 0 1 1
jinrong Q209A 0 0 2 2
jinrong Q210A Q208A Q209A 2 1
jinrong Q212A Q208A Q209A 2 2
jinrong Q213A Q208A Q209A 2 1
jinrong Q214A Q208A Q209A 2 2
jinrong Q305 Q208A Q209A 1 1
jinrong Q306 Q305 Q308 2 2
jinrong Q307 Q305 Q308 1 1
jinrong Q308 0 0 2 1
jinrong q211B Q208A Q209A 2 2
weijun LN10 Q204A LN19 2 1
weijun LN11 LN20 Q202A 2 2
weijun LN19 0 0 2 1
weijun LN20 0 0 1 1
weijun Q202A Q204A LN19 2 2
weijun Q204A 0 0 1 1
It remains as before.

Second
To solve this problem, i hust picked out chromosome 1 to run:
##fileformat=VCFv4.0
#CHROM POS ID REF ALT QUAL FILTER INFO FORMAT Q101 Q103 Q104 Q105 Q108 Q109 Q114 Q115 Q116 Q117 Q120 Q78 Q79 Q208A Q209A Q210A q211B Q212A Q213A Q214A Q305 Q306 Q307 Q308 LN10 LN11 LN19 LN20 Q202A Q204A
1 13273 . G C 4148.27 PASS gnomAD_EAS=0.0572 GT 0|0 0|0 0|0 0|0 0|0 0|0 0|0 0|0 0|0 0|0 0|0 0|0 0|0 0|0 0|0 0|0 0|0 0|0 0|0 0|0 0|1 0|0 1|1 0|0 0|0 0|0 0|0 0|0 0|0 0|0
1 13302 rs180734498 C T 154.80 PASS gnomAD_EAS=0.0912 GT 0|0 0|0 0|0 0|0 0|0 0|0 0|0 0|0 0|0 0|0 0|0 0|0 0|0 0|0 0|0 0|0 0|0 0|0 0|0 0|0 0|0 1|1 0|0 0|0 0|0 0|0 0|0 0|0 0|0 0|0
......
......
......

MESSAGE: Binary trait detected in [/annogene/cloud/bioinfo/PMO/shouweizhang/Analysis/B_MED-001/rvnpl_association/class2/collapse_step1_newtest2/class2.final.m0to1.ped]
MESSAGE: Checking local resources 5/5 ...
MESSAGE: 30 samples found in [/annogene/cloud/bioinfo/PMO/shouweizhang/Analysis/B_MED-001/rvnpl_association/class2/collapse_step1_newtest2/test_chr1/class2.m.chr1.final.vcf.gz]
MESSAGE: 4 families with a total of 30 samples will be scanned for 28,488 pre-defined units
No change, it remains.

Third
I just want to picked fewer to run, eg： 1000 site, but it remains.

"Fourth"
This is my command (before):
rvnpl collapse --fam /annogene/cloud/bioinfo/PMO/shouweizhang/Analysis/B_MED-001/rvnpl_association/class2/collapse_step1_newtest2/class2.final.m0to1.ped --vcf /annogene/cloud/bioinfo/PMO/shouweizhang/Analysis/B_MED-001/rvnpl_association/class2/collapse_step1_newtest2/test_chr1/class2.m.chr1.final.vcf.gz --output /annogene/cloud/bioinfo/PMO/shouweizhang/Analysis/B_MED-001/rvnpl_association/class2/collapse_step1_newtest2/test_chr1/rep1 --freq gnomAD_EAS -c 0.01 --rvhaplo
When i run test set in the directory of the sofware, it include "--include_vars", i make a new file (chr1.txt) and add this parameter. The command is as follows:
rvnpl collapse --fam /annogene/cloud/bioinfo/PMO/shouweizhang/Analysis/B_MED-001/rvnpl_association/class2/collapse_step1_newtest2/class2.final.m0to1.ped --vcf /annogene/cloud/bioinfo/PMO/shouweizhang/Analysis/B_MED-001/rvnpl_association/class2/collapse_step1_newtest2/test_chr1/class2.m.chr1.final.vcf.gz --output /annogene/cloud/bioinfo/PMO/shouweizhang/Analysis/B_MED-001/rvnpl_association/class2/collapse_step1_newtest2/test_chr1_include/rep1 --freq gnomAD_EAS -c 0.01 --rvhaplo --include_vars /annogene/cloud/bioinfo/PMO/shouweizhang/Analysis/B_MED-001/rvnpl_association/class2/collapse_step1_newtest2/test_chr1_include/chr1.txt
chr1.txt:
1 13273
1 13302
1 13418
1 13649
1 14610
1 14653
1 14677
1 14907
1 14930
1 14933
1 14976
.......
.......
It remains as before.

Fifth
When i ran test set in the directory of software, the frequency parameter refers to "EVSMAF", my frequency is "gnomAD_EAS", so i changed it to “EASMAF”. Then i ran again, it remains.

I think the software that have been installed is ok, because i finish the test by means of the test set the software provided. Whether it should have some requiredments for pedigree. Yesterday i checked the ped file and find my ped file is ok. I don't know how to resolve this problem. Look foward to your reply. Thanks you!
Best wished for you!
Phillip

[gaow]

@zhangshouwei309194 when it hangs without output, what do you see in top -- do you see rvnpl use , eg 100% CPU?

[zhangshouwei309194]

Name: testchr1includemaf-ts9b6
Namespace: shouweizhang
ServiceAccount: default
Status: Running
Created: Sun May 31 10:19:21 +0800 (50 minutes ago)
Started: Sun May 31 10:19:21 +0800 (50 minutes ago)
Duration: 50 minutes 17 seconds
Total CPU: 0 (corehour)
Total Memory: 0.25 (GBhour)
Parameters:
testchr1includemaf: sh class2_rvnpl_collapse.test.sh
work_dir: /annogene/cloud/bioinfo/PMO/shouweizhang/Analysis/B_MED-001/rvnpl_association/class2/collapse_step1_newtest2/test_chr1_MAF

STEP PODNAME DURATION MESSAGE CPU(corehour) MEMORY(GBhour) MaxCpu(core) MaxMemory(GB)
● testchr1includemaf-ts9b6 testchr1includemaf-ts9b6 50m 0 0.25 0 0.31

It have been running！ This is my newest task, some have been continued for one day.
NAME READY STATUS RESTARTS AGE
rvnpl-collapse-all-chr1-2d7mj 2/2 Running 0 1d
rvnpl-collapse-all-wrfrd 2/2 Running 0 1d
rvnpl-collapse-class1-phlg8 2/2 Running 0 1d
rvnpl-collapse-class2-mskpt 2/2 Running 0 1d
rvnpl-collapse-class3-fcdmm 2/2 Running 0 1d
rvnpl-test-97tmt 2/2 Running 0 12h
rvnpl-test-ntnxf 2/2 Running 0 11h
rvnpl-test2-sklxt 2/2 Running 0 11h
testchr1fewer-j2zck 2/2 Running 0 1h
testchr1include-ppznv 2/2 Running 0 58m
testchr1includemaf-ts9b6 2/2 Running 0 48m

[gaow]

@zhangshouwei309194 if you are running on a cluster, you have to log in ot the cluster node somehow and check the actual CPU usage. You claim that it "hangs" ( I guess that's what you meant by the word "remain"), right? I want to know if it hangs with 0% CPU usage (which means a real hang and is problematic) or it uses 100% CPU (which means it is still running).

[zhangshouwei309194]

We used the aliyun. One minutes ago, i tried the test set that the software provided. It just ran finised for sveral seconds.The details are as follows:
Command:
rvnpl collapse --fam /annogene/cloud/bioinfo/PMO/shouweizhang/rvnpl/software/rvnpl-master/example/100extend_01.ped --vcf /annogene/cloud/bioinfo/PMO/shouweizhang/rvnpl/software/rvnpl-master/example/A1BG/rep1.vcf.gz --output /annogene/cloud/bioinfo/PMO/shouweizhang/rvnpl/software/test/case_control/collapse/rep1 --freq EVSMAF -c 0.01 --rvhaplo --include_vars /annogene/cloud/bioinfo/PMO/shouweizhang/rvnpl/software/rvnpl-master/example/A1BG.txt

LOG:
MESSAGE: Binary trait detected in [/annogene/cloud/bioinfo/PMO/shouweizhang/rvnpl/software/rvnpl-master/example/100extend_01.ped]
MESSAGE: Checking local resources 5/5 ...
MESSAGE: 1,000 samples found in [/annogene/cloud/bioinfo/PMO/shouweizhang/rvnpl/software/rvnpl-master/example/A1BG/rep1.vcf.gz]
MESSAGE: 100 families with a total of 1,000 samples will be scanned for 28,488 pre-defined units
MESSAGE: 1 units (from 24 variants) processed; 0 Mendelian inconsistencies and 0 recombination events handled
MESSAGE: 406 variants ignored due to having MAF > 0.01 and other specified constraints
MESSAGE: 28,487 units ignored due to absence in VCF file
MESSAGE: Archiving regional marker data to directory [/annogene/cloud/bioinfo/PMO/shouweizhang/rvnpl/software/test/case_control/collapse/cache]
MESSAGE: 1 units will be converted to MERLIN format
MESSAGE: 1 units successfully converted to MERLIN format
MESSAGE: Archiving MERLIN format to directory [/annogene/cloud/bioinfo/PMO/shouweizhang/rvnpl/software/test/case_control/collapse/cache]
MESSAGE: Saving data to [/annogene/cloud/bioinfo/PMO/shouweizhang/rvnpl/software/test/case_control/collapse/rep1]

-rw-r--r-- 1 shouweizhang bioinfo 445 May 22 23:28 test1_step1.sh
-rw-r--r-- 1 shouweizhang bioinfo 0 May 22 23:31 rvnpl_teststep1.o.rvnpl-teststep1-chjqx
-rw-r--r-- 1 shouweizhang bioinfo 192 May 22 23:37 cmd
-rw-r--r-- 1 shouweizhang bioinfo 9.1K May 22 23:37 rvnpl_teststep1.e.rvnpl-teststep1-chjqx
-rw-r--r-- 1 shouweizhang bioinfo 0 May 31 11:17 rvnpl_teststep1.o.rvnpl-teststep1-2qpkq
drwxr-xr-x 2 shouweizhang bioinfo 88 May 31 11:18 cache
drwxr-xr-x 3 shouweizhang bioinfo 24 May 31 11:18 rep1
-rw-r--r-- 1 shouweizhang bioinfo 9.9K May 31 11:18 rvnpl_teststep1.e.rvnpl-teststep1-2qpkq

[gaow]

@zhangshouwei309194 Okay good to know! But my question remains whether or not you see actual CPU usage is 0% or 100%. ...? Whatever cloud service you use there has to be a way to monitor the actual CPU usage on a compute node.

[zhangshouwei309194]

At that time, i didn't know how to gointo a yuncluster. The details is as mentioned above. CPU is always “0.0”. Thank you.

[zhangshouwei309194]

Now i am running as this way. If it has a new progress, i will tell you detaills.

[zhangshouwei309194]

My process is always like this. Then i gointo a yuncluster and run the test set. A minutes late. "rvnpl collapse" has been finished.

So i think the yun cluster node is ok. Maybe it's owing to the ped file or vcf. But i check them carefully and could not find its problem. Thank you! If you need, I can send my data for you, just a few MB. Best wishes for you!

[gaow]

Closing this issue as we are redirecting the conversation to #6