chipseeker uses all cores available

[kelly-sovacool]

nextflow error

ERROR ~ Error executing process > 'CHIPSEQ:ANNOTATE:CHIPSEEKER_PEAKPLOT (A549_CTCF.macs_broad)'

Caused by:
  Process `CHIPSEQ:ANNOTATE:CHIPSEEKER_PEAKPLOT (A549_CTCF.macs_broad)` terminated with an error exit status (1)

Command executed:

  chipseeker_peakplot.R \
      --peak A549_CTCF.macs_broad.sorted.merged.consensus.bed \
      --outfile-prefix A549_CTCF.macs_broad \
      --genome-txdb TxDb.Hsapiens.UCSC.hg38.knownGene \
      --genome-annot org.Hs.eg.db

Command exit status:
  1

Command output:
  GRanges object with 45826 ranges and 2 metadata columns:
            seqnames            ranges strand |    pvalue    qvalue
               <Rle>         <IRanges>  <Rle> | <numeric> <numeric>
        [1]     chr1     267823-268191      * |   58.9665   56.0841
        [2]     chr1     586021-586363      * |   19.9394   17.4389
        [3]     chr1     778628-779118      * |   10.4606    8.0559
        [4]     chr1     869676-870185      * |   67.5469   64.6303
        [5]     chr1     904573-905010      * |   83.9057   80.8652
        ...      ...               ...    ... .       ...       ...
    [45822]     chrY 19503150-19503517      * |  10.99270   8.47082
    [45823]     chrY 19817032-19817274      * |   8.49199   6.12316
    [45824]     chrY 20558735-20559105      * |  12.35630   9.92217
    [45825]     chrY 20740733-20740950      * |   8.60358   6.12389
    [45826]     chrY 26315104-26315418      * |   7.09308   4.76851
    -------
    seqinfo: 24 sequences from an unspecified genome; no seqlengths
  >> binning method is used...2024-01-15 22:26:49
  >> preparing body regions by gene... 2024-01-15 22:26:49
  >> preparing tag matrix by binning...  2024-01-15 22:26:49
  >> preparing matrix with extension from (TSS-20%)~(TTS+20%)... 2024-01-15 22:26:49
  >> 113 peaks(0.6451981%), having lengths smaller than 800bp, are filtered... 2024-01-15 22:27:12

Command error:
  Warning messages:
  1: In normalizePath("~") :
    path[1]="/home/sovacoolkl": No such file or directory
  2: package 'dplyr' was built under R version 4.3.2
  Error in t.star[r, ] <- res[[r]] :
    number of items to replace is not a multiple of replacement length
  Calls: plotPeakProf2 ... plotAvgProf.binning.internal -> getTagCount -> getTagCiMatrix -> boot
  In addition: Warning message:
  In parallel::mclapply(seq_len(RR), fn, mc.cores = ncpus) :
    scheduled cores 25, 26, 29, 32, 34, 36, 38, 39, 44, 46, 53, 57, 58, 60, 61, 64, 69, 71, 72, 77, 79, 84, 85, 91, 95, 96, 97, 99, 102, 105, 108, 110, 111, 114 did not deliver results, all values of the jobs will be affected
  Execution halted

Work dir:
  /gpfs/gsfs12/users/sovacoolkl/champagne_test/work/ea/6d2e28a62aba59f7f0430dd940bd04

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

 -- Check '.nextflow.log' file for details

out of memory

Also got an email from Biowulf about out of memory kills, all jobs were from chipseeker from this same run:

We noticed that your biowulf jobs have caused 675 memory kills amongst 8 jobs
on the cluster in the last hour. These are the result of jobs that attempt to
use more memory than was allocated to the job. In addition to the wasted compute
resources, these frequent out of memory kills can destabilize nodes and shared
file systems and therefore affect other user's jobs.

Here are some of the jobs (swarms and job arrays are denoted with _*):

17161156
17161489
17161502
17161881
17161908
17161971
17162010
17162258

These jobs can be seen with the command

dashboard_cli jobs --mem-over --since 2024-01-15T22:00:00 --until 2024-01-15T23:00:00

You can also go to https://hpc.nih.gov/dashboard -> Job Info and click on the
jobid(s) to visualize the job history. Jobs with a yellow triangle in the eval
column had out of memory kill events.

Please carefully check your job submissions. We would be glad to help if you
have any questions.

[kelly-sovacool]

Opened an issue in chipseeker about specifying cores https://www.github.com/YuLab-SMU/ChIPseeker/issues/229

[kelly-sovacool]

Idea from @lobanovav: for jobs like chipseeker that use all cores available, use the slurm --exclusive flag to allocate the entire node so the job will not be killed. I'm not 100% certain how to do this in nextflow but I think it should be possible.

[kelly-sovacool]

clusterOptions = '--exclusive' should do the trick
https://www.nextflow.io/docs/latest/reference/process.html#process-clusteroptions