Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

star2p fails - cannot make directory #131

Closed
kelly-sovacool opened this issue Nov 19, 2024 · 3 comments
Closed

star2p fails - cannot make directory #131

kelly-sovacool opened this issue Nov 19, 2024 · 3 comments
Labels
bug Something isn't working
Milestone
_not-planned

Comments

@kelly-sovacool
Copy link
Member

kelly-sovacool commented Nov 19, 2024 •
edited
Loading

the pipeline works for me, but for wilfried it fails at star2p

wilfried's log (fail)

/vf/users/CCBR/charlie_test_wil/charlie4/logs/40895447.40907635.star2p.sample=GI1_T.err

Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 56
Rules claiming more threads will be scaled down.
Provided resources: mem_mb=2653, mem_mib=2531, disk_mb=2653, disk_mib=2531
Select jobs to execute...

[Mon Nov 18 14:41:57 2024]
rule star2p:
    input: /vf/users/CCBR/charlie_test_wil/charlie4/results/GI1_T/trim/GI1_T.R1.trim.fastq.gz, /vf/users/CCBR/charlie_test_wil/charlie4/results/GI1_T/trim/GI1_T.R2.trim.fastq.gz, /vf/users/CCBR/charlie_test_wil/charlie4/ref/ref.fixed.gtf, /vf/users/CCBR/charlie_test_wil/charlie4/results/pass1.out.tab
    output: /vf/users/CCBR/charlie_test_wil/charlie4/results/GI1_T/STAR2p/GI1_T_p2.Chimeric.out.junction, /vf/users/CCBR/charlie_test_wil/charlie4/results/GI1_T/STAR2p/GI1_T_p2.Aligned.out.bam, /vf/users/CCBR/charlie_test_wil/charlie4/results/GI1_T/STAR2p/GI1_T_p2.bam, /vf/users/CCBR/charlie_test_wil/charlie4/results/GI1_T/STAR2p/GI1_T_p2.chimeric.bam, /vf/users/CCBR/charlie_test_wil/charlie4/results/GI1_T/STAR2p/GI1_T_p2.non_chimeric.bam, /vf/users/CCBR/charlie_test_wil/charlie4/results/GI1_T/STAR2p/GI1_T_p2.ReadsPerGene.out.tab
    jobid: 0
    reason: Forced execution
    wildcards: sample=GI1_T
    threads: 56
    resources: mem_mb=2653, mem_mib=2531, disk_mb=2653, disk_mib=2531, tmpdir=/tmp


set -exo pipefail

limitSjdbInsertNsj=$(wc -l /vf/users/CCBR/charlie_test_wil/charlie4/results/pass1.out.tab|awk '{print $1+1}')
if [ "$limitSjdbInsertNsj" -lt "400000" ];then limitSjdbInsertNsj="400000";fi

output_prefix=$(dirname /vf/users/CCBR/charlie_test_wil/charlie4/results/GI1_T/STAR2p/GI1_T_p2.bam)/GI1_T_p2.

if [ "SE" == "PE" ];then
# paired-end
    overhang=$(zcat /vf/users/CCBR/charlie_test_wil/charlie4/results/GI1_T/trim/GI1_T.R1.trim.fastq.gz /vf/users/CCBR/charlie_test_wil/charlie4/results/GI1_T/trim/GI1_T.R2.trim.fastq.gz | awk -v maxlen=100 'NR%4==2 {if (length($1) > maxlen+0) maxlen=length($1)}; END {print maxlen-1}')
    echo "sjdbOverhang for STAR: ${overhang}"
    cd /vf/users/CCBR/charlie_test_wil/charlie4/results/GI1_T/STAR2p
    STAR --genomeDir /vf/users/CCBR/charlie_test_wil/charlie4/ref/STAR_no_GTF \
    --outSAMstrandField None  \
    --outFilterType BySJout \
    --outFilterMultimapNmax 20 \
    --outSJfilterOverhangMin 15 15 15 15 \
    --alignSJoverhangMin 15 \
    --alignSJDBoverhangMin 15 \
    --outFilterScoreMin 1 \
    --outFilterMatchNmin 1 \
    --outFilterMismatchNmax 2 \
    --outFilterMismatchNoverLmax 0.3  \
    --alignIntronMin 20 \
    --alignIntronMax 2000000 \
    --alignMatesGapMax 2000000 \
    --readFilesIn /vf/users/CCBR/charlie_test_wil/charlie4/results/GI1_T/trim/GI1_T.R1.trim.fastq.gz /vf/users/CCBR/charlie_test_wil/charlie4/results/GI1_T/trim/GI1_T.R2.trim.fastq.gz \
    --readFilesCommand  zcat \
    --runThreadN 56 \
    --outFileNamePrefix $output_prefix \
    --sjdbFileChrStartEnd /vf/users/CCBR/charlie_test_wil/charlie4/results/pass1.out.tab \
    --chimSegmentMin 15 \
    --chimScoreMin 15 \
    --chimJunctionOverhangMin 18 \
    --chimScoreJunctionNonGTAG 0 \
    --chimOutType Junctions WithinBAM \
    --chimMultimapNmax 10 \
    --limitSjdbInsertNsj $limitSjdbInsertNsj \
    --alignTranscriptsPerReadNmax 20000 \
    --outSAMtype BAM Unsorted \
    --alignEndsProtrude 10 ConcordantPair \
    --outFilterIntronMotifs None \
    --sjdbGTFfile /vf/users/CCBR/charlie_test_wil/charlie4/ref/ref.fixed.gtf \
    --quantMode GeneCounts \
    --outTmpDir /lscratch/$SLURM_JOB_ID/2c4d0ede-8e82-4b06-a59a-8799954ddd04 \
    --sjdbOverhang $overhang \
    --outBAMcompression 0 \
    --outSAMattributes All

    rm -rf GI1_T_p2._STARgenome

else
#single-end
    overhang=$(zcat /vf/users/CCBR/charlie_test_wil/charlie4/results/GI1_T/trim/GI1_T.R1.trim.fastq.gz | awk -v maxlen=100 'NR%4==2 {if (length($1) > maxlen+0) maxlen=length($1)}; END {print maxlen-1}')
    echo "sjdbOverhang for STAR: ${overhang}"
    cd /vf/users/CCBR/charlie_test_wil/charlie4/results/GI1_T/STAR2p
    STAR --genomeDir /vf/users/CCBR/charlie_test_wil/charlie4/ref/STAR_no_GTF \
    --outSAMstrandField None  \
    --outFilterType BySJout \
    --outFilterMultimapNmax 20 \
    --outSJfilterOverhangMin 15 15 15 15 \
    --alignSJoverhangMin 15 \
    --alignSJDBoverhangMin 15 \
    --seedSearchStartLmax 30 \
    --outFilterScoreMin 1 \
    --outFilterMatchNmin 1 \
    --outFilterMismatchNmax 2 \
    --outFilterMismatchNoverLmax 0.3  \
    --alignIntronMin 20 \
    --alignIntronMax 2000000 \
    --alignMatesGapMax 2000000 \
    --readFilesIn /vf/users/CCBR/charlie_test_wil/charlie4/results/GI1_T/trim/GI1_T.R1.trim.fastq.gz \
    --readFilesCommand  zcat \
    --runThreadN 56 \
    --outFileNamePrefix $output_prefix \
    --sjdbFileChrStartEnd /vf/users/CCBR/charlie_test_wil/charlie4/results/pass1.out.tab \
    --chimSegmentMin 15 \
    --chimScoreMin 15 \
    --chimJunctionOverhangMin 18 \
    --chimScoreJunctionNonGTAG 0 \
    --chimOutType Junctions WithinBAM \
    --chimMultimapNmax 10 \
    --limitSjdbInsertNsj $limitSjdbInsertNsj \
    --alignTranscriptsPerReadNmax 20000 \
    --outSAMtype BAM Unsorted \
    --alignEndsProtrude 10 ConcordantPair \
    --outFilterIntronMotifs None \
    --sjdbGTFfile /vf/users/CCBR/charlie_test_wil/charlie4/ref/ref.fixed.gtf \
    --quantMode GeneCounts \
    --outTmpDir /lscratch/$SLURM_JOB_ID/2c4d0ede-8e82-4b06-a59a-8799954ddd04 \
    --sjdbOverhang $overhang \
    --outBAMcompression 0 \
    --outSAMattributes All

    rm -rf ${output_prefix}_STARgenome
fi
sleep 120
mkdir -p /lscratch/$SLURM_JOB_ID/2c4d0ede-8e82-4b06-a59a-8799954ddd04
samtools view -H /vf/users/CCBR/charlie_test_wil/charlie4/results/GI1_T/STAR2p/GI1_T_p2.Aligned.out.bam > /lscratch/$SLURM_JOB_ID/2c4d0ede-8e82-4b06-a59a-8799954ddd04/GI1_T_p2.non_chimeric.sam
cp /lscratch/$SLURM_JOB_ID/2c4d0ede-8e82-4b06-a59a-8799954ddd04/GI1_T_p2.non_chimeric.sam /lscratch/$SLURM_JOB_ID/2c4d0ede-8e82-4b06-a59a-8799954ddd04/GI1_T_p2.chimeric.sam
# ref https://github.com/alexdobin/STAR/issues/678
samtools view -@ 56 /vf/users/CCBR/charlie_test_wil/charlie4/results/GI1_T/STAR2p/GI1_T_p2.Aligned.out.bam | grep "ch:A:1" >> /lscratch/$SLURM_JOB_ID/2c4d0ede-8e82-4b06-a59a-8799954ddd04/GI1_T_p2.chimeric.sam
samtools view -@ 56 /vf/users/CCBR/charlie_test_wil/charlie4/results/GI1_T/STAR2p/GI1_T_p2.Aligned.out.bam | grep -v "ch:A:1" >> /lscratch/$SLURM_JOB_ID/2c4d0ede-8e82-4b06-a59a-8799954ddd04/GI1_T_p2.non_chimeric.sam
ls -alrth
for i in 1 2 3;do
    mkdir -p /lscratch/$SLURM_JOB_ID/2c4d0ede-8e82-4b06-a59a-8799954ddd04_${i}
done
samtools view -@ 56 -b -S /lscratch/$SLURM_JOB_ID/2c4d0ede-8e82-4b06-a59a-8799954ddd04/GI1_T_p2.chimeric.sam | \
samtools sort \
    -l 9 \
    -T /lscratch/$SLURM_JOB_ID/2c4d0ede-8e82-4b06-a59a-8799954ddd04_1 \
    --write-index \
    -@ 56 \
    --output-fmt BAM \
    -o /vf/users/CCBR/charlie_test_wil/charlie4/results/GI1_T/STAR2p/GI1_T_p2.chimeric.bam -
samtools view -@ 56 -b -S /lscratch/$SLURM_JOB_ID/2c4d0ede-8e82-4b06-a59a-8799954ddd04/GI1_T_p2.non_chimeric.sam | \
samtools sort \
    -l 9 \
    -T /lscratch/$SLURM_JOB_ID/2c4d0ede-8e82-4b06-a59a-8799954ddd04_2 \
    --write-index \
    -@ 56 \
    --output-fmt BAM \
    -o /vf/users/CCBR/charlie_test_wil/charlie4/results/GI1_T/STAR2p/GI1_T_p2.non_chimeric.bam -
samtools sort \
    -l 9 \
    -T /lscratch/$SLURM_JOB_ID/2c4d0ede-8e82-4b06-a59a-8799954ddd04_3 \
    --write-index \
    -@ 56 \
    --output-fmt BAM \
    -o /vf/users/CCBR/charlie_test_wil/charlie4/results/GI1_T/STAR2p/GI1_T_p2.bam /vf/users/CCBR/charlie_test_wil/charlie4/results/GI1_T/STAR2p/GI1_T_p2.Aligned.out.bam

Activating singularity image /vf/users/CCBR/charlie_test_wil/charlie4/.snakemake/singularity/3ce85b29f85e5d3fe17c6af860c5f059.simg
++ wc -l /vf/users/CCBR/charlie_test_wil/charlie4/results/pass1.out.tab
++ awk '{print $1+1}'
+ limitSjdbInsertNsj=2426
+ '[' 2426 -lt 400000 ']'
+ limitSjdbInsertNsj=400000
++ dirname /vf/users/CCBR/charlie_test_wil/charlie4/results/GI1_T/STAR2p/GI1_T_p2.bam
+ output_prefix=/vf/users/CCBR/charlie_test_wil/charlie4/results/GI1_T/STAR2p/GI1_T_p2.
+ '[' SE == PE ']'
++ zcat /vf/users/CCBR/charlie_test_wil/charlie4/results/GI1_T/trim/GI1_T.R1.trim.fastq.gz
++ awk -v maxlen=100 'NR%4==2 {if (length($1) > maxlen+0) maxlen=length($1)}; END {print maxlen-1}'
+ overhang=150
+ echo 'sjdbOverhang for STAR: 150'
+ cd /vf/users/CCBR/charlie_test_wil/charlie4/results/GI1_T/STAR2p
+ STAR --genomeDir /vf/users/CCBR/charlie_test_wil/charlie4/ref/STAR_no_GTF --outSAMstrandField None --outFilterType BySJout --outFilterMultimapNmax 20 --outSJfilterOverhangMin 15 15 15 15 --alignSJoverhangMin 15 --alignSJDBoverhangMin 15 --seedSearchStartLmax 30 --outFilterScoreMin 1 --outFilterMatchNmin 1 --outFilterMismatchNmax 2 --outFilterMismatchNoverLmax 0.3 --alignIntronMin 20 --alignIntronMax 2000000 --alignMatesGapMax 2000000 --readFilesIn /vf/users/CCBR/charlie_test_wil/charlie4/results/GI1_T/trim/GI1_T.R1.trim.fastq.gz --readFilesCommand zcat --runThreadN 56 --outFileNamePrefix /vf/users/CCBR/charlie_test_wil/charlie4/results/GI1_T/STAR2p/GI1_T_p2. --sjdbFileChrStartEnd /vf/users/CCBR/charlie_test_wil/charlie4/results/pass1.out.tab --chimSegmentMin 15 --chimScoreMin 15 --chimJunctionOverhangMin 18 --chimScoreJunctionNonGTAG 0 --chimOutType Junctions WithinBAM --chimMultimapNmax 10 --limitSjdbInsertNsj 400000 --alignTranscriptsPerReadNmax 20000 --outSAMtype BAM Unsorted --alignEndsProtrude 10 ConcordantPair --outFilterIntronMotifs None --sjdbGTFfile /vf/users/CCBR/charlie_test_wil/charlie4/ref/ref.fixed.gtf --quantMode GeneCounts --outTmpDir /lscratch/40907635/2c4d0ede-8e82-4b06-a59a-8799954ddd04 --sjdbOverhang 150 --outBAMcompression 0 --outSAMattributes All

EXITING because of fatal ERROR: could not make run-time genome directory directory: /vf/users/CCBR/charlie_test_wil/charlie4/results/GI1_T/STAR2p/GI1_T_p2._STARgenome/
SOLUTION: please check the path and writing permissions

my log (success)

Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 56
Rules claiming more threads will be scaled down.
Provided resources: mem_mb=2653, mem_mib=2531, disk_mb=2653, disk_mib=2531
Select jobs to execute...

[Fri Nov 15 18:17:42 2024]
rule star2p:
    input: /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/trim/GI1_T.R1.trim.fastq.gz, /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/trim/GI1_T.R2.trim.fastq.gz, /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/ref/ref.fixed.gtf, /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/pass1.out.tab
    output: /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/STAR2p/GI1_T_p2.Chimeric.out.junction, /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/STAR2p/GI1_T_p2.Aligned.out.bam, /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/STAR2p/GI1_T_p2.bam, /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/STAR2p/GI1_T_p2.chimeric.bam, /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/STAR2p/GI1_T_p2.non_chimeric.bam, /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/STAR2p/GI1_T_p2.ReadsPerGene.out.tab
    jobid: 0
    reason: Forced execution
    wildcards: sample=GI1_T
    threads: 56
    resources: mem_mb=2653, mem_mib=2531, disk_mb=2653, disk_mib=2531, tmpdir=/tmp


set -exo pipefail

limitSjdbInsertNsj=$(wc -l /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/pass1.out.tab|awk '{print $1+1}')
if [ "$limitSjdbInsertNsj" -lt "400000" ];then limitSjdbInsertNsj="400000";fi

output_prefix=$(dirname /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/STAR2p/GI1_T_p2.bam)/GI1_T_p2.

if [ "SE" == "PE" ];then
# paired-end
    overhang=$(zcat /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/trim/GI1_T.R1.trim.fastq.gz /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/trim/GI1_T.R2.trim.fastq.gz | awk -v maxlen=100 'NR%4==2 {if (length($1) > maxlen+0) maxlen=length($1)}; END {print maxlen-1}')
    echo "sjdbOverhang for STAR: ${overhang}"
    cd /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/STAR2p
    STAR --genomeDir /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/ref/STAR_no_GTF \
    --outSAMstrandField None  \
    --outFilterType BySJout \
    --outFilterMultimapNmax 20 \
    --outSJfilterOverhangMin 15 15 15 15 \
    --alignSJoverhangMin 15 \
    --alignSJDBoverhangMin 15 \
    --outFilterScoreMin 1 \
    --outFilterMatchNmin 1 \
    --outFilterMismatchNmax 2 \
    --outFilterMismatchNoverLmax 0.3  \
    --alignIntronMin 20 \
    --alignIntronMax 2000000 \
    --alignMatesGapMax 2000000 \
    --readFilesIn /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/trim/GI1_T.R1.trim.fastq.gz /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/trim/GI1_T.R2.trim.fastq.gz \
    --readFilesCommand  zcat \
    --runThreadN 56 \
    --outFileNamePrefix $output_prefix \
    --sjdbFileChrStartEnd /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/pass1.out.tab \
    --chimSegmentMin 15 \
    --chimScoreMin 15 \
    --chimJunctionOverhangMin 18 \
    --chimScoreJunctionNonGTAG 0 \
    --chimOutType Junctions WithinBAM \
    --chimMultimapNmax 10 \
    --limitSjdbInsertNsj $limitSjdbInsertNsj \
    --alignTranscriptsPerReadNmax 20000 \
    --outSAMtype BAM Unsorted \
    --alignEndsProtrude 10 ConcordantPair \
    --outFilterIntronMotifs None \
    --sjdbGTFfile /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/ref/ref.fixed.gtf \
    --quantMode GeneCounts \
    --outTmpDir /lscratch/$SLURM_JOB_ID/0f59031f-b8dc-4bf9-89c4-93b66fa0785c \
    --sjdbOverhang $overhang \
    --outBAMcompression 0 \
    --outSAMattributes All

    rm -rf GI1_T_p2._STARgenome

else
#single-end
    overhang=$(zcat /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/trim/GI1_T.R1.trim.fastq.gz | awk -v maxlen=100 'NR%4==2 {if (length($1) > maxlen+0) maxlen=length($1)}; END {print maxlen-1}')
    echo "sjdbOverhang for STAR: ${overhang}"
    cd /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/STAR2p
    STAR --genomeDir /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/ref/STAR_no_GTF \
    --outSAMstrandField None  \
    --outFilterType BySJout \
    --outFilterMultimapNmax 20 \
    --outSJfilterOverhangMin 15 15 15 15 \
    --alignSJoverhangMin 15 \
    --alignSJDBoverhangMin 15 \
    --seedSearchStartLmax 30 \
    --outFilterScoreMin 1 \
    --outFilterMatchNmin 1 \
    --outFilterMismatchNmax 2 \
    --outFilterMismatchNoverLmax 0.3  \
    --alignIntronMin 20 \
    --alignIntronMax 2000000 \
    --alignMatesGapMax 2000000 \
    --readFilesIn /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/trim/GI1_T.R1.trim.fastq.gz \
    --readFilesCommand  zcat \
    --runThreadN 56 \
    --outFileNamePrefix $output_prefix \
    --sjdbFileChrStartEnd /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/pass1.out.tab \
    --chimSegmentMin 15 \
    --chimScoreMin 15 \
    --chimJunctionOverhangMin 18 \
    --chimScoreJunctionNonGTAG 0 \
    --chimOutType Junctions WithinBAM \
    --chimMultimapNmax 10 \
    --limitSjdbInsertNsj $limitSjdbInsertNsj \
    --alignTranscriptsPerReadNmax 20000 \
    --outSAMtype BAM Unsorted \
    --alignEndsProtrude 10 ConcordantPair \
    --outFilterIntronMotifs None \
    --sjdbGTFfile /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/ref/ref.fixed.gtf \
    --quantMode GeneCounts \
    --outTmpDir /lscratch/$SLURM_JOB_ID/0f59031f-b8dc-4bf9-89c4-93b66fa0785c \
    --sjdbOverhang $overhang \
    --outBAMcompression 0 \
    --outSAMattributes All

    rm -rf ${output_prefix}_STARgenome
fi
sleep 120
mkdir -p /lscratch/$SLURM_JOB_ID/0f59031f-b8dc-4bf9-89c4-93b66fa0785c
samtools view -H /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/STAR2p/GI1_T_p2.Aligned.out.bam > /lscratch/$SLURM_JOB_ID/0f59031f-b8dc-4bf9-89c4-93b66fa0785c/GI1_T_p2.non_chimeric.sam
cp /lscratch/$SLURM_JOB_ID/0f59031f-b8dc-4bf9-89c4-93b66fa0785c/GI1_T_p2.non_chimeric.sam /lscratch/$SLURM_JOB_ID/0f59031f-b8dc-4bf9-89c4-93b66fa0785c/GI1_T_p2.chimeric.sam
# ref https://github.com/alexdobin/STAR/issues/678
samtools view -@ 56 /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/STAR2p/GI1_T_p2.Aligned.out.bam | grep "ch:A:1" >> /lscratch/$SLURM_JOB_ID/0f59031f-b8dc-4bf9-89c4-93b66fa0785c/GI1_T_p2.chimeric.sam
samtools view -@ 56 /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/STAR2p/GI1_T_p2.Aligned.out.bam | grep -v "ch:A:1" >> /lscratch/$SLURM_JOB_ID/0f59031f-b8dc-4bf9-89c4-93b66fa0785c/GI1_T_p2.non_chimeric.sam
ls -alrth
for i in 1 2 3;do
    mkdir -p /lscratch/$SLURM_JOB_ID/0f59031f-b8dc-4bf9-89c4-93b66fa0785c_${i}
done
samtools view -@ 56 -b -S /lscratch/$SLURM_JOB_ID/0f59031f-b8dc-4bf9-89c4-93b66fa0785c/GI1_T_p2.chimeric.sam | \
samtools sort \
    -l 9 \
    -T /lscratch/$SLURM_JOB_ID/0f59031f-b8dc-4bf9-89c4-93b66fa0785c_1 \
    --write-index \
    -@ 56 \
    --output-fmt BAM \
    -o /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/STAR2p/GI1_T_p2.chimeric.bam -
samtools view -@ 56 -b -S /lscratch/$SLURM_JOB_ID/0f59031f-b8dc-4bf9-89c4-93b66fa0785c/GI1_T_p2.non_chimeric.sam | \
samtools sort \
    -l 9 \
    -T /lscratch/$SLURM_JOB_ID/0f59031f-b8dc-4bf9-89c4-93b66fa0785c_2 \
    --write-index \
    -@ 56 \
    --output-fmt BAM \
    -o /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/STAR2p/GI1_T_p2.non_chimeric.bam -
samtools sort \
    -l 9 \
    -T /lscratch/$SLURM_JOB_ID/0f59031f-b8dc-4bf9-89c4-93b66fa0785c_3 \
    --write-index \
    -@ 56 \
    --output-fmt BAM \
    -o /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/STAR2p/GI1_T_p2.bam /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/STAR2p/GI1_T_p2.Aligned.out.bam

Activating singularity image /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/.snakemake/singularity/3ce85b29f85e5d3fe17c6af860c5f059.simg
++ wc -l /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/pass1.out.tab
++ awk '{print $1+1}'
+ limitSjdbInsertNsj=2426
+ '[' 2426 -lt 400000 ']'
+ limitSjdbInsertNsj=400000
++ dirname /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/STAR2p/GI1_T_p2.bam
+ output_prefix=/gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/STAR2p/GI1_T_p2.
+ '[' SE == PE ']'
++ zcat /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/trim/GI1_T.R1.trim.fastq.gz
++ awk -v maxlen=100 'NR%4==2 {if (length($1) > maxlen+0) maxlen=length($1)}; END {print maxlen-1}'
+ overhang=150
+ echo 'sjdbOverhang for STAR: 150'
+ cd /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/STAR2p
+ STAR --genomeDir /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/ref/STAR_no_GTF --outSAMstrandField None --outFilterType BySJout --outFilterMultimapNmax 20 --outSJfilterOverhangMin 15 15 15 15 --alignSJoverhangMin 15 --alignSJDBoverhangMin 15 --seedSearchStartLmax 30 --outFilterScoreMin 1 --outFilterMatchNmin 1 --outFilterMismatchNmax 2 --outFilterMismatchNoverLmax 0.3 --alignIntronMin 20 --alignIntronMax 2000000 --alignMatesGapMax 2000000 --readFilesIn /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/trim/GI1_T.R1.trim.fastq.gz --readFilesCommand zcat --runThreadN 56 --outFileNamePrefix /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/STAR2p/GI1_T_p2. --sjdbFileChrStartEnd /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/pass1.out.tab --chimSegmentMin 15 --chimScoreMin 15 --chimJunctionOverhangMin 18 --chimScoreJunctionNonGTAG 0 --chimOutType Junctions WithinBAM --chimMultimapNmax 10 --limitSjdbInsertNsj 400000 --alignTranscriptsPerReadNmax 20000 --outSAMtype BAM Unsorted --alignEndsProtrude 10 ConcordantPair --outFilterIntronMotifs None --sjdbGTFfile /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/ref/ref.fixed.gtf --quantMode GeneCounts --outTmpDir /lscratch/40723983/0f59031f-b8dc-4bf9-89c4-93b66fa0785c --sjdbOverhang 150 --outBAMcompression 0 --outSAMattributes All
+ rm -rf /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/STAR2p/GI1_T_p2._STARgenome
+ sleep 120
+ mkdir -p /lscratch/40723983/0f59031f-b8dc-4bf9-89c4-93b66fa0785c
+ samtools view -H /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/STAR2p/GI1_T_p2.Aligned.out.bam
+ cp /lscratch/40723983/0f59031f-b8dc-4bf9-89c4-93b66fa0785c/GI1_T_p2.non_chimeric.sam /lscratch/40723983/0f59031f-b8dc-4bf9-89c4-93b66fa0785c/GI1_T_p2.chimeric.sam
+ samtools view -@ 56 /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/STAR2p/GI1_T_p2.Aligned.out.bam
+ grep ch:A:1
+ samtools view -@ 56 /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/STAR2p/GI1_T_p2.Aligned.out.bam
+ grep -v ch:A:1
+ ls -alrth
+ for i in 1 2 3
+ mkdir -p /lscratch/40723983/0f59031f-b8dc-4bf9-89c4-93b66fa0785c_1
+ for i in 1 2 3
+ mkdir -p /lscratch/40723983/0f59031f-b8dc-4bf9-89c4-93b66fa0785c_2
+ for i in 1 2 3
+ mkdir -p /lscratch/40723983/0f59031f-b8dc-4bf9-89c4-93b66fa0785c_3
+ samtools view -@ 56 -b -S /lscratch/40723983/0f59031f-b8dc-4bf9-89c4-93b66fa0785c/GI1_T_p2.chimeric.sam
+ samtools sort -l 9 -T /lscratch/40723983/0f59031f-b8dc-4bf9-89c4-93b66fa0785c_1 --write-index -@ 56 --output-fmt BAM -o /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/STAR2p/GI1_T_p2.chimeric.bam -
[bam_sort_core] merging from 0 files and 56 in-memory blocks...
+ samtools view -@ 56 -b -S /lscratch/40723983/0f59031f-b8dc-4bf9-89c4-93b66fa0785c/GI1_T_p2.non_chimeric.sam
+ samtools sort -l 9 -T /lscratch/40723983/0f59031f-b8dc-4bf9-89c4-93b66fa0785c_2 --write-index -@ 56 --output-fmt BAM -o /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/STAR2p/GI1_T_p2.non_chimeric.bam -
[bam_sort_core] merging from 0 files and 56 in-memory blocks...
+ samtools sort -l 9 -T /lscratch/40723983/0f59031f-b8dc-4bf9-89c4-93b66fa0785c_3 --write-index -@ 56 --output-fmt BAM -o /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/STAR2p/GI1_T_p2.bam /gpfs/gsfs12/users/sovacoolkl/charlie_test_dev/results/GI1_T/STAR2p/GI1_T_p2.Aligned.out.bam
[bam_sort_core] merging from 0 files and 56 in-memory blocks...
[Fri Nov 15 18:25:25 2024]
Finished job 0.
1 of 1 steps (100%) done
@kelly-sovacool kelly-sovacool added the bug Something isn't working label Nov 19, 2024
@kelly-sovacool
Copy link
Member Author

@kopardev do you have any idea why STAR would not be able to create a directory that Wilfried does have write access to? Here's the relevant part of the log file from above:

+ STAR --genomeDir /vf/users/CCBR/charlie_test_wil/charlie4/ref/STAR_no_GTF --outSAMstrandField None --outFilterType BySJout --outFilterMultimapNmax 20 --outSJfilterOverhangMin 15 15 15 15 --alignSJoverhangMin 15 --alignSJDBoverhangMin 15 --seedSearchStartLmax 30 --outFilterScoreMin 1 --outFilterMatchNmin 1 --outFilterMismatchNmax 2 --outFilterMismatchNoverLmax 0.3 --alignIntronMin 20 --alignIntronMax 2000000 --alignMatesGapMax 2000000 --readFilesIn /vf/users/CCBR/charlie_test_wil/charlie4/results/GI1_T/trim/GI1_T.R1.trim.fastq.gz --readFilesCommand zcat --runThreadN 56 --outFileNamePrefix /vf/users/CCBR/charlie_test_wil/charlie4/results/GI1_T/STAR2p/GI1_T_p2. --sjdbFileChrStartEnd /vf/users/CCBR/charlie_test_wil/charlie4/results/pass1.out.tab --chimSegmentMin 15 --chimScoreMin 15 --chimJunctionOverhangMin 18 --chimScoreJunctionNonGTAG 0 --chimOutType Junctions WithinBAM --chimMultimapNmax 10 --limitSjdbInsertNsj 400000 --alignTranscriptsPerReadNmax 20000 --outSAMtype BAM Unsorted --alignEndsProtrude 10 ConcordantPair --outFilterIntronMotifs None --sjdbGTFfile /vf/users/CCBR/charlie_test_wil/charlie4/ref/ref.fixed.gtf --quantMode GeneCounts --outTmpDir /lscratch/40907635/2c4d0ede-8e82-4b06-a59a-8799954ddd04 --sjdbOverhang 150 --outBAMcompression 0 --outSAMattributes All

EXITING because of fatal ERROR: could not make run-time genome directory directory: /vf/users/CCBR/charlie_test_wil/charlie4/results/GI1_T/STAR2p/GI1_T_p2._STARgenome/
SOLUTION: please check the path and writing permissions

@kelly-sovacool
Copy link
Member Author

this failed three times on the tumor sample but succeeded first try on the normal sample. likely a weird biowulf filesystem issue?

@kelly-sovacool
Copy link
Member Author

it succeeded on Wilfried's re-run. I'm chalking this up to an intermittent filesystem problem.

@kelly-sovacool kelly-sovacool closed this as not planned Won't fix, can't repro, duplicate, stale Nov 22, 2024
@kelly-sovacool kelly-sovacool added this to the _not-planned milestone Nov 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
_not-planned
Development

No branches or pull requests
1 participant
@kelly-sovacool