Use external assemblies

[SilasK]

Several user asked for this

Currently, I am working on a rather difficult dataset in terms of library size. I was therefore forced to assemble the samples individually, rather than pooling all reads first. I now have megahit-assembled meta-assemblies per sample and I was wondering if there is a simple way to now use your pipeline for the downstream steps. Could you maybe advise me how it is best to use your pipeline from that point onwards in terms of what files and directory structure is needed?

In essence I have now the following files:

1. QC and host-filtered reads
2. megahit assembly per sample
3. Then I have standard downstream results, like binning and annotations, but I would like to try your downstream pipeline.

[SilasK]

Yes, what you want can be done. Other users were asking the same. So what you can do is to init a new project using your qc fastq files

atlas init --skip-qc --db-dir /path/to/databse_location --assembler megahit path/to/fastq

Then, you should copy your assemblies into the working directory {sample}/assembly/{sample}_raw_contigs.fasta for each sample.

Run run atlas run assembly
Atlas should continue from there without assembling again and rename the assemblies.
Maybe try with one sample and do always a dryrun to see if atlas does what you want.

• I think I should add an option to provide assemblies in the sample table.

If you have difficult samples to assemble also have a look at #375

This is also the answer I gave @ slambrechts Did it worked for you?

[SilasK]

After some testing, we managed to make it run.

What need to be done is to touch finished_QC in order to tell atlas that you don't want to do any of the QC steps.

atlas states that it will do a rule named assembly which is not the assembly per see but only a target rule to gather all rules for the finalized assembly. If you don't have run_spadesor run_megahityou are fine.

Is importing the bam files possible?
In theory yes, but as the contigs become renamed.

[SilasK]

For @masun and others.

I suggest to

1. init an atlas project with your QC reads.
2. Run atlas qc which creates some stats and a QC report.
3. Copy your assemblies in {sample}/assembly/{sample}_raw_contigs.fasta for each sample
4. Run atlas run assembly

Atlas should continue from there without assembling again and rename the assemblies.
atlas states that it will do a rule named assembly which is not the assembly per see but only a target rule to gather all rules for the finalized assembly. If you don't have run_spades or run_megahit you are fine.

5. This creates also an assembly report, and statistics.
6. Then you can go on to run atlas run all or atlas run binning

touch finished_QCor touch finished_assembly shouldn't be necessary but can be used to tell atlas not to do the step.

[slambrechts]

I'm about to try this.

1. init an atlas project with your QC reads.
2. Run atlas qc which creates some stats and a QC report.

So running atlas qc will not actually do QC in this case? How does atlas know QC has already been done? Or do you have to copy your QC reads in the QC folder after you atlas init with the qc reads?

[SilasK]

@slambrechts You don't want to add external assemblies only qc reads.
Then it's simple as

atlas init --skip-qc --db-dir /path/to/databse_location path/to/fastqs
atlas run qc --dryrun

... check that atlas only does the stats and no nit_rawfor example.
Run
atlas run qc

[slambrechts]

I do want to add external assemblies, sorry for the confusion.

So when running

atlas init --skip-qc --db-dir $VSC_SCRATCH_VO/vsc42339/MICROBIAN/atlas/DB --assembler megahit $VSC_SCRATCH_VO/vsc42339/MICROBIAN/CLEAN_READS
 
atlas run qc --dryrun --resources mem=125 --jobs 48 -w $VSC_SCRATCH_VO/vsc42339/MICROBIAN/CLEAN_READS

I get:

Job counts:
        count   jobs
        1       build_qc_report
        43      calculate_insert_size
        1       combine_insert_stats
        1       combine_read_counts
        1       combine_read_length_stats
        43      finalize_sample_qc
        43      get_read_stats
        43      initialize_qc
        1       qc
        43      write_read_counts
        220
This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.

Does this mean atlas will only do the stats and create a QC report?

[SilasK]

seems alright go ahead

[slambrechts]

Ok this worked perfectly to create the QC stats and report without actually doing QC! However, I do get run_megahit in the next step.

I did:

atlas init --skip-qc --db-dir $VSC_SCRATCH_VO/vsc42339/MICROBIAN/atlas/DB --assembler megahit $VSC_SCRATCH_VO/vsc42339/MICROBIAN/CLEAN_READS
 
atlas run qc --resources mem=125 --jobs 48 --cores 48 -w $VSC_SCRATCH_VO/vsc42339/MICROBIAN/CLEAN_READS

I copied the assemblies in {sample}/assembly/{sample}_raw_contigs.fasta for each sample

atlas run assembly --dryrun --resources mem=125 --jobs 48 --cores 48 -w $VSC_SCRATCH_VO/vsc42339/MICROBIAN/CLEAN_READS

And got:

Job counts:
        count   jobs
        43      align_reads_to_final_contigs
        43      align_reads_to_prefilter_contigs
        1       assembly
        43      assembly_one_sample
        43      bam_2_sam_contigs
        1       build_assembly_report
        86      calculate_contigs_stats
        1       combine_contig_stats
        43      convert_sam_to_bam
        43      error_correction
        43      filter_by_coverage
        43      finalize_contigs
        43      get_contigs_from_gene_names
        43      init_pre_assembly_processing
        43      merge_pairs
        43      pileup
        43      pileup_prefilter
        43      predict_genes
        43      rename_contigs
        43      rename_megahit_output
        43      run_megahit
        820

[SilasK]

Hmm,

Could it be that the copied assemblies have an older timestamp than the qc reads?

check with ls -lh . touch */assembly/*_raw_contigs.fasta should update the timestamp.

Could you rerun the dryrun with --reason and send me the full stdout as a file?

[botellaflotante]

I had a similar problem which I solved by removing ref, logs and reports folders (or changing their names), also removing sequence_quality_control/reads_stats if you moved your sequences, but I am not sure it is the same.. you can try and then run "atlas run binning -n" to see if the rules list get smaller...¿?

[slambrechts]

Updating the timestamp by doing touch */assembly/*_raw_contigs.fasta fixed the problem for me, thanks!

I did not run the dryrun with --reason, so I can not sent the full stdout