Skip to content

Latest commit

 

History

History
189 lines (152 loc) · 9.89 KB

README.md

File metadata and controls

189 lines (152 loc) · 9.89 KB
Raw

CPTAC3 data catalog

Tables listing CPTAC3 data at GDC and Ding Lab genomic processing results at DCC, as well as details about downloaded data

Overview

Description of files in this project

  • CPTAC3.cases.dat: All known cases associated with CPTAC3 project. This is the master list
    • This also defines the cohort (discovery or confirmatory) and batch of each case. Note that each case may be in multiple batches
  • CPTAC3.Catalog.dat: Details about all sequence data (WGS, WXS, RNA-Seq, miRNA-Seq, Methylation Array, Targeted Sequencing) at GDC associated with all known cases
    • Note that this was previously called CPTAC3.AR.dat
  • CPTAC3.Demographics.dat: Demographic information associated with all known cases
  • CPTAC3.Catalog.Summary.txt: Summary of files available for each case on GDC.
  • ./BamMap - has details about GDC data downloaded to Ding Lab
    • *.BamMap.dat: "BamMap" files for various systems indicating locations of downloaded hg19, hg38, and FASTQ sequence data
    • *.BamMap-summary.txt - summary of files available on a given system as well as GDC.
      • For given system (e.g., katmai), format is similar to CPTAC3.file-summary.txt, except that upper-case symbol indicates presence on given system and lower-case symbol indicates that that sample is in GDC but not on system
  • ./DCC_Analysis_Summary - has details about analyses uploaded to DCC

CPTAC3 Catalog v2.0 update - Nov 14 2019

Files CPTAC3.Catalog.dat and CPTAC3.Catalog.Summary.dat are v2.0 release of CPTAC3 discovery as generated by CPTAC3.case.discover. These replace CPTAC3.AR.dat and CPTAC3.file-summary.txt and have small format differences. New features include,

  • Methylation array support
  • Targeted sequencing support
  • Aliquot information replaces sample information
  • Added column 10, result_type, and shifted remaining columns to right. This column codes for two distinct things:
    • For Methylation Array data, it is the channel (Green or Red)
    • For RNA-Seq harmonized BAMs, it is the result type, with values of genomic, chimeric, transcriptome
  • Added sample type column with long sample type names
  • AR file renamed Catalog file
  • file_summary file renamed Catalog.Summary

File Details

CPTAC3.Catalog.dat

List of all WGS, WXS, RNA-Seq, miRNA-Seq, Targeted Sequencing, Methylation Array data available at GDC. Generated by CPATC3 Case Discover.

Catalog file columns:

  • sample_name - ad hoc name for this file, generated for convenience and consistency.
  • case
  • disease
  • experimental_strategy - WGS, WXS, RNA-Seq, miRNA-Seq, Methylation Array, Targeted Sequencing
  • short_sample_type - short name for sample_type: blood_normal, tissue_normal, tumor, buccal_normal, tumor_bone_marrow, tumor_peripheral_blood
  • aliquot - name of aliquot used
  • filename
  • filesize
  • data_format - BAM, FASTQ, IDAT
  • result_type - ad hoc value specific to sample type
    • "chimeric", "genomic", "transcriptome" for RNA-Seq BAMs,
    • "Red" or "Green" for Methylation Array
    • "NA" otherwise
  • UUID
  • MD5
  • reference - best guess at reference of aligned sequence data. Note that these assumptions may not hold in future
    • hg19 for submitted aligned reads
      • hg38 for miRNA submitted reads
    • NA for submitted unaligned reads
    • hg38 for harmonized reads
  • sample_type - sample type as reported from GDC, e.g., Blood Derived Normal, Solid Tissue Normal, Primary Tumor, and others

Specific changes associated with v2.0

  • Added experimental_strategies "MethArray" and "Targeted Sequencing"
  • column "sample_type" renamed "short_sample_name", contents unchanged
  • Column "samples" replaced with "aliquot", and reports aliquot submitter ID associated with each sample
  • New column 13 : "sample_type". Reports verbatim GDC sample_type associated with sample
  • Added column 10, result_type, and shifted remaining columns to right. This column codes for two distinct things:
    • For Methylation Array data, it is the channel (Green or Red)
    • For RNA-Seq harmonized BAMs, it is the result type, with values of genomic, chimeric, transcriptome

CPTAC3.Catalog.Summary.dat

Catalog summary files provide a one-line representation of data available for a given case on GDC. Following case and disease, each column represents a particular data type, and one-letter codes T, N, A indicate availability of tumor, blood normal, and tissue adjacent normal samples, respectively. Repeated codes indicate repeated data files.

Lists counts of tumor (T), blood normal (N), and adjacent / tissue normal (A) for each of

  • WGS.hg19 - WGS data as submitted to GDC, assumed hg19
  • WXS.hg19 - WXS (aka WES, exome) data as submitted to GDC, assumed hg19
  • RNA.fq - RNA-Seq data as submitted to GDC, FASTQ format
    • R1 and R2 FASTQs are listed individually, so will typically have two of each sample type
  • miRNA.fq - miRNA-Seq data as submitted to GDC, BAM format
  • WGS.hg38 - Harmonized WGS data
  • WXS.hg38 - Harmonized WXS data
  • RNA.hg38 - Harmonized RNA-Seq data
    • Harmonization generates chimeric, genomic, and transcriptome BAM files, so each entry will have 3 of each sample type
  • miRNA.hg38 - Harmonized miRNA-Seq data
  • MethArray - Methylation Array data

Example:

C3L-00001   LUAD        WGS.hg19 T N A      WXS.hg19 T N A      RNA.fq TT  AA       miRNA.fq T  A       WGS.hg38 T N A      WXS.hg38 T N A      RNA.hg38 TTT  AAA       miRNA.hg38 T  A     MethArray TT  AA

This line indicates that LUAD case C3L-00001 has tumor, blood normal, and adjacent normal samples for WGS and WXS data as submitted (hg19); tumor and adjacent normal RNA-Seq data (TT, AA because FASTQ data comes in pairs); and tumor and adjacent miRNA data in FASTQ format. All these are available as harmonized hg38 WGS and WXS, and harmonized hg38 RNA-Seq chimeric, genomic, and transcriptome BAMs are available for tumor and adjacent normal. Methylation array data for tumor and tissue adjacent also available (Green and Red channel for each).

CPTAC3.cases.dat

Comprehensive list of cases along with their disease, cohort, and batch information.

  • Current cases list consists of 3696 cases and their disease.
    • Obtained from file Batches1through9_samples_attribute_tumorcode_added.xlsx, personal communication with Mathangi Thiagarajan
  • Cohort is an ad hoc column which tries to categorize cases according to Discovery or Confirmatory cohort, per year of contract.
  • Batch column indicates the year and batch(es) in which each case was processed. Y1 and Y2 correspond to Year 1 and 2, respectively. Note that a given case may belong to several different batches, since not all data for a given case was available at a given time. Such batches are listed as comma-separated names. In the future batch information should be indicated in a different file.

DCC_Analysis_Summary directory

Files here track analyses uploaded to DCC, with one file per analysis pipeline. See DCC_Analysis_Summary/README.md for additional details.

BamMap directory

Contents of ./BamMap directory track in-house data downloaded to Ding Lab servers from GDC. These change frequently and are specific to Ding Lab systems.

Example line from BamMap with header names:

     1  sample_name   C3L-00017.RNA-Seq.R1.A
     2  case    C3L-00017
     3  disease PDA
     4  experimental_strategy   RNA-Seq
     5  sample_type tissue_normal
     6  data_path   /gscmnt/gc2521/dinglab/mwyczalk/somatic-wrapper-data/GDC_import/data/7829f978-5fd7-436a-9ec2-2e58a7bcb1f7/180508_UNC31-K00269_0127_AHTV7YBBXX_ACTGAT_S59_L007_R1_001.fastq.gz
     7  filesize    4470007340
     8  data_format FASTQ
     9  reference   NA
    10  UUID    7829f978-5fd7-436a-9ec2-2e58a7bcb1f7
    11  system  MGI

BamMap summary

As an example from MGI.BamMap-summary.txt:

CCRCC	    WGS.hg19 t n a	    WXS.hg19 t n a	    RNA.fq TT  AA	    miRNA.fq t  a	    WGS.hg38 T N a	    WXS.hg38 T N A	    RNA.hg38 Ttt  Aaa

This indicates that all RNA-Seq FASTQ, harmonized WGS tumor and blood normal, all harmonized WXS, and genomic hg38 RNA-Seq data are available at MGI. Lower case letters indicate which data are available at GDC but not at MGI.

NOTE BamMap summary files are not updated regularly and are considered deprectated.

Custom sample names

SampleRename.dat is a TSV file used to add suffixes based on matches to UUID, aliquot, and experimental strategy. Input TSV file format is one of the following: a) uuid, suffix b) aliquot, experimental_strategy, suffix * The wildcard * will be used to indicate all experimental strategies multiple matches will give multiple sequential suffixes It is parsed by CPTAC3 Case Discover src/make_catalog.sh

Currently, it is used to add suffixes .core and .high_cov to select PDA samples.

  • Aliquots associated with core biopsies are obtained from
    • DeepCoverage_Broad_PDA.xlsx for WXS
    • PDA_Bulk_WGS_2_13.xlsx for WGS
  • Aliquots for high coverage WXS samples are obtained from
    • DeepCoverage_Broad_PDA.xlsx
    • CPTAC_SupplementalData_WGS&WES_renamingneeded_Aug2019_93samples_check.xlsx All files from Mathangi Thiagarajan and Ana Robles

DO_NOT_USE sample names

A number of samples are marked with a suffix .DO_NOT_USE. These labels are based on an analysis of duplicate aliquots, and correspond largely to instances where one aliquot has been superceded by another, with the original not removed from GDC. Details of this analysis can be found on shiso:/Users/mwyczalk/Projects/CPTAC3/CPTAC3.Cases/20200501.find_duplicates/README.md

Contact

Matthew Wyczalkowski [email protected] Ding Lab Washinton University School of Medicine