Skip to content

How to use the Pipeline

Jump to bottom
Simon Torres edited this page May 30, 2017 · 9 revisions

Introduction

The main goal of the Goodman Pipelines is to have a tool that let us, without too much effort, have reduced data very quick enough. At the Final version we plane to have the following.

  • Basic CCD Reduction (bias, flat, etc)
  • Spectrum Identification
  • Spectrum Tracing
  • Spectrum Extraction (sum within aperture and background subtraction)
  • Optimal Extraction
  • Wavelength Calibration (Interactive and Automatic)
  • Flux Calibration

The ticked options are already implemented and the others are planned to be complete for the version 1.0 release.

Operation

The pipeline is split in two parts, redccd and redspec (short for reduce CCD and reduce spectrum). They have to be run in sequence, there is a plan to create a single one that can handle either spectra or images.

For help use the argument -h or --help.

    $ redccd --help
    .
    # or
    .
    $ redspec --help

REDCCD

    $ redccd -h
    usage: redccd [-h] [-c] [--ignore-bias] [--auto-clean] [--saturation <Value>]
                  [--raw-path raw_path] [--red-path red_path] [--debug]    
                  [--log-to-file] [--flat-normalize <Normalization Method>]
                  [--flat-norm-order <Order>]
    
    PyGoodman CCD Reduction - CCD reductions for Goodman spectroscopic data
    
    optional arguments:
      -h, --help            show this help message and exit
      -c, --cosmic          Clean cosmic rays from science data.
      --ignore-bias         Ignore bias correction
      --auto-clean          Automatically clean reduced data directory
      --saturation <Value>  Saturation limit. Default to 55.000 ADU (counts)
      --raw-path raw_path   Path to raw data.
      --red-path red_path   Path to reduced data.
      --debug               Show detailed information of the process.
      --log-to-file         Write log to a file.
      --flat-normalize <Normalization Method>
                            Chose a method to normalize the flat for spectroscoy.
                            Choices are: mean, simple (model) and full (fits model
                            to each line).
      --flat-norm-order <Order>
                            Defines the order of the model to be fitted.

You can simply run redccd alone and it will work with certain defaults.

    --cosmic              False
    --ignore-bias         False
    --auto-clean          False
    --debug               False
    --log-to-file         False
    --saturation          55000
    --raw-path            ./
    --red-path            ./RED/
    --flat-normalize      simple
    --flat-norm-order     15

So, --raw-path is the directory where your data is located and if you don't put any --red-path it will try to use $(--raw-path)/RED

REDSPEC

    $ redspec -h
    usage: redspec [-h] [-p <Source Path>] [-d <Destination Path>]
                   [-s <Search Pattern>] [-m <Processing Mode>]
                   [-r <Reference Lamp>] [-l <Lamp File>] [-o <Out Prefix>]
                   [-R <Reference Dir>] [-i] [--debug] [--log-to-file]
    
    Extracts goodman spectra and does wavelength calibration.
    
    Supported Processing Modes are:
        <0>: (Default) reads lamps taken at the begining or end of the night.
        <1>: one or more lamps around science exposure.
    
    optional arguments:
      -h, --help            show this help message and exit
      -p <Source Path>, --data-path <Source Path>
                            Path for location of raw data. Default <./>
      -d <Destination Path>, --proc-path <Destination Path>
                            Path for destination of processed data. Default <./>
      -s <Search Pattern>, --search-pattern <Search Pattern>
                            Pattern for matching the goodman's reduced data.
      -m <Processing Mode>, --proc-mode <Processing Mode>
                            Defines the mode of matching lamps to science targets.
      -r <Reference Lamp>, --reference-lamp <Reference Lamp>
                            Name of reference lamp file for mode 0. If not
                            present, the first one in the list will be selected
      -l <Lamp File>        Name of an ASCII file describing which science target
                            uses which lamp. default <lamp.txt>
      -o <Out Prefix>, --output-prefix <Out Prefix>
                            Prefix to add to calibrated spectrum.
      -R <Reference Dir>, --reference-files <Reference Dir>
                            Directory of Reference files location
      -i, --non-interactive
                            Interactive wavelength solution. Enabled by default.
      --debug               Debugging Mode
      --log-to-file         Write log to a file

Again you can run it alone, i.e. without any arguments but most likely you will have to use --search-pattern because it will depend on the previous step.

    --data-path         ./
    --proc-path         ./
    --search-pattern    cfzsto
    --proc-mode         0
    --reference-lamp    (empty string)
    --lamp-file         lamps.txt
    --output-prefix     g
    --reference-files   refdata
    --interactive       False
    --debug             False
    --log-to-file       False

Reducing your Data

Since a Pipeline or Data Reduction Software can cover a limited number of scenarios it is absolutely necessary to do some preparation before. Our idea is to evolve this software in a way that can be used as a library to allow to reduce multiple instrument configurations and cases but for now, read on.

Preparing your folder

I would recommend running the pipeline from the same directory where your data is but you can certainly run from anywhere using the appropriate keyword telling where to look for data. For ccdred however you can run it from a directory containing as many sub directories as you want. You can mix imaging with spectroscopy but not within a single sub directory. Regarding data you can have as many configurations as you want but is important not to mix ROIs CCD readout modes/speeds

Data content for Imaging

You need the following OBSTYPE data.

  • BIAS (at least three, recommended 10 or more)
  • Flats (at least three)
  • Objects

Bias are absolutely necessary since there is no overscan region in Imaging. And (quite obviously) you need the flats and objects as well.

It is possible to mix different Imaging configurations as well.

DO NOT MIX ROI NOR CCD READOUT MODES

Data content for Spectroscopy

For spectroscopy there is more variety. but the same principles apply.

The OBSTYPE needed for processing spectroscopic data are:

  • BIAS
  • FLAT with grating
  • FLAT without grating
  • OBJECT
  • COMP

Again, you can mix (optical) instrument configurations but not ROI nor CCD readout.

Run redccd

You can simply run redccd and it should work fine with the defaults, it does most of the cases. If it doesn't check what variables you need to modify in order to get the best results. Most likely you will want to use --saturation to change the saturation level --cosmic to activate Cosmic Ray Rejection (uses LACosmic) and a very useful one, --auto-clean if you are running the pipeline a second time and there are files left in the destination folder. So here is one important point: Your raw data folder can not be the same as the processed data folder or in other words the processed data must go to a different folder.

Check if successful

So, how do I know if the execution of the pipeline was a success?. First of all the obvious way. If the program ended with no error is already a good sign. Then you should look into the reduced data folder, default is [raw data path]/RED/ and look for certain files.

  • Master Bias master_bias.fits or master_bias_N.fits in case there are more than one. N is an integer that counts them.

  • Master Flats. There is a naming convention for master flats. Then they are easy to catch for reuse. All of them start with master_flat of course, for imaging and for spectroscopy.

    Imaging flat naming convention

    master_flat_name += '_' + flat_filter + dome_sky + '.fits'

    Here master_flat_name = 'master_flat', flat_filter is the name of the filter used for imaging. dome_sky is either of three options. 'dome' for domeflats taken before sunset, 'sky' for domeflats taken between sunset and end of twilight and finally 'night' for domeflats taken after afternoon twilight and before morning twilight.

    Spectroscopy flat naming convention

     master_flat_name += target_name + flat_grating + wavmode + filter2 + '_' + flat_slit + dome_sky + '.fits'

    Here master_flat_name = 'master_flat', target_name is optional. Sets of flats taken in the afternoon or isolated in the night do not take the target_name value. flat_grating is the grating frequency. wavmode is the wavelength mode, a set of predefined angles for grating an camera in Goodman. filter2 is the order blocking filter that usually resides in the second filter wheel. If there is no filter it will be set to empty string. flat_slit is the width of the slit in arcseconds with two decimal points. dome_sky is either of three options which depends on the time of the day at which the set of flats was taken. 'dome' for before sunset, 'sky' for flats taken between sunset and the end of the twilight and finally 'night' for domeflats taken after the end of the twilight. Examples:

    master_flat_400m2_GG455_1.03_dome.fits
master_flat_Feige110_2100Custom_650nm_1.03_night.fits

    After a master flat is normalized the prefix norm_ is added.

  • Science and other images. For science images and others (COMP for instance) the original name of the image is maintained plus a prefix in which every letter will represent a process

    Letter Meaning
    c Cosmic Ray Rejection
    f Flat fielded
    z Bias corrected
    s Slit section trimmed
    t Initial trim section applied
    o Overscan Correction

    For example:

    # Spectroscopy
cfzsto_0281_HILT600_2100.fits
cfzsto_0285_comp_2100.fits
cfzsto_0296_CVSO166_400m2_gg455.fits
# Imaging 
cfzt_0191_SA98_618.fits 
cfzt_0197_SA98_618.fits
cfzt_0203_SA98_618.fits

Run redspec

(will complete this after I have reduced the number of bugs on redccd)

redccd will add the prefix g to the filenames.

Goodman Data Reduction Pipeline WIKI

Goodman Pipeline

Clone this wiki locally