diff --git a/docs/astrocut/file_formats.rst b/docs/astrocut/file_formats.rst index 2f7b664c..5ca7fc0b 100644 --- a/docs/astrocut/file_formats.rst +++ b/docs/astrocut/file_formats.rst @@ -24,7 +24,7 @@ ORIGIN STScI/MAST DATE File creation date PROCVER Software version RA_OBJ Center coordinate right ascension (deg) -DEC_OBJ Center coordinat declination (deg) +DEC_OBJ Center coordinate declination (deg) CHECKSUM HDU checksum DATASUM Data unit checksum ========= =================================================== @@ -43,13 +43,14 @@ Cube Files ========== See the `TESS Science Data Products Description Document `__ -for detailed information on the TESS full-frame image file format. +for detailed information on the TESS full-frame image file format. See the `TESS Image CAlibrator Full Frame Images page `__ +for information on the format of the TICA HLSP full-frame images. PrimaryHDU (Extension 0) ^^^^^^^^^^^^^^^^^^^^^^^^ -The Primary Header of the TESS cube fits file is the same as that from +The Primary Header of the TESS Mission, SPOC cube FITS file is the same as that from an individual FFI with the following exceptions: ========= =================================================== @@ -59,27 +60,44 @@ Keyword Value DATE Date the cube was created CAMERA From the ImageHDU (EXT 1) of an FFI CCD From the ImageHDU (EXT 1) of an FFI - Sector The TESS observing sector, passed by the user - DATE-OBS From the ImageHDU (EXT 1) of the sector's first FFI - DATE-END From the ImageHDU (EXT 1) of the sector's last FFI - TSTART From the ImageHDU (EXT 1) of the sector's first FFI - TSTOP From the ImageHDU (EXT 1) of the sector's last FFI + SECTOR The TESS observing Sector, passed by the user + DATE-OBS From the ImageHDU (EXT 1) of the Sector's first FFI + DATE-END From the ImageHDU (EXT 1) of the Sector's last FFI + TSTART From the ImageHDU (EXT 1) of the Sector's first FFI + TSTOP From the ImageHDU (EXT 1) of the Sector's last FFI +========= =================================================== + +The Primary Header of the TICA cube FITS file is the same as that from +an individual TICA FFI with the following exceptions: + +========= =================================================== +Keyword Value +========= =================================================== + ORIGIN STScI/MAST + DATE Date the cube was created + SECTOR The TESS observing Sector, passed by the user + STARTTJD From the PrimaryHDU (EXT 0) of the first TICA FFI in the Sector + MIDTJD From the PrimaryHDU (EXT 0) of the middle TICA FFI in the Sector + ENDTJD From the PrimaryHDU (EXT 0) of the last TICA FFI in the Sector + MJD-BEG From the PrimaryHDU (EXT 0) of the first TICA FFI in the Sector + MJD-END From the PrimaryHDU (EXT 0) of the last TICA FFI in the Sector ========= =================================================== ImageHDU (Extension 1) ^^^^^^^^^^^^^^^^^^^^^^ -The ImageHDU extension contains the TESS FFI datacube. -It is 4 diminsional, the two spacial dimensions, time, and data vs. -error flux values. Pixel valules are 32 bit floats. +The ImageHDU extension contains the TESS (or TICA) FFI data cube. +It is 4 dimensional, with two spatial dimensions, time, data and +error flux values. Note, error flux values are only included in the +cubes generated from SPOC products. Pixel values are 32 bit floats. The cube dimensions are ordered in the FITS format as follows: ========= =================================================== Keyword Value ========= =================================================== NAXIS 4 (number of array dimensions) -NAXIS1 2 (data value, error value) -NAXIS2 Total umber of FFis +NAXIS1 2 (For SPOC products, data value, error value) or 1 (For TICA products, data value only) +NAXIS2 Total number of FFIs NAXIS3 Length of first array dimension (NAXIS1 from FFIs) NAXIS4 Length of second array dimension (NAXIS2 from FFIs) ========= =================================================== @@ -88,27 +106,27 @@ NAXIS4 Length of second array dimension (NAXIS2 from FFIs) BinTableHDU (Extension 2) ^^^^^^^^^^^^^^^^^^^^^^^^^ -The BinTableHDU extension contains a table that holds all of the Image extension -header keywords from the individual FFIs. There is one column for each keyword -plus one additional column called "FFI_FILE" that contains FFI filename for each -row. Each column name keyword also has an entry in the extension header, with -the value being the keyword comment from the FFI header. This last allows the -FFI Image extension headers to be reacreated completely if desired. +The BinTableHDU extension, in both the SPOC and TICA cubes, contains a table that +holds all of the Image extension header keywords from the individual FFIs. There +is one column for each keyword plus one additional column called "FFI_FILE" that +contains FFI filename for each row. Each column name keyword also has an entry in the +Image extension header, with the value being the keyword value from the FFI header. +This last column allows the FFI Image extension headers to be recreated completely if desired. Target Pixel Files ================== The Astrocut target pixel file (TPF) format conforms as closely as possible to the -TESS mission TPFs. See the `TESS Science Data Products Description Document `__ -for detailed information on the TESS mission target pixel file format, here I -describe how Astrocut TPFs differ from mission pipeline TPFs. +TESS Mission TPFs. See the `TESS Science Data Products Description Document `__ +for detailed information on the TESS Mission TPF format, here it is +described how Astrocut TPFs differ from Mission pipeline TPFs. PRIMARY PrimaryHDU (Extension 0) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The Primary Header of an Astrocut TPF is the same as that from -a mission TPF with the following exceptions: +a Mission TPF with the following exceptions: ========= ==================================================== Keyword Value @@ -138,18 +156,18 @@ TICID None PIXELS BinTableHDU (Extension 1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The Astrocut PIXELS BinTableHDU extension comprises the same columns as that from -the mission pipeline, with one addition. The extra column is ``FFI_FILE`` and contains -the name of the FFI file that that row's pixels come from. +The Astrocut PIXELS BinTableHDU comprises the same columns as those included in +the Mission pipeline TPFs, with one addition: an extra column, ``FFI_FILE``, contains +the name of the FFI file that the row's pixels come from. -While all of the columns present in mission pipeline TPFs are present, they do not all -contain data. The columns that are empty in Astrocut TPFs are: +While all of the columns present in Mission pipeline TPFs are present in cutouts created +from SPOC cubes, they do not all contain data. The columns that are empty in Astrocut SPOC TPFs are: ============ ==================================================== Column Value ============ ==================================================== CADENCENO 0 filled array in cutout shape -RAW_CNTS -1 filles array in cutout shape +RAW_CNTS -1 filled array in cutout shape FLUX_BKG 0 filled array in cutout shape FLUX_BKG_ERR 0 filled array in cutout shape POS_CORR1 0 @@ -158,7 +176,24 @@ POS_CORR2 0 The ``TIME`` column is formed by taking the average of the ``TSTART`` and ``TSTOP`` values from the corresponding FFI for each row. The ``QUALITY`` column is taken from the ``DQUALITY`` -image keyword in the individual FFI files. +image keyword in the individual SPOC FFI files. + +For cutouts created from TICA cubes, the ``TIMECORR`` column has been removed from the +PIXELS BinTableHDU. Similar to cutouts made from SPOC cubes, the other columns (aside from +the ``TIMECORR`` column) present in Mission pipeline TPFs are present in cutouts created +from TICA cubes, but do not all contain data. The columns that are empty in Astrocut TICA TPFs are: + +============ ==================================================== +Column Value +============ ==================================================== +RAW_CNTS -1 filled array in cutout shape +FLUX_ERR 0 filled array in cutout shape +FLUX_BKG 0 filled array in cutout shape +FLUX_BKG_ERR 0 filled array in cutout shape +QUALITY 0 +POS_CORR1 0 +POS_CORR2 0 +============ ==================================================== Three keywords have also been added to the PIXELS extension header to give additional information about the cutout world coordinate system (WCS). TESS FFIs are large and therefore are described @@ -176,10 +211,10 @@ for their purpose, and to retrieve the original WCS with distortion coefficients | WCS_FFI | | The name of the FFI file used to build the original WCS | | | | from which the cutout and cutout WCS were calculated. | +---------+----------------------------------------------------------------+ -| WCS_MSEP| | The maximum separation in degrees between the cutout’s | -| | | linear WCS and the FFI’s full WCS. | +| WCS_MSEP| | The maximum separation in degrees between the cutout's | +| | | linear WCS and the FFI's full WCS. | +---------+----------------------------------------------------------------+ -| WCS_SIG | | The error in the cutout’s linear WCS, calculated as | +| WCS_SIG | | The error in the cutout's linear WCS, calculated as | | | | ``sqrt((dist(Po_ij, Pl_ij)^2)`` where ``dist(Po_ij, Pl_ij)`` | | | | is the angular distance in degrees between the sky position | | | | of of pixel i,j in the original full WCS and the new linear | @@ -190,10 +225,10 @@ for their purpose, and to retrieve the original WCS with distortion coefficients APERTURE ImageHDU (Extension 2) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The APERTURE ImageHDU extension is similar to that of mission pipeline TPFs, but contains -slightly different data. For mission pipeline files, the aperture image gives information about -each pixel, whether it was collected and what calculations it was used in. Because astrocut does -not do any of the more complex claculations used in the mission pipeline, each pixel in the +The APERTURE ImageHDU extension is similar to that of Mission pipeline TPFs, but contains +slightly different data. For Mission pipeline files, the aperture image gives information about +each pixel, whether it was collected and whether it was used in calculating e.g., the background flux. +Because Astrocut does not do any of the more complex calculations used in the Mission pipeline, each pixel in the aperture image will either be 1 (pixel was collected and contains data in the cutout) or 0 (pixel is off the edge of the detector and contains no data in the cutout). @@ -201,18 +236,18 @@ aperture image will either be 1 (pixel was collected and contains data in the cu Cosmic Ray Binary Table Extension ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -This extension is not present in Astrocut TPFs, although it is a part of mission pipeline TPFs. +This extension is not present in Astrocut TPFs, although it is a part of the Mission pipeline TPFs. Path Focused Target Pixel Files =============================== -When the `~astrocut.center_on_path` function is used to create cutout target pixel files (TPFs) -where the individualimage cutouts move along a path in time and space, the TPF format has to be -adjusted accordingly. It still conformes as closely as possible to the TESS mission pipeline TPF -file format, but differs in several cruicial ways. The `~astrocut.center_on_path` function works -on astrocut TPFs, so that is the baseline file format. I will describe here only the differences -between path focussed astrocut TPFs and regular astrocut TPFs (see `Target Pixel Files`_ for +When the `~astrocut.center_on_path` function is used to create cutout TPFs +where the individual image cutouts move along a path in time and space, the TPF format has to be +adjusted accordingly. It still conforms as closely as possible to the TESS Mission pipeline TPF +file format, but differs in several crucial ways. The `~astrocut.center_on_path` function works +on Astrocut TPFs, so that is the baseline file format. Only the differences +between path focused Astrocut TPFs and regular Astrocut TPFs are described here (see `Target Pixel Files`_ for regular Astrocut TPF format). PRIMARY PrimaryHDU (Extension 0) @@ -223,7 +258,7 @@ Additional or updated keywords: ========= ======================================================= Keyword Value ========= ======================================================= -DATE Set the the time the path focussed cutout was performed +DATE Set the the time the path focused cutout was performed OBJECT Moving target object name/identifier, only present if set by the user ========= ======================================================= @@ -263,7 +298,7 @@ The APERTURE extension may or may not be present in a path focussed TPF, to be p the user must have passed an FFI WCS object into the `~astrocut.center_on_path` function. The APERTURE ImageHDU extension of path focussed TPFs is very different from other -TESS TPFs. The aperture image, instead of being the size and shape of an individeual cutout, +TESS TPFs. The aperture image, instead of being the size and shape of an individual cutout, is the size of the full FFI image the cutouts were drawn from. All pixels used in any individual cutout are marked with 1, while the rest of the pixels are 0, so the entire trajectory of the cutout path is captured. Additionally the WCS information in the header diff --git a/docs/astrocut/index.rst b/docs/astrocut/index.rst index 3bfad1e3..aba32769 100644 --- a/docs/astrocut/index.rst +++ b/docs/astrocut/index.rst @@ -11,8 +11,8 @@ Astrocut provides tools for making cutouts from sets of astronomical images with Three main areas of functionality are included: -- Solving the specific problem of creating image cutouts from sectors of Transiting Exoplanet Survey Satellite (TESS) full-frame images. -- General fits file cutouts incuding from single images and sets of images with the shared WCS/pixel scale. +- Solving the specific problem of creating image cutouts from Sectors of Transiting Exoplanet Survey Satellite (TESS) full-frame images, and related High-Level Science Product images (TICA). +- General FITS file cutouts including from single images and sets of images with shared WCS/pixel scale. - Cutout post-processing functionality, including centering cutouts along a path (for moving targets) and combining cutouts. @@ -20,9 +20,9 @@ Three main areas of functionality are included: FITS file image cutouts ======================= -These functions provide general purpose astronomical cutout functionality on FITS files. +These functions provide general purpose astronomical cutout functionality for FITS files. There are two main cutout functions, `~astrocut.fits_cut` for creating cutout FITS files, -and `~astrocut.img_cut` for creating cutout jpg or png files. An image normalization +and `~astrocut.img_cut` for creating cutout JPG or PNG files. An image normalization (`~astrocut.normalize_img`) function is also available. Creating FITS cutouts @@ -31,9 +31,9 @@ Creating FITS cutouts The function `~astrocut.fits_cut` takes one or more FITS files and performs the same cutout on each, returning the result either in a single FITS file or as one FITS file per cutout. It is important to remember that while the expectation is that all input images are aligned -and have the same pixel scale, no checking is done. +and have the same pixel scale, no checking is done by Astrocut. -The cutout FITS file format is decribed `here `__. +The cutout FITS file format is described `here `__. .. code-block:: python @@ -89,9 +89,9 @@ Creating image cutouts ---------------------- The function `~astrocut.img_cut` takes one or more FITS files and performs the same cutout -on each, returning a single jpg or png file for each cutout. +on each, returning a single JPG or PNG file for each cutout. It is important to remember that while the expectation is that all input images are -aligned and have the same pixel scale, no checking is done. +aligned and have the same pixel scale, no checking is done by Astrocut. .. code-block:: python @@ -113,8 +113,8 @@ aligned and have the same pixel scale, no checking is done. .. image:: imgs/png_ex_cutout.png -Color images can also be produced using `~astrocut.img_cut` given three input files, which will be -treated as the R, G, and B channels respectively. +Color images can also be produced using `~astrocut.img_cut`, given three input files, which will be +treated as the R, G, and B channels, respectively. .. code-block:: python @@ -142,45 +142,58 @@ treated as the R, G, and B channels respectively. TESS Full-Frame Image Cutouts ============================= -There are two parts of the package involved in this task, the `~astrocut.CubeFactory` -class allows you to create a large image cube from a list of FFI files. +There are two parts of the package involved in creating cutouts from TESS full-frame images (FFIs). +First, the `~astrocut.CubeFactory` (if working with SPOC products, or `~astrocut.TicaCubeFactory` if working +with TICA FFIs) class allows you to create a large image cube from a list of FFI files. This is what allows the cutout operation to be performed efficiently. -The `~astrocut.CutoutFactory` class performs the actual cutout and builds -a target pixel file (TPF) that is compatible with TESS pipeline TPFs. - -The basic work-flow is to first create an image cube from individual FFI files -(this is one-time work), and then make individual cutout TPFs from this -large cube file. If you are doing a small number of cutouts, it may make -sense for you to use our tesscut web service: -`mast.stsci.edu/tesscut `_ +Next, the `~astrocut.CutoutFactory` class performs the actual cutout and builds +a target pixel file (TPF) that is similar to the TESS Mission-produced TPFs. + +The basic procedure is to first create an image cube from individual FFI files +(this only needs to be completed once per set of FFIs), and to then make individual cutout TPFs from this +large cube file for targets of interest. Note, you can only make cubes from a set of FFIs +with the same product type (i.e., only SPOC *or* only TICA products) that were observed in +the same Sector, camera, and CCD. +If you are creating a small number of cutouts, the TESSCut web service +may suit your needs: `mast.stsci.edu/tesscut `_ Making image cubes ------------------ -Making an image cube is a simple operation, but comes with an important -time/memory trade-off. - .. important:: - **Time/Memory Trade-off** + **Time-Memory Trade-off** + + Making an image cube is a simple operation, but comes with an important + time-memory trade-off. The ``max_memory`` argument determines the maximum memory in GB that will be used - for the image data cube while it is being built. This is *only* for the data cube, - and so is somewhat smaller than the amount of memory needed for the program to run. - Never set it to your system's total memory. - - Because of this, it is possible to build cube files with much less memory than will - hold the final product. However there is a large time trade-off, as the software must - run through the list of files multiple times instead of just once. The default value - of 50 GB was chosen because it comfortably fits a main mission sector of TESS FFIs, - with the default setting on a system with 65 GB of memory it takes about 15 min to - build a cube file. On a system with enough less memory that 3 passes through the - list of files are required this time rises to ~45 min. + for the image data cube while it is being built. This is the amount of memory required + *only* for the data cube, so is somewhat smaller than the total amount of memory needed + for the program to run. You should never set it to your system's total memory. + + Because of this, cube files do not need to allocate their total size in + memory all at once. Instead, a smaller memory allocation can be used while + the cube file is constructed; however, this will significantly increase the + execution time as bytes are swapped into and out of the memory allocation + being used. The default value of 50 GB was chosen because it fits all of the + TESS FFIs from a single Prime Mission Sector (Sectors 1-26); however, in the + current TESS Extended Mission 2, where 6 times more FFIs are observed per Sector + (compared to the number of FFIs observed per Sector in the Prime Mission), 50 GB + is not enough space to hold all of the FFIs in memory, and the cubes will be + written in multiple blocks. With the default settings, on a system with 64 GB of + memory, it takes about 3 hours to build a single cube file. On a system with less + memory or where ``max_memory`` is set to a value less than 50 GB, more passes + through the list of files are required, and the time to create a cube can increase + significantly. -By default `~astrocut.CubeFactory.make_cube` runs in verbose mode and prints out its progress, however setting -verbose to false will silence all output. +Assuming that you have set of calibrated TESS (or TICA) FFI files stored locally, you can +create a cube using the `~astrocut.CubeFactory.make_cube` method (or +`~astrocut.TicaCubeFactory.make_cube` for TICA products). By default, both `~astrocut.CubeFactory.make_cube` +and `~astrocut.TicaCubeFactory.make_cube` run in verbose mode and prints out progress; setting `verbose` to false will silence +all output. -The image cube file format is decribed `here `__. +The output image cube file format is described `here `__. .. code-block:: python @@ -218,17 +231,17 @@ The image cube file format is decribed `here `__. Making cutout target pixel files -------------------------------- -To make a cutout, you must already have an image cube to cut out from. +To make a cutout, you must already have an image cube from which to create the cutout. Assuming that you have a TESS cube file stored locally, you can give the central -coordinate and cutout size (in either pixels or angular `~astropy.Quantity`) +coordinate of your target of interest and cutout size (in either pixels or angular degrees/arcseconds `~astropy.Quantity`) to the `~astrocut.CutoutFactory.cube_cut` function. -You can either specify a target pixel file name, or it will be built as: +You can optionally specify an output TPF name; if no output name is provided, the file name will be built as: "____astrocut.fits". You can optionally -also specify a output path, the directory in which the target pixel file will -be saved, if unspecified it defaults to the current directory. +also specify an output path, the directory in which the TPF will +be saved; if unspecified, this will default to the current directory. -The cutout target pixel file format is decribed `here `__. +The cutout target pixel file format is described `here `__. .. code-block:: python @@ -260,15 +273,15 @@ The cutout target pixel file format is decribed `here `__. +The output target pixel file format is described `here `__. This example starts with a path, and uses several `TESScut services `__ to retrieve all of the inputs for the `~astrocut.center_on_path` function. We also use the helper function -`~astrocut.path_to_footprints` that takes in a path table, cutout size, and WCS object and returns the +`~astrocut.path_to_footprints` that takes in a path table, cutout size, and WCS object, and returns the cutout location/size(s) necesary to cover the entire path. .. code-block:: python @@ -405,20 +417,19 @@ cutout location/size(s) necesary to cover the entire path. Combining cutouts ----------------- -The `~astrocut.CutoutsComibner` class allows the user to take one or more Astrocut cutout +The `~astrocut.CutoutsCombiner` class allows the user to take one or more Astrocut cutout FITS files (as from `~astrocut.fits_cut`) with a shared WCS object, and combine them into -a single cutout. In practical terms this means that you should make the same cutout in the -all of the images you want to combine. +a single cutout. This means that you should request the same cutout size in all of the images you want to combine. -The default is to combine the images with a mean combiner such that every pixel is the mean of all -pixels that have data at that point. This combiner is made with the `~astrocut.build_default_combine_function` -which takes the input image huds and allows the user to specify a null data value (default is NaN). +The default setting combines the images with a mean combiner, such that every combined pixel is the mean of all +pixels that have data at that point. This mean combiner is made with the `~astrocut.build_default_combine_function`, +which takes the input image HDUs and allows the user to specify a null data value (default is NaN). Users can write a custom combiner function, either by directly setting the -`~astrocut.CutoutsComibner.combine_images` function, or by writing a custom combiner function builder -and passing it to the `~astrocut.CutoutsComibner.build_img_combiner` function. The main reason to -write a function builder is that the `~astrocut.CutoutsComibner.combine_images` function must work -*only* on the images being combines=d, any usage of header keywords for example, must be set in that +`~astrocut.CutoutsCombiner.combine_images` function, or by writing a custom combiner function builder +and passing it to the `~astrocut.CutoutsCombiner.build_img_combiner` function. The main reason to +write a function builder is that the `~astrocut.CutoutsCombiner.combine_images` function must work +*only* on the images being combined; any usage of header keywords, for example, must be set in that function. See the `~astrocut.build_default_combine_function` for an example of how this works.