From f67bf8b9d97e740d55e767b99e79594c44cda7be Mon Sep 17 00:00:00 2001 From: Steve Goldman <32876747+s-goldman@users.noreply.github.com> Date: Tue, 22 Oct 2024 11:49:39 -0400 Subject: [PATCH 1/9] initial commit. --- .../mast_data_products/hap-parameters.rst | 63 ++++++++++++++----- 1 file changed, 49 insertions(+), 14 deletions(-) diff --git a/doc/source/mast_data_products/hap-parameters.rst b/doc/source/mast_data_products/hap-parameters.rst index 9616323b2..d601e1806 100644 --- a/doc/source/mast_data_products/hap-parameters.rst +++ b/doc/source/mast_data_products/hap-parameters.rst @@ -2,11 +2,46 @@ HAP Parameters ----------------------------------- .. _hap-parameters: +The HAP parameter files are housed in the drizzlepac/pars/hap_pars directory. There are different json parameter files for the SVM and MVM products in separate directories. The svm_parameter json files are also used for pipeline products. + + +In either directory, there is a separate detector json file for each instrument/detector. Each file specifies additional json files used for different tasks and scenarios. The primary differences are in the case of different number of exposures: + + +Detector JSON Files +******************* + +acs_hrc_index.json + +.. code-block:: python + + { + "alignment": { + "all": "acs/hrc/acs_hrc_alignment_all.json" + }, + "astrodrizzle": { + "any_n1": "acs/hrc/acs_hrc_astrodrizzle_any_n1.json", + "acs_hrc_any_n2": "acs/hrc/acs_hrc_astrodrizzle_any_n2.json", + "acs_hrc_any_n4": "acs/hrc/acs_hrc_astrodrizzle_any_n4.json", + "acs_hrc_any_n6": "acs/hrc/acs_hrc_astrodrizzle_any_n6.json", + "filter_basic": "any/any_astrodrizzle_filter_hap_basic.json", + "single_basic": "any/any_astrodrizzle_single_hap_basic.json", + "total_basic": "acs/hrc/acs_hrc_astrodrizzle_any_total.json" + }, + "catalog generation": { + "all": "acs/hrc/acs_hrc_catalog_generation_all.json" + }, + "quality control": { + "all": "acs/hrc/acs_hrc_quality_control_all.json" + } + } + +The different files for "astrodrizzle" are for the case of 1 exposure ("any_n1"), 2 exposures (acs_hrc_any_n2), etc. Each json file with an associated task above (e.g. astrodrizzle) will have the parameters for that task. These files will be used to instantiate variables that are required in the processing of the data and, in the case of "astrodrizzle", are separated into different steps. Shown below are the parameters that are used by the Hubble Advanced Products. We include the parameter, the default value for WFC3 processing, and a description of that parameter. .. run_hap_processing -.. identified in json files. +.. identified in json files. Initialize HAP @@ -153,7 +188,7 @@ driz_sep_bits: int (*default="16"*) Integer sum of all the DQ bit values from the input image's DQ array that should be considered "good" when building the weighting mask. This can also be used to reset pixels to good if they had been flagged as cosmic rays during a previous run of ``AstroDrizzle``, by adding the value 4096 for ACS and WFPC2 data. Please see the section on Selecting the ``Bits`` Parameter for a more detailed discussion. driz_sep_kernel: str (*default="turbo"*) - Used for the initial separate drizzling operation only, this parameter specifies the form of the kernel function used to distribute flux onto the separate output images. The current options are: "square", "point", "turbo", "gaussian", and "lanczos3". The latter two options ("gaussian" and "lanczos3") are not guaranteed to conserve flux, but may produce reasonable results; understand the effects of these kernels before using them. A former option "tophat" has been removed as it was found to produce poor results. See adrizzle.help for more details. + Used for the initial separate drizzling operation only, this parameter specifies the form of the kernel function used to distribute flux onto the separate output images. The current options are: "square", "point", "turbo", "gaussian", and "lanczos3". The latter two options ("gaussian" and "lanczos3") are not guaranteed to conserve flux, but may produce reasonable results; understand the effects of these kernels before using them. A former option "tophat" has been removed as it was found to produce poor results. See adrizzle.help for more details. driz_sep_wt_scl: float (*default=exposure time (from image header)*) This parameter specifies the weighting factor for input image. If ``driz_sep_wt_scl``\ =\ ``exptime``, then the scaling value will be set equal to the exposure time found in the image header. The use of the default value is recommended for producing optimal behavior for most scenarious. It is possible to set ``wt_scl``\ =\ "expsq" for weighting by the square of the exposure time, which is optimal for read-noise dominated images. @@ -266,7 +301,7 @@ driz_cr: bool (*default=False*) driz_cr_snr: str (*default="5.0 4.0"*) Driz_cr.SNR parameter* - + driz_cr_grow: int (*default=1*) Driz_cr_grow parameter @@ -356,7 +391,7 @@ MAX_FIT_RMS: int (*default=20*) Not currently in use. MAX_SOURCES_PER_CHIP: int (*default=250*) - Not currently in use. + Not currently in use. run_align (*primarily in align.py*) @@ -378,19 +413,19 @@ MIN_CATALOG_THRESHOLD: int (*default=3*) Not currently in use. MIN_OBSERVABLE_THRESHOLD: int (*default=10*) - Not currently in use. The minimum number of observed sources required to continue fitting. If below this threshold, the code will return a status=1 and try with another catalog. + Not currently in use. The minimum number of observed sources required to continue fitting. If below this threshold, the code will return a status=1 and try with another catalog. MAX_FIT_LIMIT: int (*default=150*) Not currently in use. mosaic_catalog_list: list of strings (*default=["GAIAeDR3", "GSC242", "2MASS"]*) - List of available catalogs for aligning for both pipeline and SVM products. The code will go through each catalog in this order. + List of available catalogs for aligning for both pipeline and SVM products. The code will go through each catalog in this order. mosaic_fit_list: list of strings (*default=["match_relative_fit", "match_2dhist_fit", "match_default_fit"]*) List of available fit algorithms for aligning for both pipeline and SVM products; match_default_fit relative alignment without using 2dhist and different throusholds (see json configuration files). mosaic_fitgeom_list: dict (*default={"rshift": 10, "rscale": 10, "general": 6}*) - The different fit geometries tried in alignment as well as their minobj value which specifies the number of matched sources required for a successful fit. For pipeline products, the fitgeometry value is ignored and defaults to a fit geometry of ``rscale``. The fitgeom for the pipeline products is specified as a default in *align_utils.perform_fit*. The value for minobj specified here, however, is used for the pipeline products. + The different fit geometries tried in alignment as well as their minobj value which specifies the number of matched sources required for a successful fit. For pipeline products, the fitgeometry value is ignored and defaults to a fit geometry of ``rscale``. The fitgeom for the pipeline products is specified as a default in *align_utils.perform_fit*. The value for minobj specified here, however, is used for the pipeline products. fit quality categories """""""""""""""""""""" @@ -410,7 +445,7 @@ generate_source_catalogs (*primarily in align_utils.py*) """""""""""""""""""""""""""""""""""""""""""""""""""""""" box_size: int (*default=13*) - The size of the box used for calculating the 2D Background of the catalog "white light" image along each axis in pixels. In nothing is specified, a default of BKG_BOX_SIZE=27 is used. + The size of the box used for calculating the 2D Background of the catalog "white light" image along each axis in pixels. In nothing is specified, a default of BKG_BOX_SIZE=27 is used. win_size: int (*default=3*) The size of the 2D filter to apply to the background of the catalog "white light" image. If nothing is specified, a default of BKG_FILTER_SIZE=3 is used. @@ -440,7 +475,7 @@ classify: bool (*default=false*) Not currently in use as cosmic rays are being removed before segmentation. Specifies whether or not to apply photutils classification algorithm when evaluating each of the identified segments (sources) from the chip. threshold: float (*default=-1.1*) - Value from the image which serves as the limit for determining sources. If None, compute a default value of (background+5*rms(background)). If threshold < 0.0, use absolute value as scaling factor for default value. If nothing is specified a default value of None is used to compute the background. + Value from the image which serves as the limit for determining sources. If None, compute a default value of (background+5*rms(background)). If threshold < 0.0, use absolute value as scaling factor for default value. If nothing is specified a default value of None is used to compute the background. @@ -459,7 +494,7 @@ perform_fit (*primarily external in tweakwcs.matchutils.XYXYMatch*) For match_relative_fit, match_default_fit, and match_2dhist_fit, the following parameters are used: fitgeom": "rscale", - As used above, this is ignored for pipeline products. + As used above, this is ignored for pipeline products. searchrad: float (*default=125*) The search radius for a match (in units of the tangent plane). @@ -478,19 +513,19 @@ determine_fit_quality """"""""""""""""""""" MIN_CATALOG_THRESHOLD": int (*default=3*) - The minimum number of catalog sources required to continue fitting. If below this threshold, the code will return a fit_quality=5 and try with another catalog. + The minimum number of catalog sources required to continue fitting. If below this threshold, the code will return a fit_quality=5 and try with another catalog. MIN_OBSERVABLE_THRESHOLD": int (*default=4*) If the number of observed sources is below this threshold, the code ends alignment and defers to an *a priori* solution. MIN_CROSS_MATCHES": int (*default=3*) - Not currently in use. + Not currently in use. MIN_FIT_MATCHES": int (*default=4*) Not currently in use. MAX_FIT_RMS": float (*default=20*) - Not currently in use. Maximum RMS value for a fit to be considered good. Currently a warning is printed but nothing is done with this parameter. + Not currently in use. Maximum RMS value for a fit to be considered good. Currently a warning is printed but nothing is done with this parameter. MAX_FIT_LIMIT": int (*default=150*) The maximum allowable RMS value for a fit to be considered good. If not, the fit is considered compromised. @@ -502,4 +537,4 @@ MAS_TO_ARCSEC: float (*default=1000*) Conversion factor from milliarcseconds to arcseconds. GOOD_FIT_QUALITY_VALUES: int (*default=[1, 2, 3, 4]*) - The fit_quality (see above) flag values that are allowable for a successful fit. \ No newline at end of file + The fit_quality (see above) flag values that are allowable for a successful fit. From f674fb36d1ae63281ac94bb6e8e0f739480379f3 Mon Sep 17 00:00:00 2001 From: Steve Goldman <32876747+s-goldman@users.noreply.github.com> Date: Tue, 22 Oct 2024 11:56:56 -0400 Subject: [PATCH 2/9] added adr. --- doc/ADRs.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/doc/ADRs.md b/doc/ADRs.md index 44199affa..d9c8e03e6 100644 --- a/doc/ADRs.md +++ b/doc/ADRs.md @@ -25,4 +25,18 @@ The second column of a poller file is now reserved (for WFPC2) for passing the a ## Consequences -Caution must be taken is using variations of the poller file while processing WFPC2 data. \ No newline at end of file +Caution must be taken is using variations of the poller file while processing WFPC2 data. + +# The Use of the TEAL Interface 11/14/23 + +## Context + +The code can be run using the interactive GUI TEAL, however, it is hard to maintain. TEAL is also used in the code to load the json parameter files. + +## Decision + +In order to make the code more easily maintainable, we will no longer support the use of the GUI for running tasks, however, TEAL will still be used in the background to load the parameter files as there is no current alternative for parsing that data. + +## Consequences + +TEAL will need to be included in drizzlepac until a replacement for parsing the json files can be found. \ No newline at end of file From 0c54cd5056df3d4b47a469e36c5481368298f681 Mon Sep 17 00:00:00 2001 From: Steve Goldman <32876747+s-goldman@users.noreply.github.com> Date: Tue, 22 Oct 2024 12:05:54 -0400 Subject: [PATCH 3/9] removed warning. --- doc/source/mast_data_products/hap-parameters.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/source/mast_data_products/hap-parameters.rst b/doc/source/mast_data_products/hap-parameters.rst index d601e1806..c8c9d1203 100644 --- a/doc/source/mast_data_products/hap-parameters.rst +++ b/doc/source/mast_data_products/hap-parameters.rst @@ -2,6 +2,7 @@ HAP Parameters ----------------------------------- .. _hap-parameters: + The HAP parameter files are housed in the drizzlepac/pars/hap_pars directory. There are different json parameter files for the SVM and MVM products in separate directories. The svm_parameter json files are also used for pipeline products. From 37dbbbafb1f765f6e662b2acf502c5440ceba6b7 Mon Sep 17 00:00:00 2001 From: Steve Goldman <32876747+s-goldman@users.noreply.github.com> Date: Mon, 28 Oct 2024 10:54:49 -0400 Subject: [PATCH 4/9] implemented michele's suggestions. --- doc/source/mast_data_products/hap-parameters.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/source/mast_data_products/hap-parameters.rst b/doc/source/mast_data_products/hap-parameters.rst index c8c9d1203..f621fe7b6 100644 --- a/doc/source/mast_data_products/hap-parameters.rst +++ b/doc/source/mast_data_products/hap-parameters.rst @@ -3,14 +3,14 @@ HAP Parameters .. _hap-parameters: -The HAP parameter files are housed in the drizzlepac/pars/hap_pars directory. There are different json parameter files for the SVM and MVM products in separate directories. The svm_parameter json files are also used for pipeline products. +In the high-level svm_parameters or mvm_parameters directories, the instrument/detector-dependent configuration index files (e.g., acs_sbc_index.json) provide a map to the configuration files used for the major HAP processing steps. The svm_parameter configuration files are also used for pipeline products. -In either directory, there is a separate detector json file for each instrument/detector. Each file specifies additional json files used for different tasks and scenarios. The primary differences are in the case of different number of exposures: +For the astrodrizzle step, there are different files which should be employed depending upon the number of exposures int the visit. -Detector JSON Files -******************* +Detector Configuration Files +**************************** acs_hrc_index.json @@ -37,7 +37,7 @@ acs_hrc_index.json } } -The different files for "astrodrizzle" are for the case of 1 exposure ("any_n1"), 2 exposures (acs_hrc_any_n2), etc. Each json file with an associated task above (e.g. astrodrizzle) will have the parameters for that task. These files will be used to instantiate variables that are required in the processing of the data and, in the case of "astrodrizzle", are separated into different steps. +The different files for "astrodrizzle" are for the case of 1 exposure ("any_n1"), 2 exposures (acs_hrc_any_n2), etc. Each json file with an associated task above (e.g. astrodrizzle) will have the parameters for that task. These files will be used to initialize variables that are required in the processing of the data and, in the case of "astrodrizzle", are separated into different steps. Shown below are the parameters that are used by the Hubble Advanced Products. We include the parameter, the default value for WFC3 processing, and a description of that parameter. From 63a509da969b28b6d0540b7e40066e29003f59b5 Mon Sep 17 00:00:00 2001 From: Steve Goldman <32876747+s-goldman@users.noreply.github.com> Date: Tue, 29 Oct 2024 09:38:27 -0400 Subject: [PATCH 5/9] WFC3 IR example labeled as such. --- doc/source/mast_data_products/hap-parameters.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/source/mast_data_products/hap-parameters.rst b/doc/source/mast_data_products/hap-parameters.rst index f621fe7b6..b9cf63eb7 100644 --- a/doc/source/mast_data_products/hap-parameters.rst +++ b/doc/source/mast_data_products/hap-parameters.rst @@ -39,7 +39,7 @@ acs_hrc_index.json The different files for "astrodrizzle" are for the case of 1 exposure ("any_n1"), 2 exposures (acs_hrc_any_n2), etc. Each json file with an associated task above (e.g. astrodrizzle) will have the parameters for that task. These files will be used to initialize variables that are required in the processing of the data and, in the case of "astrodrizzle", are separated into different steps. -Shown below are the parameters that are used by the Hubble Advanced Products. We include the parameter, the default value for WFC3 processing, and a description of that parameter. +Shown below are the parameters that are used by the Hubble Advanced Products. As an example, we include the parameter, the default value for WFC3/IR processing, and a description of that parameter. .. run_hap_processing .. identified in json files. From 681cb76ef077283442bbfe5f47cee452ad684443 Mon Sep 17 00:00:00 2001 From: Steve Goldman <32876747+s-goldman@users.noreply.github.com> Date: Tue, 12 Nov 2024 11:22:55 -0500 Subject: [PATCH 6/9] added example file name. --- doc/source/mast_data_products/hap-parameters.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/source/mast_data_products/hap-parameters.rst b/doc/source/mast_data_products/hap-parameters.rst index b9cf63eb7..f88a2687e 100644 --- a/doc/source/mast_data_products/hap-parameters.rst +++ b/doc/source/mast_data_products/hap-parameters.rst @@ -39,7 +39,7 @@ acs_hrc_index.json The different files for "astrodrizzle" are for the case of 1 exposure ("any_n1"), 2 exposures (acs_hrc_any_n2), etc. Each json file with an associated task above (e.g. astrodrizzle) will have the parameters for that task. These files will be used to initialize variables that are required in the processing of the data and, in the case of "astrodrizzle", are separated into different steps. -Shown below are the parameters that are used by the Hubble Advanced Products. As an example, we include the parameter, the default value for WFC3/IR processing, and a description of that parameter. +Shown below are the parameters that are used by the Hubble Advanced Products. As an example, we include the parameter, the default value for WFC3/IR processing (listed in "wfc3_ir_astrodrizzle_any_n1.json"), and a description of that parameter. .. run_hap_processing .. identified in json files. From 316f0523885c6c597582f044a61c8af4f347e04d Mon Sep 17 00:00:00 2001 From: Steve Goldman <32876747+s-goldman@users.noreply.github.com> Date: Fri, 15 Nov 2024 15:40:29 -0500 Subject: [PATCH 7/9] michele suggestions. --- .../mast_data_products/hap-parameters.rst | 337 +++++++++--------- 1 file changed, 174 insertions(+), 163 deletions(-) diff --git a/doc/source/mast_data_products/hap-parameters.rst b/doc/source/mast_data_products/hap-parameters.rst index f88a2687e..e7fa0a898 100644 --- a/doc/source/mast_data_products/hap-parameters.rst +++ b/doc/source/mast_data_products/hap-parameters.rst @@ -39,6 +39,179 @@ acs_hrc_index.json The different files for "astrodrizzle" are for the case of 1 exposure ("any_n1"), 2 exposures (acs_hrc_any_n2), etc. Each json file with an associated task above (e.g. astrodrizzle) will have the parameters for that task. These files will be used to initialize variables that are required in the processing of the data and, in the case of "astrodrizzle", are separated into different steps. + +Detector Configuration Files for Alignment +****************************************** + +The following parameters are examples of the parameters used in the alignment configuration files. The example values are from "wfc3_ir_alignment_all.json". + +General +""""""" + +MIN_FIT_MATCHES: int (*default=4*) + Minimum number of acceptable cross-matches for a good fit. + +MAX_FIT_RMS: int (*default=20*) + Not currently in use. + +MAX_SOURCES_PER_CHIP: int (*default=250*) + Not currently in use. + + +run_align (*primarily in align.py*) +""""""""""""""""""""""""""""""""""" + +update_hdr_wcs: bool (*default=True*) + Not currently in use. Write newly computed WCS information to image headers? + +catalog_list: list of strings (*default=["GAIAeDR3", "GSC242", "2MASS"]*) + Not currently in use. Set of astrometric catalogs which should be used as references for fitting the input images. A separate fit will be performed for each catalog specified. The catalog name will also be used as part of the output ``WCSNAME`` value for the fit determined from that catalog. + +fit_algorithm_list_ngt1: list of strings (*default=["match_relative_fit", "match_2dhist_fit", "match_default_fit"]*) + Not currently in use. + +fit_algorithm_list_ng1: list of strings (*default=["match_2dhist_fit", "match_default_fit"]*) + Not currently in use. + +MIN_CATALOG_THRESHOLD: int (*default=3*) + Not currently in use. + +MIN_OBSERVABLE_THRESHOLD: int (*default=10*) + Not currently in use. The minimum number of observed sources required to continue fitting. If below this threshold, the code will return a status=1 and try with another catalog. + +MAX_FIT_LIMIT: int (*default=150*) + Not currently in use. + +mosaic_catalog_list: list of strings (*default=["GAIAeDR3", "GSC242", "2MASS"]*) + List of available catalogs for aligning for both pipeline and SVM products. The code will go through each catalog in this order. + +mosaic_fit_list: list of strings (*default=["match_relative_fit", "match_2dhist_fit", "match_default_fit"]*) + List of available fit algorithms for aligning for both pipeline and SVM products; match_default_fit relative alignment without using 2dhist and different throusholds (see json configuration files). + +mosaic_fitgeom_list: dict (*default={"rshift": 10, "rscale": 10, "general": 6}*) + The different fit geometries tried in alignment as well as their minimum number of objects (minobj) value which specifies the number of matched sources required for a successful fit. For pipeline products, the fitgeometry value is ignored and defaults to a fit geometry of ``rscale``; the code governing the default values is in *align_utils.perform_fit*. The value for minobj specified here, however, is used for the pipeline products. + + +.. note:: + + The quality of the fit is specified by the fit_quality parameter. The following values are used: + + * 1 = valid solution with rms < 10 mas + + * 2 = Valid but compromised solution with rms < 10 mas + + * 3 = Valid solution with RMS >= 10 mas + + * 4 = Valid but compromised solution with RMS >= 10 mas + + * 5 = Not valid solution + + +generate_source_catalogs (*primarily in align_utils.py*) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +.. warning:: + The following parameters are used in the generation of source catalogs for alignment only, and not used for the output source catalogs. + +box_size: int (*default=13*) + The size of the box used for calculating the 2D Background of the catalog "white light" image along each axis in pixels. In nothing is specified, a default of BKG_BOX_SIZE=27 is used. + +win_size: int (*default=3*) + The size of the 2D filter to apply to the background of the catalog "white light" image. If nothing is specified, a default of BKG_FILTER_SIZE=3 is used. + +nsigma: float (*default=3.0*) + The number of signma used for setting the level of the average background rms. If nothing is specified, a value of 5.0 is used. + +centering_mode: str (*default="starfind"*) + Algorithm to use when computing the positions of the detected sources. Options are "segmentaton" or ``starfind``. Centering will only take place after ``threshold`` has been determined, and sources are identified using segmentation. Centering using ``segmentation`` will rely on ``photutils.segmentation.SourceCatalog`` to generate the properties for the source catalog. Centering using ``starfind`` will use ``photutils.detection.IRAFStarFinder`` to characterize each source in the catalog. + +bkg_estimator: str (*default="MedianBackground"*) + Name of the algorithm to use for background calculation in *photutils.Background2D*. + +rms_estimator: str (*default="StdBackgroundRMS"*) + Name of the rms estimation algorithm used by *photutils.Background2D*. + +num_sources: int (*default=250*) + Not currently in use. Maximum number of brightest/faintest sources to return in catalog. + +deblend: bool (*default=false*) + Specify whether or not to apply photutils deblending algorithm when evaluating each of the identified segments (sources) from the chip. + +fwhmpsf: float (*default=0.13*) + The full width at half maximum of the PSF in arcseconds used for the starfind algorithm. + +classify: bool (*default=false*) + Not currently in use as cosmic rays are being removed before segmentation. Specifies whether or not to apply photutils classification algorithm when evaluating each of the identified segments (sources) from the chip. + +threshold: float (*default=-1.1*) + Value from the image which serves as the limit for determining sources. If None, compute a default value of (background+5*rms(background)). If threshold < 0.0, use absolute value as scaling factor for default value. If nothing is specified a default value of None is used to compute the background. + + + +generate_astrometric_catalog +"""""""""""""""""""""""""""" + +gaia_only: bool (*default=false*) + Not currently in use. Specify whether or not to only use sources from GAIA in output catalog. + +existing_wcs: Class Instance (*default=null*) + Existing WCS object specified by the user + + +perform_fit (*primarily external in tweakwcs.matchutils.XYXYMatch*) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" +For match_relative_fit, match_default_fit, and match_2dhist_fit, the following parameters are used: + +fitgeom": str, + As used above, this is ignored for pipeline products. + +searchrad: float + The search radius for a match (in units of the tangent plane). + +separation: float + The minimum separation in the tangent plane (in units of the tangent plane) for sources in the image and reference catalogs in order to be considered to be distinct sources. Objects closer together than ``separation`` distance are removed from the image and reference coordinate catalogs prior to matching. + +tolerance: float + The matching tolerance (in units of the tangent plane) after applying an initial solution derived from the ``triangles`` algorithm. + +use2dhist: bool + Use 2D histogram to find initial offset? + + +determine_fit_quality +""""""""""""""""""""" + +MIN_CATALOG_THRESHOLD": int (*default=3*) + The minimum number of catalog sources required to continue fitting. If below this threshold, the code will return a fit_quality=5 and try with another catalog. + +MIN_OBSERVABLE_THRESHOLD": int (*default=4*) + If the number of observed sources is below this threshold, the code ends alignment and defers to an *a priori* solution. + +MIN_CROSS_MATCHES": int (*default=3*) + Not currently in use. + +MIN_FIT_MATCHES": int (*default=4*) + Not currently in use. + +MAX_FIT_RMS": float (*default=20*) + Not currently in use. Maximum RMS value for a fit to be considered good. Currently a warning is printed but nothing is done with this parameter. + +MAX_FIT_LIMIT": int (*default=150*) + The maximum allowable RMS value for a fit to be considered good. If not, the fit is considered compromised. + +MAX_SOURCES_PER_CHIP: int (*default=250*) + Maximum number of brightest sources per chip which will be used for cross-matching and fitting. + +MAS_TO_ARCSEC: float (*default=1000*) + Conversion factor from milliarcseconds to arcseconds. + +GOOD_FIT_QUALITY_VALUES: int (*default=[1, 2, 3, 4]*) + The fit_quality (see above) flag values that are allowable for a successful fit. + + +Detector Configuration Files for Astrodrizzle +********************************************* + Shown below are the parameters that are used by the Hubble Advanced Products. As an example, we include the parameter, the default value for WFC3/IR processing (listed in "wfc3_ir_astrodrizzle_any_n1.json"), and a description of that parameter. .. run_hap_processing @@ -376,166 +549,4 @@ final_crpix1: ??? (*default=null*) Reference pixel X position on output (CRPIX1) final_crpix2: ??? (*default=null*) - Reference pixel Y position on output (CRPIX2) - - -Alignment -^^^^^^^^^ - -General -""""""" - -MIN_FIT_MATCHES: int (*default=4*) - Minimum number of acceptable cross-matches for a good fit. - -MAX_FIT_RMS: int (*default=20*) - Not currently in use. - -MAX_SOURCES_PER_CHIP: int (*default=250*) - Not currently in use. - - -run_align (*primarily in align.py*) -""""""""""""""""""""""""""""""""""" - -update_hdr_wcs: bool (*default=True*) - Not currently in use. Write newly computed WCS information to image headers? - -catalog_list: list of strings (*default=["GAIAeDR3", "GSC242", "2MASS"]*) - Not currently in use. Set of astrometric catalogs which should be used as references for fitting the input images. A separate fit will be performed for each catalog specified. The catalog name will also be used as part of the output ``WCSNAME`` value for the fit determined from that catalog. - -fit_algorithm_list_ngt1: list of strings (*default=["match_relative_fit", "match_2dhist_fit", "match_default_fit"]*) - Not currently in use. - -fit_algorithm_list_ng1: list of strings (*default=["match_2dhist_fit", "match_default_fit"]*) - Not currently in use. - -MIN_CATALOG_THRESHOLD: int (*default=3*) - Not currently in use. - -MIN_OBSERVABLE_THRESHOLD: int (*default=10*) - Not currently in use. The minimum number of observed sources required to continue fitting. If below this threshold, the code will return a status=1 and try with another catalog. - -MAX_FIT_LIMIT: int (*default=150*) - Not currently in use. - -mosaic_catalog_list: list of strings (*default=["GAIAeDR3", "GSC242", "2MASS"]*) - List of available catalogs for aligning for both pipeline and SVM products. The code will go through each catalog in this order. - -mosaic_fit_list: list of strings (*default=["match_relative_fit", "match_2dhist_fit", "match_default_fit"]*) - List of available fit algorithms for aligning for both pipeline and SVM products; match_default_fit relative alignment without using 2dhist and different throusholds (see json configuration files). - -mosaic_fitgeom_list: dict (*default={"rshift": 10, "rscale": 10, "general": 6}*) - The different fit geometries tried in alignment as well as their minobj value which specifies the number of matched sources required for a successful fit. For pipeline products, the fitgeometry value is ignored and defaults to a fit geometry of ``rscale``. The fitgeom for the pipeline products is specified as a default in *align_utils.perform_fit*. The value for minobj specified here, however, is used for the pipeline products. - -fit quality categories -"""""""""""""""""""""" - - * 1 = valid solution with rms < 10 mas - - * 2 = Valid but compromised solution with rms < 10 mas - - * 3 = Valid solution with RMS >= 10 mas - - * 4 = Valid but compromised solution with RMS >= 10 mas - - * 5 = Not valid solution - - -generate_source_catalogs (*primarily in align_utils.py*) -"""""""""""""""""""""""""""""""""""""""""""""""""""""""" - -box_size: int (*default=13*) - The size of the box used for calculating the 2D Background of the catalog "white light" image along each axis in pixels. In nothing is specified, a default of BKG_BOX_SIZE=27 is used. - -win_size: int (*default=3*) - The size of the 2D filter to apply to the background of the catalog "white light" image. If nothing is specified, a default of BKG_FILTER_SIZE=3 is used. - -nsigma: float (*default=3.0*) - The number of signma used for setting the level of the average background rms. If nothing is specified, a value of 5.0 is used. - -centering_mode: str (*default="starfind"*) - Algorithm to use when computing the positions of the detected sources. Options are "segmentaton" or ``starfind``. Centering will only take place after ``threshold`` has been determined, and sources are identified using segmentation. Centering using ``segmentation`` will rely on ``photutils.segmentation.SourceCatalog`` to generate the properties for the source catalog. Centering using ``starfind`` will use ``photutils.detection.IRAFStarFinder`` to characterize each source in the catalog. - -bkg_estimator: str (*default="MedianBackground"*) - Name of the algorithm to use for background calculation in *photutils.Background2D*. - -rms_estimator: str (*default="StdBackgroundRMS"*) - Name of the rms estimation algorithm used by *photutils.Background2D*. - -num_sources: int (*default=250*) - Not currently in use. Maximum number of brightest/faintest sources to return in catalog. - -deblend: bool (*default=false*) - Specify whether or not to apply photutils deblending algorithm when evaluating each of the identified segments (sources) from the chip. - -fwhmpsf: float (*default=0.13*) - The full width at half maximum of the PSF in arcseconds used for the starfind algorithm. - -classify: bool (*default=false*) - Not currently in use as cosmic rays are being removed before segmentation. Specifies whether or not to apply photutils classification algorithm when evaluating each of the identified segments (sources) from the chip. - -threshold: float (*default=-1.1*) - Value from the image which serves as the limit for determining sources. If None, compute a default value of (background+5*rms(background)). If threshold < 0.0, use absolute value as scaling factor for default value. If nothing is specified a default value of None is used to compute the background. - - - -generate_astrometric_catalog -"""""""""""""""""""""""""""" - -gaia_only: bool (*default=false*) - Not currently in use. Specify whether or not to only use sources from GAIA in output catalog. - -existing_wcs: Class Instance (*default=null*) - Existing WCS object specified by the user - - -perform_fit (*primarily external in tweakwcs.matchutils.XYXYMatch*) -""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" -For match_relative_fit, match_default_fit, and match_2dhist_fit, the following parameters are used: - -fitgeom": "rscale", - As used above, this is ignored for pipeline products. - -searchrad: float (*default=125*) - The search radius for a match (in units of the tangent plane). - -separation: float (*default=4.0*) - The minimum separation in the tangent plane (in units of the tangent plane) for sources in the image and reference catalogs in order to be considered to be distinct sources. Objects closer together than ``separation`` distance are removed from the image and reference coordinate catalogs prior to matching. - -tolerance: float (*default=2.0*) - The matching tolerance (in units of the tangent plane) after applying an initial solution derived from the ``triangles`` algorithm. - -use2dhist: bool(*default=true*) - Use 2D histogram to find initial offset? - - -determine_fit_quality -""""""""""""""""""""" - -MIN_CATALOG_THRESHOLD": int (*default=3*) - The minimum number of catalog sources required to continue fitting. If below this threshold, the code will return a fit_quality=5 and try with another catalog. - -MIN_OBSERVABLE_THRESHOLD": int (*default=4*) - If the number of observed sources is below this threshold, the code ends alignment and defers to an *a priori* solution. - -MIN_CROSS_MATCHES": int (*default=3*) - Not currently in use. - -MIN_FIT_MATCHES": int (*default=4*) - Not currently in use. - -MAX_FIT_RMS": float (*default=20*) - Not currently in use. Maximum RMS value for a fit to be considered good. Currently a warning is printed but nothing is done with this parameter. - -MAX_FIT_LIMIT": int (*default=150*) - The maximum allowable RMS value for a fit to be considered good. If not, the fit is considered compromised. - -MAX_SOURCES_PER_CHIP: int (*default=250*) - Maximum number of brightest sources per chip which will be used for cross-matching and fitting. - -MAS_TO_ARCSEC: float (*default=1000*) - Conversion factor from milliarcseconds to arcseconds. - -GOOD_FIT_QUALITY_VALUES: int (*default=[1, 2, 3, 4]*) - The fit_quality (see above) flag values that are allowable for a successful fit. + Reference pixel Y position on output (CRPIX2) \ No newline at end of file From b18916c0c160091c8c896fc7f87fb79019139aa6 Mon Sep 17 00:00:00 2001 From: Steve Goldman <32876747+s-goldman@users.noreply.github.com> Date: Fri, 15 Nov 2024 15:42:38 -0500 Subject: [PATCH 8/9] spelling mistake. --- doc/source/mast_data_products/hap-parameters.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/source/mast_data_products/hap-parameters.rst b/doc/source/mast_data_products/hap-parameters.rst index e7fa0a898..5ee54fe5b 100644 --- a/doc/source/mast_data_products/hap-parameters.rst +++ b/doc/source/mast_data_products/hap-parameters.rst @@ -86,7 +86,7 @@ mosaic_catalog_list: list of strings (*default=["GAIAeDR3", "GSC242", "2MASS"]*) List of available catalogs for aligning for both pipeline and SVM products. The code will go through each catalog in this order. mosaic_fit_list: list of strings (*default=["match_relative_fit", "match_2dhist_fit", "match_default_fit"]*) - List of available fit algorithms for aligning for both pipeline and SVM products; match_default_fit relative alignment without using 2dhist and different throusholds (see json configuration files). + List of available fit algorithms for aligning for both pipeline and SVM products; match_default_fit relative alignment without using 2dhist and different thresholds (see json configuration files). mosaic_fitgeom_list: dict (*default={"rshift": 10, "rscale": 10, "general": 6}*) The different fit geometries tried in alignment as well as their minimum number of objects (minobj) value which specifies the number of matched sources required for a successful fit. For pipeline products, the fitgeometry value is ignored and defaults to a fit geometry of ``rscale``; the code governing the default values is in *align_utils.perform_fit*. The value for minobj specified here, however, is used for the pipeline products. From dfaed96e224602d0d61d995105f3d31a35ca3339 Mon Sep 17 00:00:00 2001 From: Steve Goldman <32876747+s-goldman@users.noreply.github.com> Date: Fri, 15 Nov 2024 16:13:38 -0500 Subject: [PATCH 9/9] fixed headings. --- .../mast_data_products/hap-parameters.rst | 22 +++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/doc/source/mast_data_products/hap-parameters.rst b/doc/source/mast_data_products/hap-parameters.rst index 5ee54fe5b..701d07a36 100644 --- a/doc/source/mast_data_products/hap-parameters.rst +++ b/doc/source/mast_data_products/hap-parameters.rst @@ -1,5 +1,5 @@ HAP Parameters ------------------------------------ +############## .. _hap-parameters: @@ -219,7 +219,7 @@ Shown below are the parameters that are used by the Hubble Advanced Products. As Initialize HAP -^^^^^^^^^^^^^^ +"""""""""""""" \_mdriztab_btn\_: str (*default=Update From MDRIZTAB*) Immediately read and use the values from the MDRIZTAB. @@ -261,7 +261,7 @@ in_memory: bool (*default=False*) Process everything in memory to minimize disk I/O? Instrument Parameters -^^^^^^^^^^^^^^^^^^^^^ +""""""""""""""""""""" .. or float? @@ -284,7 +284,7 @@ exptime: str (*default=""*) .. ? State of input files -^^^^^^^^^^^^^^^^^^^^ +"""""""""""""""""""" restore: bool (*default=False*) Copy input files FROM archive directory for processing? @@ -299,7 +299,7 @@ clean: bool (*default=True*) Delete temporary files after completion? Step 1: Static mask -^^^^^^^^^^^^^^^^^^^ +""""""""""""""""""" static: bool (*default=True*) Create static bad-pixel mask from the data? @@ -308,7 +308,7 @@ static_sig: float (*default=4.0*) Sigma*rms below mode to clip for static mask Step 2: Sky Subtraction -^^^^^^^^^^^^^^^^^^^^^^^ +""""""""""""""""""""""" skysub: bool (*default=False*) Turn on or off sky subtraction on the input data. When ``skysub`` is set to ``no``, then ``skyuser`` field will be enabled and if user specifies a header keyword showing the sky value in the image, then that value will be used for CR-rejection but it will not be subtracted from the (drizzled) image data. If user sets ``skysub`` to ``yes`` then ``skyuser`` field will be disabled (and if it is not empty - it will be ignored) and user can use one of the methods available through the ``skymethod`` parameter to compute the sky or provide a file (see ``skyfile`` parameter) with values that should be subtracted from (single) drizzled images. @@ -353,7 +353,7 @@ skyuser: str (*default""*) KEYWORD indicating a sky subtraction value if done by user Step 3: Drizzle Separate images -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +""""""""""""""""""""""""""""""" driz_separate : bool (*default=False*) This parameter specifies whether or not to drizzle each input image onto separate output images. The separate output images will all have the same WCS as the final combined output frame. These images are used to create the median image, needed for cosmic ray rejection. @@ -412,7 +412,7 @@ driz_sep_crpix2: float or null (*default=null*) Reference pixel Y position on output (CRPIX2). Step 4: Create Median Image -^^^^^^^^^^^^^^^^^^^^^^^^^^^ +""""""""""""""""""""""""""" median: bool (*default=False*) Create a median image? @@ -450,7 +450,7 @@ combine_bufsize: ??? (*default=null*) Step 5: Blot back the median image -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +"""""""""""""""""""""""""""""""""" blot: bool (*default=False*) Blot the median back to the input frame? @@ -468,7 +468,7 @@ blot_skyval: float (*default=0.0*) Custom sky value to be added to blot image Step 6: Remove cosmic rays with deriv, driz_cr -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +"""""""""""""""""""""""""""""""""""""""""""""" driz_cr: bool (*default=False*) Perform CR rejection with deriv and driz_cr? @@ -489,7 +489,7 @@ driz_cr_corr: bool (*default=False*) Create CR cleaned _crclean file and a _crmask file? Step 7: Drizzle final combined image -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +"""""""""""""""""""""""""""""""""""" driz_combine: bool (*default=True*) Perform final drizzle image combination?