HLA-1352: Description of JSON files #1901

s-goldman · 2024-10-22T17:29:36Z

Resolves HLA-1352

Closes #1898

This PR adds a description of the json files. It also adds an entry into the architecture design comments.

Checklist for maintainers

added entry in CHANGELOG.rst within the relevant release section
updated or added relevant tests
updated relevant documentation
added relevant label(s)
ran regression tests, post a link to the Jenkins job below.
How to run regression tests on a PR

jenkins test

doc/source/mast_data_products/hap-parameters.rst

mdlpstsci · 2024-11-11T22:22:41Z

doc/source/mast_data_products/hap-parameters.rst


 .. run_hap_processing
-.. identified in json files. 
+.. identified in json files.



The discussion here seems out of context. If you are describing the contents of a configuration file, provide the name of the specific file, and then the contents. Note each numbered section maps to a corresponding section of the NameOfConfigurationFile. Perhaps this would be more obvious to me if I were looking at the formatted output. Having said this, I will look at this with a phased approach in that more and clarifying information can added later once the basics are described.

Sounds good. I've added the example filename to the docs.

mdlpstsci · 2024-11-12T14:07:08Z

doc/source/mast_data_products/hap-parameters.rst

I think you should add section headings where the discussions of the catalog configuration file and the quality control (of the catalog information) configuration file will eventually be written.

I created a ticket for working on those. I tentatively assign you, as you know more about the catalogs, but I can also add empty sections in this PR.

mdlpstsci

I think this is a much needed start to the discussion of the important configuration parameters. While the discussion needs to be expanded, this PR forms the fundamental basis for the additional information which will be added later.

There are only a few changes requested at this time.

Changes addressed.

mdlpstsci · 2024-11-14T16:31:42Z

doc/source/mast_data_products/hap-parameters.rst

@@ -410,7 +446,7 @@ generate_source_catalogs (*primarily in align_utils.py*)
 """"""""""""""""""""""""""""""""""""""""""""""""""""""""


The "fit quality categories" title seems out of place here.

Add a note so the user is reminded...The generate_source_catalogs section is used for alignment purposes only. This has nothing to do with the output HAP catalog products.

I've added them as a note now. I agree that it's not the ideal place, but I couldn't find a better place. With a "note", it seems to flow better now.

mdlpstsci · 2024-11-14T16:39:07Z

doc/source/mast_data_products/hap-parameters.rst


 .. run_hap_processing
-.. identified in json files. 
+.. identified in json files.


It is a bit confusing below as there are sections of the configuration file, wfc3_ir_astrodrizzle_any_n1.json which I though were skipped, but they are just out of order. This needs to be corrected to avoid confusion.

I've reorganized the sections.

mdlpstsci · 2024-11-14T16:52:55Z

doc/source/mast_data_products/hap-parameters.rst


 MAX_FIT_LIMIT: int (*default=150*)
    Not currently in use.

 mosaic_catalog_list: list of strings (*default=["GAIAeDR3", "GSC242", "2MASS"]*)
-    List of available catalogs for aligning for both pipeline and SVM products. The code will go through each catalog in this order. 
+    List of available catalogs for aligning for both pipeline and SVM products. The code will go through each catalog in this order.

 mosaic_fit_list: list of strings (*default=["match_relative_fit", "match_2dhist_fit", "match_default_fit"]*)
    List of available fit algorithms for aligning for both pipeline and SVM products; match_default_fit relative alignment without using 2dhist and different throusholds (see json configuration files).


Fix spelling: throusholds

mdlpstsci · 2024-11-14T16:55:37Z

doc/source/mast_data_products/hap-parameters.rst


 mosaic_fit_list: list of strings (*default=["match_relative_fit", "match_2dhist_fit", "match_default_fit"]*)
    List of available fit algorithms for aligning for both pipeline and SVM products; match_default_fit relative alignment without using 2dhist and different throusholds (see json configuration files).

 mosaic_fitgeom_list: dict (*default={"rshift": 10, "rscale": 10, "general": 6}*)
-    The different fit geometries tried in alignment as well as their minobj value which specifies the number of matched sources required for a successful fit. For pipeline products, the fitgeometry value is ignored and defaults to a fit geometry of ``rscale``. The fitgeom for the pipeline products is specified as a default in *align_utils.perform_fit*. The value for minobj specified here, however, is used for the pipeline products. 
+    The different fit geometries tried in alignment as well as their minobj value which specifies the number of matched sources required for a successful fit. For pipeline products, the fitgeometry value is ignored and defaults to a fit geometry of ``rscale``. The fitgeom for the pipeline products is specified as a default in *align_utils.perform_fit*. The value for minobj specified here, however, is used for the pipeline products.


Change "minobj" to minimum number of objects. Fix this description regarding the type of products and the minimum number of objects.

mdlpstsci · 2024-11-14T17:04:07Z

doc/source/mast_data_products/hap-parameters.rst

@@ -459,7 +495,7 @@ perform_fit (*primarily external in tweakwcs.matchutils.XYXYMatch*)
 For match_relative_fit, match_default_fit, and match_2dhist_fit, the following parameters are used:

 fitgeom": "rscale",
-    As used above, this is ignored for pipeline products. 
+    As used above, this is ignored for pipeline products.


Since you are discussing three categories at once for brevity, I would not quote the "default" values as they are not the same for all categories.

default values removed.

mdlpstsci

A few more changes to avoid confusion and provide context.

Changes addressed.

mdlpstsci

A lot of work has been done on this topic. Adding more information should be captured by other PRs.

s-goldman added 3 commits October 22, 2024 11:49

initial commit.

f67bf8b

added adr.

f674fb3

removed warning.

0c54cd5

s-goldman added the Documentation label Oct 22, 2024

s-goldman requested review from mdlpstsci and a team as code owners October 22, 2024 17:29

mdlpstsci reviewed Oct 23, 2024

View reviewed changes

doc/source/mast_data_products/hap-parameters.rst Outdated Show resolved Hide resolved

mdlpstsci reviewed Oct 24, 2024

View reviewed changes

doc/source/mast_data_products/hap-parameters.rst Outdated Show resolved Hide resolved

mdlpstsci reviewed Oct 24, 2024

View reviewed changes

doc/source/mast_data_products/hap-parameters.rst Outdated Show resolved Hide resolved

s-goldman added 2 commits October 28, 2024 10:30

Merge branch 'main' into hla-1351_document_json_parameter_files_10_22_24

967e4de

implemented michele's suggestions.

37dbbba

s-goldman requested a review from mdlpstsci October 28, 2024 14:55

Merge branch 'main' into hla-1351_document_json_parameter_files_10_22_24

b49d4ac

mdlpstsci reviewed Oct 28, 2024

View reviewed changes

doc/source/mast_data_products/hap-parameters.rst Show resolved Hide resolved

s-goldman added 3 commits October 29, 2024 09:38

WFC3 IR example labeled as such.

63a509d

Merge branch 'main' into hla-1351_document_json_parameter_files_10_22_24

6005bea

Merge branch 'main' into hla-1351_document_json_parameter_files_10_22_24

891e6e5

s-goldman requested a review from mdlpstsci October 29, 2024 19:50

Merge branch 'main' into hla-1351_document_json_parameter_files_10_22_24

735de0b

mdlpstsci reviewed Nov 11, 2024

View reviewed changes

mdlpstsci reviewed Nov 12, 2024

View reviewed changes

mdlpstsci previously requested changes Nov 12, 2024

View reviewed changes

added example file name.

681cb76

s-goldman requested a review from mdlpstsci November 12, 2024 16:55

Merge branch 'main' into hla-1351_document_json_parameter_files_10_22_24

352d7d4

mdlpstsci reviewed Nov 14, 2024

View reviewed changes

mdlpstsci previously requested changes Nov 14, 2024

View reviewed changes

s-goldman added 2 commits November 15, 2024 15:40

michele suggestions.

316f052

spelling mistake.

b18916c

s-goldman requested a review from mdlpstsci November 15, 2024 20:43

fixed headings.

dfaed96

mdlpstsci approved these changes Nov 18, 2024

View reviewed changes

s-goldman merged commit 685e46d into spacetelescope:main Nov 18, 2024
18 checks passed

s-goldman deleted the hla-1351_document_json_parameter_files_10_22_24 branch November 18, 2024 14:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HLA-1352: Description of JSON files #1901

HLA-1352: Description of JSON files #1901

s-goldman commented Oct 22, 2024 •

edited

Loading

mdlpstsci Nov 11, 2024 •

edited

Loading

s-goldman Nov 12, 2024

mdlpstsci Nov 12, 2024

s-goldman Nov 12, 2024

mdlpstsci left a comment

mdlpstsci Nov 14, 2024 •

edited

Loading

s-goldman Nov 15, 2024

mdlpstsci Nov 14, 2024 •

edited

Loading

s-goldman Nov 15, 2024

mdlpstsci Nov 14, 2024

s-goldman Nov 15, 2024

mdlpstsci Nov 14, 2024

s-goldman Nov 15, 2024

mdlpstsci Nov 14, 2024

s-goldman Nov 15, 2024

mdlpstsci left a comment

mdlpstsci left a comment

		@@ -410,7 +446,7 @@ generate_source_catalogs (primarily in align_utils.py)
		""""""""""""""""""""""""""""""""""""""""""""""""""""""""

HLA-1352: Description of JSON files #1901

HLA-1352: Description of JSON files #1901

Conversation

s-goldman commented Oct 22, 2024 • edited Loading

mdlpstsci Nov 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mdlpstsci left a comment

Choose a reason for hiding this comment

mdlpstsci Nov 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mdlpstsci Nov 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mdlpstsci left a comment

Choose a reason for hiding this comment

mdlpstsci left a comment

Choose a reason for hiding this comment

s-goldman commented Oct 22, 2024 •

edited

Loading

mdlpstsci Nov 11, 2024 •

edited

Loading

mdlpstsci Nov 14, 2024 •

edited

Loading

mdlpstsci Nov 14, 2024 •

edited

Loading