v2 zcatalog

[rongpu]

This PR revamps the zcatalogs to 1) add columns needed for selecting confident redshifts and add the redshift confidence flags and 2) reduce the file sizes by splitting a single catalog into row-by-row matched catalogs. This PR also updates validredshifts.py which is now required by zcatalog.

Summary:

• The core catalog (e.g., zpix-main-dark.fits) now only contains the frequently used columns. The less frequently used columns are stored in two row-by-row matched FITS catalogs, *-imaging.fits that stores the imaging-related columns and *-extra.fits that stores the rest of the columns (e.g., zpix-main-dark-imaging.fits zpix-main-dark-extra.fits).
• Add the GOOD_SPEC column to indicate good hardware status. It is defined as: ((COADD_FIBERSTATUS==0) OR (COADD_FIBERSTATUS==2**3)) AND (OBJTYPE=='TGT'). The "TGT" requirement means that sky fibers will always have GOOD_SPEC==False.
• Add the Z_QSO column (and ZERR_QSO), which is the recommended redshift for quasars. If an object meets both the criteria for QuasarNET afterburner rerun (i.e., rerunning redrock using on QSO templates and QuasarNET prior and returns "Z_NEW") and the criteria for Z_NEW adoption, Z_QSO takes the Z_NEW value, otherwise it takes the original redrock Z value. Also add the boolean GOOD_Z_QSO flag that indicates the redshift confidence of Z_QSO, and it is True if 1) it is a QSO target AND 2) it passes the QSO redshift quality cut AND 3) GOOD_SPEC==True.
• Add the boolean flags GOOD_Z_BGS, GOOD_Z_LRG and GOOD_Z_ELG that indicate the redshift confidence of Z for the three tracers. GOOD_Z_BGS is True if 1) it is a BGS target AND 2) it passes the BGS redshift quality cut AND 3) GOOD_SPEC==True. Similarly for the LRG and ELG flags.
• Add a merged redshift confidence flag Z_CONF for Z. It is the recommended redshift quality flag to use for non-quasar objects. It is an integer that can take one of three values:
  2: high confidence; definition: is a primary galaxy (BGS, LRG or ELG) target, AND passes the corresponding redshift quality cut, AND has GOOD_SPEC==True.
  1: low confidence; definition: fails the criteria for Z_CONF=2, AND has GOOD_SPEC==True AND ZWARN==0
  0: no confidence (fails the criteria for both 1 and 2).

The relevant email thread is [desi-data 6673].

[akremin]

I don't have a solution (or even a strong opinion) but I thought I should write this down as an item for discussion before I forgot the offline discussion we had:

Here you're adding 'GOOD_Z_QSO', 'Z_QSO', and 'ZERR_QSO'; but your definition of Z_CONF has no "level 2" criteria for QSO. I think many people will be confused why something called 'GOOD_Z_QSO' is not 'Z_CONF'==2. The answer, I believe, is that you're distinguishing between 'Z' and 'Z_QSO'. The question is whether we should be doing that or not.

Offline we had discussed moving 'Z' to 'Z_RR' and creating a derived redshift column that merges 'Z' and 'Z_QSO'. If we do that, which we probably should, then we should also set 'GOOD_Z_QSO'==True as 'Z_CONF'==2.

Whether we want to do this or leave quasars as a separate set of columns is up for discussion, so let's discuss.