Add DB schema for LSST alerts

[JulienPeloton]

It is time to define the HBase table schemas that will be used for LSST alerts.

The ZTF strategy

For ZTF, we had a brute force strategy:

• main table: contains ALL original fields, including image stamps
• auxiliary tables: contains a subset of all fields (typically a couple of dozens)

This led to huge tables (as of 2024/09):

7.5 T    7.5 T    /hbase_fink2/data/default/ztf
193.6 M  193.6 M  /hbase_fink2/data/default/ztf.anomaly
642.0 G  642.0 G  /hbase_fink2/data/default/ztf.class
562.4 G  562.4 G  /hbase_fink2/data/default/ztf.jd
9.5 K    9.5 K    /hbase_fink2/data/default/ztf.metadata
1.9 M    1.9 M    /hbase_fink2/data/default/ztf.orb_cand
38.2 G   38.2 G   /hbase_fink2/data/default/ztf.pixel128
9.8 G    9.8 G    /hbase_fink2/data/default/ztf.pixel131072
9.4 G    9.4 G    /hbase_fink2/data/default/ztf.pixel4096
61.3 G   61.3 G   /hbase_fink2/data/default/ztf.ssnamenr
20.3 M   20.3 M   /hbase_fink2/data/default/ztf.sso_cand
201.0 M  201.0 M  /hbase_fink2/data/default/ztf.sso_resolver
3.6 K    3.6 K    /hbase_fink2/data/default/ztf.ssocand_oid
1.1 G    1.1 G    /hbase_fink2/data/default/ztf.tns
125.9 M  125.9 M  /hbase_fink2/data/default/ztf.tns_resolver
585.7 M  585.7 M  /hbase_fink2/data/default/ztf.tracklet
315.9 G  315.9 G  /hbase_fink2/data/default/ztf.upper
59.2 G   59.2 G   /hbase_fink2/data/default/ztf.uppervalid

Note that the auxiliary ztf.class table for example is up to 10% of the main table! While this strategy has some advantages (procedure is simple, all data available), it is not optimal: a lot of data duplication, heavy writes (because of images), and a lot of zombie data (things never accessed).

LSST strategy

Main vs auxiliary

TBD

Compression

[Login required] some optimisation work was done on compression & cell versionning: https://gitlab.in2p3.fr/fink/rubin-performance-check/-/tree/main/hbase?ref_type=heads

The main conclusion of that study is that we should use snappy compression, and limit the number of images stored on the portal. Here is a typical table initialisation:

create "lsst", 
{NAME => 'i', VERSIONS => 50000, COMPRESSION => 'SNAPPY'}, # LSST provided fields
{NAME => 'd', VERSIONS => 50000, COMPRESSION => 'SNAPPY'}, # Fink added values
{NAME => 'b', VERSIONS => 5, COMPRESSION => 'SNAPPY'} # images

Salting

TBD