Skip to content

Commit

Permalink
fix up listings text
Browse files Browse the repository at this point in the history
  • Loading branch information
troyraen committed Jul 22, 2024
1 parent bf71b36 commit 13a406d
Showing 1 changed file with 53 additions and 47 deletions.
100 changes: 53 additions & 47 deletions docs/source/listings.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,14 @@
Data Listings
=============

This page contains a listing of all data products served by Pitt-Google Alert Broker along with the connection information needed for access.
The listings are organized by survey, then data product and format.
This page contains a listing of all data products served by Pitt-Google Alert Broker and includes the connection information needed for access.

All data is served through Pitt-Google's public data repositories in Google Cloud.
Pitt-Google's Google Cloud project ID, which will be needed to access the data stores listed below, is:
.. contents:: Jump to:
:depth: 1
:local:

All data is served through public data repositories in Google Cloud.
Pitt-Google's Google Cloud project ID, which will be needed for access, is:

.. code-block::
Expand All @@ -16,67 +19,70 @@ Pitt-Google's Google Cloud project ID, which will be needed to access the data s
For examples of how to use the information on this page, please see our :ref:`API Reference <api reference>` and `User Demos <https://github.com/mwvgroup/pittgoogle-user-demos/>`__ repo.

.. _data ztf:

Zwicky Transient Facility (:ref:`ZTF <survey ztf>`)
---------------------------------------------------

ZTF is a wide-field, optical survey in three bands: r, g, and i.
It produces an alert stream of transients detected by difference imaging at a rate of 10^5 - 10^6 alerts per night.
It scans the entire Northern sky every two days.
-------------------------

..
Tables below use ':class: tight-table' so that longer blocks of text will wrap
instead of rendering as a single line per row with a horizontal scroll bar.
The class is defined in docs/source/_static/css/custom.css.
.. _data ztf:

ZTF Mission resources:

- `ZTF Homepage <https://www.ztf.caltech.edu/>`__
- `ZTF Science Data System Explanatory Supplement <https://irsa.ipac.caltech.edu/data/ZTF/docs/ztf_explanatory_supplement.pdf>`__
Zwicky Transient Facility (ZTF)
-------------------------------

Pitt-Google Broker serves the ZTF alert data in the formats listed below.
:ref:`ZTF <survey ztf>` is a wide-field, optical survey producing an alert stream at a rate of 10^5 - 10^6 per night.

Pub/Sub Message Streams
^^^^^^^^^^^^^^^^^^^^^^^
Pub/Sub Alert Streams
^^^^^^^^^^^^^^^^^^^^^

.. list-table:: Pitt-Google's ZTF Streams
.. list-table::
:class: tight-table
:widths: 25 75
:header-rows: 1

* - Topic Name
* - Topic
- Description

* - **ztf-loop**
- Use this stream for testing. Recent ZTF alerts are published to this topic
at a roughly constant rate of 1 per second.
* - .. centered:: *Data Streams*
-

* - ztf-alerts
- Full ZTF alert stream
- ZTF's alert stream served through Pub/Sub and cleaned of duplicate alerts.
Messages contain the original alert bytes and metadata.

* - ztf-lite
- Lite version of ztf-alerts (every alert, subset of fields)
- Lite version of ztf-alerts (every alert, subset of fields).

* - ztf-tagged
- ztf-lite with basic categorizations such as is pure and “is likely extragalactic
transient” added to the message metadata.
- ztf-lite with basic categorizations such as "is pure" and “is likely extragalactic
transient” added to the message metadata to aid filtering.

* - ztf-SuperNNova
- ztf-tagged plus SuperNNova classification results (Ia vs non-Ia).

* - .. centered:: *Notification Streams*
-

* - ztf-alert_avros
- Notification stream from the ztf-alert_avros Cloud Storage bucket indicating
that a new alert packet is in file storage.
These messages contain no data, only attributes.
The file name is in the attribute "objectId",
and the bucket name is in the attribute "bucketId".
- One message per alert indicating that the Avro file is available in a Cloud Storage bucket.
The file name and bucket name are in the message attributes "objectId" and "bucketId" respectively.
Messages contain no data.

* - ztf-BigQuery
- Notification stream indicating that alert data is available in BigQuery tables.
- One message per alert indicating that the alert data is available in a BigQuery table.

* - .. centered:: *Testing Stream*
-

* - ztf-loop
- Recent messages from the ztf-alerts stream, published at a rate of 1 per second.
This stream is intended for testing and development use cases.

BigQuery Tables
^^^^^^^^^^^^^^^

.. list-table:: Pitt-Google's ZTF Datasets and Tables
.. list-table::
:class: tight-table
:widths: 15 15 70
:header-rows: 1
Expand All @@ -87,15 +93,14 @@ BigQuery Tables

* - ztf_alerts
- alerts
- Complete alert packet, excluding image cutouts.
Same schema as the original alert, including nested and repeated fields.
- ZTF alert data. This table is an archive of the ztf-alerts Pub/Sub stream, excluding image cutouts and metadata.
It has the same schema as the original alert bytes (except cutouts), including nested and repeated fields.

* - ztf_alerts
- DIASource
- Alert packet data for the triggering source only. Including the object ID and a
list of source IDs for the previous sources included in the alert,
excluding cutouts and data for previous sources.
Flat schema.
- ZTF alert data of the DIA source that triggered the alert. Includes the object ID and a
list of source IDs for the previous sources included in the alert, excluding cutouts and
data for previous sources. The schema is flat.

* - ztf_alerts
- SuperNNova
Expand All @@ -108,17 +113,18 @@ BigQuery Tables
bucket name and filename, etc.).


Cloud Storage Files
^^^^^^^^^^^^^^^^^^^
Cloud Storage Buckets
^^^^^^^^^^^^^^^^^^^^^

.. list-table:: Pitt-Google's ZTF Buckets
.. list-table::
:class: tight-table
:widths: 40 60
:header-rows: 1

* - Bucket Name
* - Bucket
- Description

* - ardent-cycling-243415-ztf-alert_avros
- Contains the complete, original alert packets as Avro files.
Filename syntax is: `<ztf_topic>/<objectId>/<candid>.avro`
- ZTF alert data. This bucket contains an archive of the ztf-alerts Pub/Sub stream, including
image cutouts and metadata. Each alert is stored as a separate Avro file.
The filename syntax is: `<ztf_topic>/<objectId>/<candid>.avro`.

0 comments on commit 13a406d

Please sign in to comment.