diff --git a/docs/source/conf.py b/docs/source/conf.py index 29daf1b..9f0ff9f 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -21,7 +21,7 @@ # -- Project information ----------------------------------------------------- project = "pittgoogle-client" -copyright = "2021, The Pitt-Google Broker Team" +copyright = "2021, The Pitt-Google Alert Broker Team" author = "Troy Raen" # The full version, including alpha/beta/rc tags diff --git a/docs/source/index.rst b/docs/source/index.rst index 735b1c6..eb1ab1f 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -20,6 +20,7 @@ If you run into trouble, please listings one-time-setup/index faq/index + surveys/index for-developers/index .. toctree:: diff --git a/docs/source/listings.rst b/docs/source/listings.rst index 5e06aa8..f31c331 100644 --- a/docs/source/listings.rst +++ b/docs/source/listings.rst @@ -1,14 +1,42 @@ +.. _listings: + Data Listings ============= -This page contains a listing of the data resources served by Pitt-Google Alert Broker. +This page contains a listing of all data products served by Pitt-Google Alert Broker and includes the connection information needed for access. + +.. contents:: Jump to: + :depth: 1 + :local: + +All data is served through public data repositories in Google Cloud. +Pitt-Google's Google Cloud project ID, which will be needed for access, is: + +.. code-block:: + + # Pitt-Google's Google Cloud project ID + ardent-cycling-243415 + +For examples of how to use the information on this page, please see our :ref:`API Reference ` and `User Demos `__ repo. + +------------------------- -.. _data pubsub: +.. + Tables below use ':class: tight-table' so that longer blocks of text will wrap + instead of rendering as a single line per row with a horizontal scroll bar. + The class is defined in docs/source/_static/css/custom.css. -Pub/Sub Message Streams ------------------------- +.. _data ztf: -.. list-table:: ZTF Streams +Zwicky Transient Facility (ZTF) +------------------------------- + +:ref:`ZTF ` is a wide-field, optical survey producing an alert stream at a rate of 10^5 - 10^6 per night. + +Pub/Sub Alert Streams +^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: :class: tight-table :widths: 25 75 :header-rows: 1 @@ -16,39 +44,45 @@ Pub/Sub Message Streams * - Topic - Description + * - .. centered:: *Data Streams* + - + * - ztf-alerts - - Full ZTF alert stream + - ZTF alert stream in Pub/Sub, cleaned of duplicate alerts. + Messages contain the original alert bytes and metadata. * - ztf-lite - - Lite version of ztf-alerts (every alert, subset of fields) + - Lite version of ztf-alerts (every alert, subset of fields). * - ztf-tagged - - ztf-lite with basic categorizations such as “is pure” and “is likely extragalactic - transient” added to the message metadata. + - ztf-lite with basic categorizations such as "is pure" and “is likely extragalactic + transient” added to the message metadata to aid filtering. * - ztf-SuperNNova - ztf-tagged plus SuperNNova classification results (Ia vs non-Ia). + * - .. centered:: *Notification Streams* + - + * - ztf-alert_avros - - Notification stream from the ztf-alert_avros Cloud Storage bucket indicating - that a new alert packet is in file storage. - These messages contain no data, only attributes. - The file name is in the attribute "objectId", - and the bucket name is in the attribute "bucketId". + - One message per alert indicating that the Avro file is available in a Cloud Storage bucket. + The file and bucket names are in the message attributes. Messages contain no data. * - ztf-BigQuery - - Notification stream indicating that alert data is available in BigQuery tables. + - One message per alert indicating that the alert data is available in a BigQuery table. + Table names are in the message attributes. Messages contain no data. - * - **ztf-loop** - - Use this stream for testing. Recent ZTF alerts are published to this topic - at a roughly constant rate of 1 per second. + * - .. centered:: *Testing Stream* + - -.. _data bigquery: + * - ztf-loop + - Recent messages from the ztf-alerts stream, published at a rate of 1 per second. + This stream is intended for testing and development use cases. -BigQuery Catalogs ------------------------- +BigQuery Tables +^^^^^^^^^^^^^^^ -.. list-table:: ZTF Datasets and Tables +.. list-table:: :class: tight-table :widths: 15 15 70 :header-rows: 1 @@ -57,41 +91,37 @@ BigQuery Catalogs - Table - Description - * - ztf_alerts - - alerts - - Complete alert packet, excluding image cutouts. - Same schema as the original alert, including nested and repeated fields. + * - ztf + - alerts_v4_02 + - Alert data for ZTF schema version 4.02. This table is an archive of the ztf-alerts Pub/Sub stream, + excluding image cutouts and metadata. + It has the same schema as the original alert bytes (except cutouts), including nested and repeated fields. + Equivalent tables exist for previous schema versions: alerts_v3_3, alerts_v3_1, alerts_v3_0, alerts_v1_8. - * - ztf_alerts + * - ztf - DIASource - - Alert packet data for the triggering source only. Including the object ID and a - list of source IDs for the previous sources included in the alert, - excluding cutouts and data for previous sources. - Flat schema. + - ZTF alert data of the DIA source that triggered the alert. Includes the object ID and a + list of source IDs for the previous sources included in the alert, excluding cutouts and + data for previous sources. The schema is flat. - * - ztf_alerts + * - ztf - SuperNNova - Results from a SuperNNova (Möller \& de Boissière, 2019) Type Ia supernova classification (binary). - * - ztf_alerts - - metadata - - Information recording Pitt-Google processing (e.g., message publish times, - bucket name and filename, etc.). - -.. _data cloud storage: - -Cloud Storage ------------------------- +Cloud Storage Buckets +^^^^^^^^^^^^^^^^^^^^^ -.. list-table:: ZTF Buckets +.. list-table:: :class: tight-table :widths: 40 60 :header-rows: 1 - * - Bucket Name + * - Bucket - Description - * - ardent-cycling-243415-ztf-alert_avros - - Contains the complete, original alert packets as Avro files. - Filename syntax is: `//.avro` + * - ardent-cycling-243415-ztf_alerts_v4_02 + - Alert data for ZTF schema version 4.02. This bucket is an Avro file archive of the ztf-alerts Pub/Sub stream, + including image cutouts and metadata. Each alert is stored as a separate Avro file. + The filename syntax is: `//.avro`. + Equivalent buckets exist for previous schema versions: v3_3, v3_1, v3_0, v1_8. diff --git a/docs/source/surveys/index.rst b/docs/source/surveys/index.rst new file mode 100644 index 0000000..0e7fc01 --- /dev/null +++ b/docs/source/surveys/index.rst @@ -0,0 +1,11 @@ +Surveys +======= + +Pitt-Google Broker serves data produced by the surveys and collaborations listed below. +Each linked page contains a brief overview and relevant facts, including specification of the data products served by Pitt-Google and links to the original documentation. +Note that data products may be served in multiple formats -- a complete listing of what's available through Pitt-Google is maintained at :ref:`listings`. + +.. toctree:: + :maxdepth: 1 + + ztf diff --git a/docs/source/surveys/ztf.rst b/docs/source/surveys/ztf.rst new file mode 100644 index 0000000..95db1a6 --- /dev/null +++ b/docs/source/surveys/ztf.rst @@ -0,0 +1,18 @@ +.. _survey ztf: + +Zwicky Transient Facility (ZTF) +=============================== + +ZTF is a wide-field, optical survey imaging in three bands: r, g, and i. +It produces an alert stream of astronomical transients detected by difference imaging at a rate of 10^5 - 10^6 alerts per night. +It scans the entire Northern sky every two days. + +Mission resources: + +- `ZTF Homepage `__ +- `ZTF Science Data System Explanatory Supplement `__ + +Data products: + +- Alert Stream. The schema can be viewed at ``__. + Complete documentation can be found at the mission resources above. diff --git a/pittgoogle/pubsub.py b/pittgoogle/pubsub.py index abb0560..c01e113 100644 --- a/pittgoogle/pubsub.py +++ b/pittgoogle/pubsub.py @@ -310,7 +310,7 @@ class Subscription: subscription.touch() # Pull a small batch of alerts - alerts = subscription.pull_batch(subscription, max_messages=4) + alerts = subscription.pull_batch(max_messages=4) ---- """