Releases: mwvgroup/Pitt-Google-Broker
Releases · mwvgroup/Pitt-Google-Broker
v0.5
v0.4
Major Updates and New Features:
- Accommodate generic survey: (1) Renames resources. (2) Adds a "survey" keyword. (3) Adds a schema_maps module and configures all components to use it.
- Creates a broker_utils python package and configures broker components to use it. Includes modules: beam_transforms, consumer_sim, data_utils, gcp_utils, schema_maps
- Listen for Kafka topics: Consumer VM now polls for available Kafka topics and begins consuming once its assigned topic becomes active. Prior to this, the broker could not be started before the topic was active.
- Documentation: updates for above changes, and reorganization in preparation for integration with ReadtheDocs.
v0.3
Implements the following new items:
- Consumer: VM running a Kafka -> Pub/Sub connector
- BQ sink: Dataflow job storing alerts to BigQuery
- Avro storage sink: Cloud Function storing alerts to Cloud Storage as Avro files
- Value-added pipeline: Dataflow job running Salt2 fits on likely extragalactic events
- Night Conductor: VM running a set of scripts that start/stop the broker each night/morning
- Consumer Simulator: Python module allowing user to manually simulate the Consumer by publishing alert packets to a Pub/Sub topic
- Broker Testing Instances: tutorial plus code integrated with all items above that allows users (PGB members) to create and use a testing/development instance of the broker