Merge branch 'dev' into dependabot/pip/dev/dramatiq-1.16.0

CODE_OF_CONDUCT.md

@@ -0,0 +1,128 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, religion, or sexual identity
+and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment for our
+community include:
+
+* Demonstrating empathy and kindness toward other people
+* Being respectful of differing opinions, viewpoints, and experiences
+* Giving and gracefully accepting constructive feedback
+* Accepting responsibility and apologizing to those affected by our mistakes,
+  and learning from the experience
+* Focusing on what is best not just for us as individuals, but for the
+  overall community
+
+Examples of unacceptable behavior include:
+
+* The use of sexualized language or imagery, and sexual attention or
+  advances of any kind
+* Trolling, insulting or derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or email
+  address, without their explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Enforcement Responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of
+acceptable behavior and will take appropriate and fair corrective action in
+response to any behavior that they deem inappropriate, threatening, offensive,
+or harmful.
+
+Community leaders have the right and responsibility to remove, edit, or reject
+comments, commits, code, wiki edits, issues, and other contributions that are
+not aligned to this Code of Conduct, and will communicate reasons for moderation
+decisions when appropriate.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when
+an individual is officially representing the community in public spaces.
+Examples of representing our community include using an official e-mail address,
+posting via an official social media account, or acting as an appointed
+representative at an online or offline event.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the community leaders responsible for enforcement at [email protected]
+.
+All complaints will be reviewed and investigated promptly and fairly.
+
+All community leaders are obligated to respect the privacy and security of the
+reporter of any incident.
+
+## Enforcement Guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining
+the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+
+**Community Impact**: Use of inappropriate language or other behavior deemed
+unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing
+clarity around the nature of the violation and an explanation of why the
+behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+
+**Community Impact**: A violation through a single incident or series
+of actions.
+
+**Consequence**: A warning with consequences for continued behavior. No
+interaction with the people involved, including unsolicited interaction with
+those enforcing the Code of Conduct, for a specified period of time. This
+includes avoiding interactions in community spaces as well as external channels
+like social media. Violating these terms may lead to a temporary or
+permanent ban.
+
+### 3. Temporary Ban
+
+**Community Impact**: A serious violation of community standards, including
+sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public
+communication with the community for a specified period of time. No public or
+private interaction with the people involved, including unsolicited interaction
+with those enforcing the Code of Conduct, is allowed during this period.
+Violating these terms may lead to a permanent ban.
+
+### 4. Permanent Ban
+
+**Community Impact**: Demonstrating a pattern of violation of community
+standards, including sustained inappropriate behavior,  harassment of an
+individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within
+the community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 2.0, available at
+https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
+
+Community Impact Guidelines were inspired by [Mozilla's code of conduct
+enforcement ladder](https://github.com/mozilla/diversity).
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see the FAQ at
+https://www.contributor-covenant.org/faq. Translations are available at
+https://www.contributor-covenant.org/translations.

CONTRIBUTING.md

@@ -0,0 +1,6 @@
+## How to contribute
+
+If you'd like to contribute to the TOM Toolkit, first of all, thanks! Secondly, we
+have a [contribution guide](https://tom-toolkit.readthedocs.io/en/stable/introduction/contributing.html) that
+you might find helpful. We are particularly interested in the contribution of
+observation and alert modules.

README.md

@@ -61,10 +61,13 @@ have a [contribution guide](https://tom-toolkit.readthedocs.io/en/stable/introdu
 you might find helpful. We are particularly interested in the contribution of
 observation and alert modules.
 
+### Community Code of Conduct
+We encourage developers to maintian a lively and civil discourse either on github or in our slack workspace.
+Please review our [code of conduct](CODE_OF_CONDUCT.md) for information about what is considered unacceptable behavior and how to report it.
+
 ## Developer information
 For development information targeted at the maintainers of the project, please see [README-dev.md](README-dev.md).
 
-
 ## Plugins
 
 ### tom_alerts_dash

docs/managing_data/stream_pub_sub.rst

@@ -12,7 +12,9 @@ TOM Toolkit supports publishing data to a Kafka stream such as `Hermes <https://
 
 When sharing photometry data via Hermes, the TOM publishes the data to be shared to a topic on the HOPSKOTCH
 Kafka stream. At this time, only photometry data is supported by TOM Toolkit. To submit via the Hermes API, you will
-need to copy your Hermes API Key from your Hermes profile page.
+need to copy your Hermes API Key from your Hermes profile page. When hermes sharing is configured, you will also see
+buttons to open your data in hermes with the form pre-filled - this is a good option if you want to make slight changes
+to your message or data before sharing.
 
 
 Configuring your TOM to Publish Data to a stream:
@@ -29,7 +31,8 @@ for the various streams with which you wish to share data.
            'DISPLAY_NAME': os.getenv('HERMES_DISPLAY_NAME', 'Hermes'),
            'BASE_URL': os.getenv('HERMES_BASE_URL', 'https://hermes.lco.global/'),
            'HERMES_API_KEY': os.getenv('HERMES_API_KEY', 'set HERMES_API_KEY value in environment'),
-           'USER_TOPICS': ['hermes.test', 'tomtoolkit.test']
+           'DEFAULT_AUTHORS': os.getenv('HERMES_DEFAULT_AUTHORS', 'set your default authors here'),
+           'USER_TOPICS': ['hermes.test', 'tomtoolkit.test']  # You must have write permissions on these topics
         },
    }
 

tom_dataproducts/alertstreams/hermes.py

@@ -2,6 +2,7 @@
 from dateutil.parser import parse
 
 from django.conf import settings
+from django.core.cache import cache
 
 # from hop.io import Metadata
 
@@ -28,43 +29,88 @@ def __init__(self, title='', submitter='', authors='', message='', topic='hermes
         self.extra_info = kwargs
 
 
-def publish_photometry_to_hermes(message_info, datums, **kwargs):
+def publish_to_hermes(message_info, datums, targets=Target.objects.none(), **kwargs):
     """
-    Submits a typical hermes photometry alert using the datums supplied to build a photometry table.
+    Submits a typical hermes alert using the photometry and targets supplied to build a photometry table.
     -- Stores an AlertStreamMessage connected to each datum to show that the datum has previously been shared.
     :param message_info: HERMES Message Object created with BuildHermesMessage
-    :param datums: Queryset of Reduced Datums to be built into table.
+    :param datums: Queryset of Reduced Datums to be built into table. (Will also pull in targets)
+    :param targets: Queryset of Targets to be built into table.
     :return: response
     """
+    if 'BASE_URL' not in settings.DATA_SHARING['hermes']:
+        return {'message': 'BASE_URL is not set for hermes in the settings.py DATA_SHARING section'}
+    if 'HERMES_API_KEY' not in settings.DATA_SHARING['hermes']:
+        return {'message': 'HERMES_API_KEY is not set for hermes in the settings.py DATA_SHARING section'}
+
     stream_base_url = settings.DATA_SHARING['hermes']['BASE_URL']
     submit_url = stream_base_url + 'api/v0/' + 'submit_message/'
     # You will need your Hermes API key. This can be found on your Hermes profile page.
     headers = {'Authorization': f"Token {settings.DATA_SHARING['hermes']['HERMES_API_KEY']}"}
 
+    alert = create_hermes_alert(message_info, datums, targets, **kwargs)
+
+    try:
+        response = requests.post(url=submit_url, json=alert, headers=headers)
+        response.raise_for_status()
+        # Only mark the datums as shared if the sharing was successful
+        hermes_alert = AlertStreamMessage(topic=message_info.topic, exchange_status='published')
+        hermes_alert.save()
+        for tomtoolkit_photometry in datums:
+            tomtoolkit_photometry.message.add(hermes_alert)
+    except Exception:
+        return response
+
+    return response
+
+
+def preload_to_hermes(message_info, reduced_datums, targets):
+    stream_base_url = settings.DATA_SHARING['hermes']['BASE_URL']
+    preload_url = stream_base_url + 'api/v0/submit_message/preload/'
+    # You will need your Hermes API key. This can be found on your Hermes profile page.
+    headers = {'Authorization': f"Token {settings.DATA_SHARING['hermes']['HERMES_API_KEY']}"}
+
+    alert = create_hermes_alert(message_info, reduced_datums, targets)
+    try:
+        response = requests.post(url=preload_url, json=alert, headers=headers)
+        response.raise_for_status()
+        return response.json()['key']
+    except Exception as ex:
+        logger.error(repr(ex))
+        logger.error(response.json())
+
+    return ''
+
+
+def create_hermes_alert(message_info, datums, targets=Target.objects.none(), **kwargs):
     hermes_photometry_data = []
-    hermes_target_list = []
-    hermes_alert = AlertStreamMessage(topic=message_info.topic, exchange_status='published')
-    hermes_alert.save()
+    hermes_target_dict = {}
+
     for tomtoolkit_photometry in datums:
-        if tomtoolkit_photometry.target.name not in [target['name'] for target in hermes_target_list]:
-            hermes_target_list.append(create_hermes_target_table_row(tomtoolkit_photometry.target, **kwargs))
-        tomtoolkit_photometry.message.add(hermes_alert)
+        if tomtoolkit_photometry.target.name not in hermes_target_dict:
+            hermes_target_dict[tomtoolkit_photometry.target.name] = create_hermes_target_table_row(
+                tomtoolkit_photometry.target, **kwargs)
         hermes_photometry_data.append(create_hermes_phot_table_row(tomtoolkit_photometry, **kwargs))
+
+    # Now go through the targets queryset and ensure we have all of them in the table
+    # This is needed since some targets may have no corresponding photometry datums but that is still valid to share
+    for target in targets:
+        if target.name not in hermes_target_dict:
+            hermes_target_dict[target.name] = create_hermes_target_table_row(target, **kwargs)
+
     alert = {
         'topic': message_info.topic,
         'title': message_info.title,
         'submitter': message_info.submitter,
         'authors': message_info.authors,
         'data': {
-            'targets': hermes_target_list,
+            'targets': list(hermes_target_dict.values()),
             'photometry': hermes_photometry_data,
             'extra_data': message_info.extra_info
         },
         'message_text': message_info.message,
     }
-
-    response = requests.post(url=submit_url, json=alert, headers=headers)
-    return response
+    return alert
 
 
 def create_hermes_target_table_row(target, **kwargs):
@@ -128,16 +174,17 @@ def get_hermes_topics(**kwargs):
     Extend this method to restrict topics for individual users.
     :return: List of writable topics available for TOM.
     """
-    try:
-        stream_base_url = settings.DATA_SHARING['hermes']['BASE_URL']
-        submit_url = stream_base_url + "api/v0/profile/"
-        headers = {'Authorization': f"Token {settings.DATA_SHARING['hermes']['HERMES_API_KEY']}"}
-
-        response = requests.get(url=submit_url, headers=headers)
-
-        topics = response.json()['writable_topics']
-    except (KeyError, requests.exceptions.JSONDecodeError):
-        topics = settings.DATA_SHARING['hermes']['USER_TOPICS']
+    topics = cache.get('hermes_writable_topics', [])
+    if not topics:
+        try:
+            stream_base_url = settings.DATA_SHARING['hermes']['BASE_URL']
+            submit_url = stream_base_url + "api/v0/profile/"
+            headers = {'Authorization': f"Token {settings.DATA_SHARING['hermes']['HERMES_API_KEY']}"}
+            response = requests.get(url=submit_url, headers=headers)
+            topics = response.json()['writable_topics']
+            cache.set('hermes_writable_topics', topics, 86400)
+        except (KeyError, requests.exceptions.JSONDecodeError):
+            pass
     return topics
 
 

tom_dataproducts/forms.py

@@ -54,7 +54,7 @@ def __init__(self, *args, **kwargs):
 class DataShareForm(forms.Form):
     share_destination = forms.ChoiceField(required=True, choices=[], label="Destination")
     share_title = forms.CharField(required=False, label="Title")
-    share_message = forms.CharField(required=False, label="Message", widget=forms.Textarea())
+    share_message = forms.CharField(required=False, label="Message", widget=forms.Textarea(attrs={'rows': 4}))
     share_authors = forms.CharField(required=False, widget=forms.HiddenInput())
     data_type = forms.ChoiceField(required=False, choices=DATA_TYPE_OPTIONS, label="Data Type")
     target = forms.ModelChoiceField(

tom_dataproducts/processors/atlas_processor.py

@@ -21,7 +21,7 @@ def process_data(self, data_product):
         ingestion
         :type data_product: DataProduct
 
-        :returns: python list of 2-tuples, each with a timestamp and corresponding data
+        :returns: python list of 3-tuples, each with a timestamp and corresponding data, and source
         :rtype: list
         """
 
@@ -37,6 +37,7 @@ def _process_photometry_from_plaintext(self, data_product):
         Processes the photometric data from a plaintext file into a list of dicts. File is read using astropy as
         specified in the below documentation. The file is expected to be a multi-column delimited space delimited
         text file, as produced by the ATLAS forced photometry service at https://fallingstar-data.com/forcedphot
+        See https://fallingstar-data.com/forcedphot/resultdesc/ for a description of the output format.
 
         The header looks like this:
         ###MJD   m   dm  uJy   duJy F err chi/N   RA  Dec   x   y  maj  min   phi  apfit mag5sig Sky   Obs
@@ -48,6 +49,7 @@ def _process_photometry_from_plaintext(self, data_product):
         :rtype: list
         """
         photometry = []
+        signal_to_noise_cutoff = 3.0  # cutoff to turn magnitudes into non-detection limits
 
         data = astropy.io.ascii.read(data_product.data.path)
         if len(data) < 1:
@@ -60,10 +62,19 @@ def _process_photometry_from_plaintext(self, data_product):
                 time.format = 'datetime'
                 value = {
                     'timestamp': time.to_datetime(timezone=utc),
-                    'magnitude': float(datum['m']),
-                    'magnitude_error': float(datum['dm']),
-                    'filter': str(datum['F'])
+                    'filter': str(datum['F']),
+                    'error': float(datum['dm']),
+                    'telescope': 'ATLAS',
                 }
+                # If the signal is in the noise, set the non-detection limit to the
+                # absolute value of the reported magnitude.
+                # see https://fallingstar-data.com/forcedphot/resultdesc/
+                signal_to_noise = abs(float(datum['uJy']))/abs(float(datum['duJy']))
+                if signal_to_noise <= signal_to_noise_cutoff:
+                    value['limit'] = abs(float(datum['m']))
+                else:
+                    value['magnitude'] = abs(float(datum['m']))
+
                 photometry.append(value)
         except Exception as e:
             raise InvalidFileFormatException(e)