Skip to content

Commit

Permalink
Merge pull request #676 from TOMToolkit/feature/tom-tom-target-sharing
Browse files Browse the repository at this point in the history 
Feature/tom tom target sharing
  • Loading branch information
@jchate6
jchate6 authored Oct 16, 2023
2 parents 1eec81a + 9d87392 commit 07c41f5
Show file tree
Hide file tree
Showing 23 changed files with 1,071 additions and 181 deletions.
100 changes: 71 additions & 29 deletions docs/managing_data/tom_direct_sharing.rst
View file
Original file line number Diff line number Diff line change
@@ -1,32 +1,74 @@
Sharing Data with Other TOMs
############################

TOM Toolkit does not yet support direct sharing between TOMs, however we hope to add this functionality soon.


.. Configuring your TOM to submit data to another TOM:
.. ***************************************************
..
.. You will need to add a ``DATA_SHARING`` configuration dictionary to your ``settings.py`` that gives the credentials
.. for the various TOMs with which you wish to share data.
..
.. .. code:: python
..
.. # Define the valid data sharing destinations for your TOM.
.. DATA_SHARING = {
.. 'tom-demo-dev': {
.. 'DISPLAY_NAME': os.getenv('TOM_DEMO_DISPLAY_NAME', 'TOM Demo Dev'),
.. 'BASE_URL': os.getenv('TOM_DEMO_BASE_URL', 'http://tom-demo-dev.lco.gtn/'),
.. 'USERNAME': os.getenv('TOM_DEMO_USERNAME', 'set TOM_DEMO_USERNAME value in environment'),
.. 'PASSWORD': os.getenv('TOM_DEMO_PASSWORD', 'set TOM_DEMO_PASSWORD value in environment'),
.. },
.. 'localhost-tom': {
.. # for testing; share with yourself
.. 'DISPLAY_NAME': os.getenv('LOCALHOST_TOM_DISPLAY_NAME', 'Local'),
.. 'BASE_URL': os.getenv('LOCALHOST_TOM_BASE_URL', 'http://127.0.0.1:8000/'),
.. 'USERNAME': os.getenv('LOCALHOST_TOM_USERNAME', 'set LOCALHOST_TOM_USERNAME value in environment'),
.. 'PASSWORD': os.getenv('LOCALHOST_TOM_PASSWORD', 'set LOCALHOST_TOM_PASSWORD value in environment'),
.. }
..
.. }
..
TOM Toolkit supports direct data sharing between TOMs.


Permissions:
************
To save data to a destination TOM your TOM will need to have access to a user account on that TOM with the correct
permissions. This is handled by your TOM's administrator as described below.

.. warning:: Any user who has permission to access the relevant target or data in your TOM will have permission to
submit that data to the destination TOM once DATA_SHARING is configured.


Configuring your TOM to submit data to another TOM:
***************************************************

You will need to add a ``DATA_SHARING`` configuration dictionary to your ``settings.py`` that gives the credentials
for the various TOMs with which you wish to share data. This should be the same ``DATA_SHARING`` dictionary that is used
to :doc:`/managing_data/stream_pub_sub` such as `Hermes <https://hermes.lco.global>`_.

.. code:: python
# Define the valid data sharing destinations for your TOM.
DATA_SHARING = {
'not-my-tom': {
# For sharing data with another TOM
'DISPLAY_NAME': os.getenv('NOT_MY_TOM_DISPLAY_NAME', 'Not My Tom'),
'BASE_URL': os.getenv('NOT_MY_TOM_BASE_URL', 'http://notmytom.com/'),
'USERNAME': os.getenv('NOT_MY_TOM_USERNAME', 'set NOT_MY_TOM_USERNAME value in environment'),
'PASSWORD': os.getenv('NOT_MY_TOM_PASSWORD', 'set NOT_MY_TOM_PASSWORD value in environment'),
},
'localhost-tom': {
# for testing; share with yourself
'DISPLAY_NAME': os.getenv('LOCALHOST_TOM_DISPLAY_NAME', 'Local'),
'BASE_URL': os.getenv('LOCALHOST_TOM_BASE_URL', 'http://127.0.0.1:8000/'),
'USERNAME': os.getenv('LOCALHOST_TOM_USERNAME', 'set LOCALHOST_TOM_USERNAME value in environment'),
'PASSWORD': os.getenv('LOCALHOST_TOM_PASSWORD', 'set LOCALHOST_TOM_PASSWORD value in environment'),
}
}
Receiving Shared Data:
**********************

Reduced Datums:
---------------
When your TOM receives a new ``ReducedDatum`` from another TOM it will be saved to your TOM's database with its source
set to the name of the TOM that submitted it. Currently, only Photometry data can be directly shared between
TOMS and a ``Target`` with a matching name or alias must exist in both TOMS for sharing to take place.

Data Products:
--------------
When your TOM receives a new ``DataProduct`` from another TOM it will be saved to your TOM's database / storage and run
through the appropriate :doc:`data_processor </managing_data/customizing_data_processing>` pipeline. Only data products
associated with a ``Target`` with a name or alias that matches that of a target in the destination TOM will be shared.

Targets:
--------
When your TOM receives a new ``Target`` from another TOM it will be saved to your TOM's database. If the target's name
or alias doesn't match that of a target that already exists in the database, a new target will be created and added to a
new ``TargetList`` called "Imported from <TOM Name>".

Target Lists:
-------------
When your TOM receives a new ``TargetList`` from another TOM it will be saved to your TOM's database. If the targets in
the ``TargetList`` are also shared, but already exist in the destination TOM, they will be added to the new
``TargetList``.






4 changes: 2 additions & 2 deletions setup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -49,11 +49,11 @@
'plotly~=5.0',
'python-dateutil~=2.8',
'requests~=2.25',
'responses~=0.23',
'specutils~=1.8',
],
extras_require={
'test': ['factory_boy>=3.2.1,<3.4.0'],
'test': ['factory_boy>=3.2.1,<3.4.0',
'responses~=0.23'],
'docs': [
'recommonmark~=0.7',
'sphinx>=4,<8',
Expand Down
48 changes: 24 additions & 24 deletions tom_base/settings.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -237,30 +237,30 @@
}

# Configuration for the TOM/Kafka Stream receiving data from this TOM
DATA_SHARING = {
'hermes': {
'DISPLAY_NAME': os.getenv('HERMES_DISPLAY_NAME', 'Hermes'),
'BASE_URL': os.getenv('HERMES_BASE_URL', 'https://hermes.lco.global/'),
'CREDENTIAL_USERNAME': os.getenv('SCIMMA_CREDENTIAL_USERNAME',
'set SCIMMA_CREDENTIAL_USERNAME value in environment'),
'CREDENTIAL_PASSWORD': os.getenv('SCIMMA_CREDENTIAL_PASSWORD',
'set SCIMMA_CREDENTIAL_PASSWORD value in environment'),
'USER_TOPICS': ['hermes.test', 'tomtoolkit.test']
},
'tom-demo-dev': {
'DISPLAY_NAME': os.getenv('TOM_DEMO_DISPLAY_NAME', 'TOM Demo Dev'),
'BASE_URL': os.getenv('TOM_DEMO_BASE_URL', 'http://tom-demo-dev.lco.gtn/'),
'USERNAME': os.getenv('TOM_DEMO_USERNAME', 'set TOM_DEMO_USERNAME value in environment'),
'PASSWORD': os.getenv('TOM_DEMO_PASSWORD', 'set TOM_DEMO_PASSWORD value in environment'),
},
'localhost-tom': {
# for testing; share with yourself
'DISPLAY_NAME': os.getenv('LOCALHOST_TOM_DISPLAY_NAME', 'Local'),
'BASE_URL': os.getenv('LOCALHOST_TOM_BASE_URL', 'http://127.0.0.1:8000/'),
'USERNAME': os.getenv('LOCALHOST_TOM_USERNAME', 'set LOCALHOST_TOM_USERNAME value in environment'),
'PASSWORD': os.getenv('LOCALHOST_TOM_PASSWORD', 'set LOCALHOST_TOM_PASSWORD value in environment'),
}
}
# DATA_SHARING = {
# 'hermes': {
# 'DISPLAY_NAME': os.getenv('HERMES_DISPLAY_NAME', 'Hermes'),
# 'BASE_URL': os.getenv('HERMES_BASE_URL', 'https://hermes.lco.global/'),
# 'CREDENTIAL_USERNAME': os.getenv('SCIMMA_CREDENTIAL_USERNAME',
# 'set SCIMMA_CREDENTIAL_USERNAME value in environment'),
# 'CREDENTIAL_PASSWORD': os.getenv('SCIMMA_CREDENTIAL_PASSWORD',
# 'set SCIMMA_CREDENTIAL_PASSWORD value in environment'),
# 'USER_TOPICS': ['hermes.test', 'tomtoolkit.test']
# },
# 'tom-demo-dev': {
# 'DISPLAY_NAME': os.getenv('TOM_DEMO_DISPLAY_NAME', 'TOM Demo Dev'),
# 'BASE_URL': os.getenv('TOM_DEMO_BASE_URL', 'http://tom-demo-dev.lco.gtn/'),
# 'USERNAME': os.getenv('TOM_DEMO_USERNAME', 'set TOM_DEMO_USERNAME value in environment'),
# 'PASSWORD': os.getenv('TOM_DEMO_PASSWORD', 'set TOM_DEMO_PASSWORD value in environment'),
# },
# 'localhost-tom': {
# # for testing; share with yourself
# 'DISPLAY_NAME': os.getenv('LOCALHOST_TOM_DISPLAY_NAME', 'Local'),
# 'BASE_URL': os.getenv('LOCALHOST_TOM_BASE_URL', 'http://127.0.0.1:8000/'),
# 'USERNAME': os.getenv('LOCALHOST_TOM_USERNAME', 'set LOCALHOST_TOM_USERNAME value in environment'),
# 'PASSWORD': os.getenv('LOCALHOST_TOM_PASSWORD', 'set LOCALHOST_TOM_PASSWORD value in environment'),
# }
# }

TOM_CADENCE_STRATEGIES = [
'tom_observations.cadences.retry_failed_observations.RetryFailedObservationsStrategy',
Expand Down
31 changes: 2 additions & 29 deletions tom_dataproducts/forms.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
from tom_dataproducts.models import DataProductGroup, DataProduct
from tom_observations.models import ObservationRecord
from tom_targets.models import Target
from tom_dataproducts.alertstreams.hermes import get_hermes_topics
from tom_dataproducts.sharing import get_sharing_destination_options


DATA_TYPE_OPTIONS = (('photometry', 'Photometry'),
Expand Down Expand Up @@ -65,33 +65,6 @@ class DataShareForm(forms.Form):
widget=forms.HiddenInput()
)

def get_sharing_destination_options(self):
"""
Build the Display options and headers for the dropdown form for choosing sharing topics.
Customize for a different selection experience.
:return: Tuple: Possible Destinations and their Display Names
"""
choices = []
try:
for destination, details in settings.DATA_SHARING.items():
new_destination = [details.get('DISPLAY_NAME', destination)]
if details.get('USER_TOPICS', None):
# If topics exist for a destination (Such as HERMES) give topics as sub-choices
# for non-selectable Destination
if destination == "hermes":
destination_topics = get_hermes_topics()
else:
destination_topics = details['USER_TOPICS']
topic_list = [(f'{destination}:{topic}', topic) for topic in destination_topics]
new_destination.append(tuple(topic_list))
else:
# Otherwise just use destination as option
new_destination.insert(0, destination)
choices.append(tuple(new_destination))
except AttributeError:
pass
return tuple(choices)

def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.fields['share_destination'].choices = self.get_sharing_destination_options()
self.fields['share_destination'].choices = get_sharing_destination_options()
Loading

0 comments on commit 07c41f5

Please sign in to comment.