AWS doesn’t support absolute paths

[griffin-h]

Describe the bug
I cannot upload reduced data products (tags "photometry" or "spectroscopy") when using AWS. When I try, I see the message "There was a problem processing your file" on site. I added a logger to tom_dataproducts.views.DataProductUploadView.form_valid to find that the exception raised is "This backend doesn't support absolute paths." Stack Overflow suggests I can fix this by replacing fits.getdata(data_product.data.path) with fits.getdata(data_product.data.name) in the methods of tom_dataproducts.data_processor.DataProcessor, but I can't get this to work either.

To Reproduce
Steps to reproduce the behavior:

1. Have a TOM configured to use AWS for storage
2. Go to a target page
3. Click on the Data tab
4. Choose a file to upload (example)
5. Choose the "Spectroscopy" tag
6. Click "Upload"

Expected behavior
The data file should be uploaded and processed. A spectrum should appear in the "Spectra" tab of that target.

Desktop (please complete the following information):

• OS: heroku-18
• Browser: Safari
• Version: 13.0.1
• Python Version: 3.7.0

Additional context
When I upload a raw data product (e.g., tag "Fits File") it goes in a directory TargetName/none/filename.fits at the top level of my AWS bucket.

[jfrostburke]

Just an update that we're running into this problem in snex2 as well. Files are being correctly uploaded to S3, but then the PhotometryProcessor can't access them. I copied some code from the django-storages docs and can correctly access the files:

(Pdb) print(data_product.data.path)
*** NotImplementedError: This backend doesn't support absolute paths.
(Pdb) print(data_product.data.name)
test/none/phot_example.csv
(Pdb) from django.core.files.storage import default_storage
(Pdb) f = default_storage.open(data_product.data.name, 'r')
(Pdb) f.read()
'time,filter,magnitude,error\n55959.06999999983,r,15.582,0.005\n55959.06999999983,V,15.676,0.007\n55959.06999999983,B,15.591,0.008\n'

We have not fully implemented this solution into snex2 yet, but the current plan is to write a custom processor using this.

[dmcollom]

I think the appropriate fix is to add some sort of abstracted method for reading data products that will handle multiple use-cases. I believe Lindy is on this, but I'll be helping as well. We'll get this figured out.

[rachel3834]

@phycodurus @jchate6 We should review this functionality to verify where our functionality stands.

[jchate6]

• Error uploading Data Products in TOM Demo LCOGT/tom-demo#62

[jchate6]

• Set up default_storage
• Configure photometry_processor to use default storage
• configure spectroscopy_processor to use default storage