SW-DES data location at the Zooniverse end

[anumore]

Where and how to transfer the SW-DES beta data and then actual data for the launch?
For CFHTLS, we transferred the data to amazon servers using globus online interface.

[aprajita]

I already spoke to cam about this, as we cannot do it trough the project
builder. they will help us when the time comes. I don't think they use that
anymore but will check.

On Wednesday, 11 May 2016, Anupreeta [email protected] wrote:

Where and how to transfer the SW-DES beta data and then actual data for
the launch?
For CFHTLS, we transferred the data to amazon servers using globus online
interface.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#230

[anumore]

yes, i remember that you mentioned something like that. but from the DES side, we would like to have more specific info for how to go about this if possible, so that we can begin setting things up in parallel to other tasks.

[aprajita]

tell me how many images you want for beta testing and i'll enquire

On 12 May 2016, at 00:39, Anupreeta [email protected] wrote:

yes, i remember that you mentioned something like that. but from the DES side, we would like to have more specific info for how to go about this if possible, so that we can begin setting things up in parallel to other tasks.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub

[anumore]

just to give you a number i'd say 500-1000 but we don't know yet for sure. also, for the beta sample, we can just use some ftp option but its good to have set things up and test with the beta sample already. thanks!

[aprajita]

Cam said this "Yeah, having the data on s3 and providing a CSV manifest (SRC URL, metadata) to us and we can import it. We can help with the s3 transfer too if needed" so should be straight forward.

If we want randomised subject IDs in the metadata - we will need to take of that from our end before delivery - these will then given a subject Id int eh db of the form .e.g. 2056727. if we were to only people uploading at the time all the panoptest subject IDs would be sequential.

e.g. one of the test data i just uploaded looks like
https://www.zooniverse.org/projects/aprajita/swdemo/talk/subjects/2056728
but it's metadata ID is '2'

[anumore]

Thanks @aprajita
What is s3? I don't quite understand the reply though. Does Cam mean we can do the same thing as before?

[aprajita]

yes, s3 i think is the amazon data cloud
http://docs.aws.amazon.com/AmazonS3/latest/dev/Welcome.html

On 17 May 2016 at 23:57, Anupreeta [email protected] wrote:

Thanks @aprajita https://github.com/aprajita

What is s3? I don't quite understand the reply though. Does Cam mean we
can do the same thing as before?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#230 (comment)

[aprajita]

they can help with teh S3 transfer too
not sure if it's different to what you did before