Where and how should we store our scientific data (both simulated and real)?

[bnord]

Let's discuss here a bit where we will store our data this summer and in the long run.

Current options

1. Here are our long-term options at Fermilab: https://computing.fnal.gov/data-storage-and-handling/
2. We also have some allocations on Wilson (250GB) and EAF (5TB).

Major questions

1. How much space will our data sets occupy? (i.e., we will need to start measuring sizes)
2. What lengths of time do we need it for? --> forever
3. What kind of backup do we need? How often? Maybe RAID is enough?
4. What format should we agree on for all the data? H5 is currently the idea.

We have an immediate need to start putting strong lensing data sets somewhere, both for basic storage and for machine learning runs in the next 1 weeks.

@AleksCipri @kadrlica @jsv1206 @AeRabelais @voetberg @MakesiP @antomaco @3ileenolan

[kadrlica]

Thanks for starting this discussion Brian!

Some context: Lenses will be downloaded in jpeg and FITS format. Conversion can be made to h5 of npy, but that will be as a subsequent process after download is complete. Space needs are estimated to be 10s of GBs within the next two weeks.

Must be on spinning disk. Direct accessibility from GPU resources is required. Very preferable if the system also has deep learning packages pre-installed/documented, so training can occur easily. The competitor is Google Colab, where I have ~infinite free GDrive space through UChicago.

[kadrlica]

That was our rough estimate. It'll shrink a bit if we go with the smaller cutouts.

For context, it was several TBs for the DES cutouts that Rob Morgan made.

[bnord]

@antomaco We should try to do a quick, order-of-magnitude estimate on the data sizes from what you've found so far.

For example, we could start with how many images are there for each data set, what is the approximate size for each image?

[amacoretta]

@bnord I put the sizes of all the datasets on the table we have in the project plan document last Friday. Does this work? Or do we need to know how many images are there? I can try to help with this but the datasets are quite large to download to my device.

[bnord]

the estimate of the data set sizes is good for now.

[amacoretta]

Awesome! I did notice that most of the datasets are zip files, so my estimate of the sizes is for the compressed data. I downloaded one of the zip files which was 2.8GB and after extracting the files in it, the final size was of 3.21 GB. I thought this would be good to know too.

[kadrlica]

For those with DES access, here is the format used for the DES and DELVE cutouts:
https://cdcvs.fnal.gov/redmine/projects/des-stronglens/wiki/Y6_Cutouts

[bnord]

we’re open to learning from and adjusting toward best practices — viz the choice of CSV, Pandas, Fits, H5.

Our current deep skies standard (but it's very new) is H5, because it seems to be the most versatile in terms of internal data structures, and I posit that there can be one h5 file per data set to minimize file management.

[bnord]

If there are other standards that we should look at, i'd be excited to see that. I could also imagine that deepskies in the long run (not for this project) could write an API for easy translation to a different file format should that be important for the community.

[kadrlica]

Is there anyone who can provide Antonella with some technical help on h5? I think h5 is "the format of the present", but I don't have any familiarity with it.

[bnord]

oh, I can def help with that @amacoretta

[bnord]

@jsv1206 You just uploaed some lens data to EAF. What format did you use? Also, how is our space looking? Should we get more?