Directory Naming Conventions
← Previous topic: File Naming Conventions | Next topic: Time Conventions →
When data collection first started at the CIL with a single station at Duck, NC, it was not difficult to remember where the data were stored or what types of images were being collected. However, with continuing multi-year then multi-decade collection at Duck, then the expansion into new stations spanning many time zones, and the growth of new types of data collection like brightest and darkest images or pixel time stacks, it became evident that if a clear organization was not developed and good conventions adopted, we would often struggling to keep track of everything. New users will undoubtedly face this same issue, especially if the work is being done by graduate students who will eventually leave and take with them the only memory of where and what everything is.
Thus if you expect to collect more than a few short sets of data, it is well worth while considered how to you should organize your data archive. The following notes represent the conventions that have been developed at the CIL based on many decades of experience. At times they can seem a bit verbose, but they allow us to use tools that will always successfully find our files and metadata based on simple inputs. The notes below were first written during the 2010 Surf Zone Optics experiment, updated somewhat, but serve as an example of how an example stations collections were set up.
Data are stored on the CIL ftp site, cil-ftp.coas.oregonstate.edu and can be downloaded using normal anonymous ftp (with a valid RFC822 format email address as the password). The decision to store on an ftp server was intentional - we wanted other scientists to be able to download data of interest without our help.
Data for any Argus station is organized using the following algorithm,
/ftp/pub/site/year/camera/day/file.
For example, images for the recent Duck Argus station (called 'argus02b') for C2 of March 2, 2015 would be located in
/ftp/pub/argus02b/2015/c2/062_Mar.02.
Within that structure, Argus files are named using a long (and useful) convention of
{epochTime longHumanTimeDate stationString cameraString imageTypeString fileFormat}.
For example:
1425294001.Mon.Mar.02_11_00_01.GMT.2015.argus02b.c1.timex.jpg.
is a time exposure image for camera C1 collected on March 2, 2015 at 11:00:01 GMT. All Argus data are saved in GMT (because we have stations in many time zones). Note that images also have timestamp and other information imprinted on the top and bottom borders. These are written in local standard time (note that merged images stored as camera “Cx” are only listed in GMT time – more on Cx below). Be careful to not confuse these (image stamp in EST, filename in GMT = EST+5 for Duck). Our primary time reference is epoch time, a computer standard that is the number of seconds since January 1, 1970 (GMT). This is the 10-digit leading number in the filename. Routines to convert between epoch to matlab’s datenum are part of the CIRN repositories (epoch2Matlab and matlab2Epoch).
Six types of images are collected:
- Snapshot
- Timex – average of 2Hz frames collected over ten-minute period
- Var – standard deviation of same image sample
- Bright – brightest that each pixel gets over the same period
- Dark - darkest that each pixel gets over the same period
- Rundark - darkest computing using a running average algorithm to suppress noise
In addition, daytimex images average all of the timex images for a day.
Data direct from the Argus station are stored under the corresponding camera directory, for example c1 or c5. This includes all of the above image products that are saved as oblique images, as well as time stacks. However processed data often involves multiple cameras. By convention, these products are stored in a virtual camera directory called cx. Examples of these products included rectified maps computed by merging and remapping oblique images from multiple cameras, or more sophisticated analysis products from cBathy or vBar calculations.
Merged images like Figure 1, are automatically processed and use the same naming conventions as above but with an additional suffix ‘merge’. For example,
1432908003.Fri.May.29_14_00_03.GMT.2015.argus02b.cx.bright.merge.png
is a merged brightest image from May 29 at 14:00:-3 GMT (~0900 EST or 1000 EDT). These are stored as png images. Note that the merged image data are also stored as .mat files so you can create your own figures, for example using ‘imagesc(y,x,I)’.
{The current format may be ‘imagesc(XYZ(:,1), XYZ(:,2), RAW)}
Examples of processed pixel instruments include cBathy, vBar and runup where the instrument type replaces timeTypeString in the file naming convention above. More information of pixel instruments will be added to the wiki later.
Figure 1. Merged snapshot showing the fields of view of the six primary cameras installed during the upgrade to argus02b in 2015.
← Previous topic: File Naming Conventions | Next topic: Time Conventions →