Generate fullfield products

[craig-willis]

We continue to have problems with the fullfield pipeline. This is a best-effort task to generate as many fullfield products as possible along with trait data.

• Implement additional logging in rule-checker and field-mosaic
• Generate remaining full-fields (brute force kick-off)
• Evaluate status of existing files (remove pre-break files)
• Troubleshooting/diagnosis of fullfield and trait generation processes

[max-zilla]

we currently have around half of the full-field products generated. this is main issue following #393

[max-zilla]

stereo rgb, flir IR have completed generation @ 80% threshold and need to be evaluted for exhaustiveness.

Remaining full field (laser 3D heightmap stitch) on hold pending fix of the heightmap PLY -> BMP conversion #417

[max-zilla]

• should queue up remaining files that were skipped due to fullfield eligibility, but not for this data release.
• how many of these fullfields were skipped given the scan program name? go through scan program names, dates, metadata and the rulechecker DB and determine what is EXPECTED BUT MISSING.

[dlebauer]

• @max-zilla should use this as a chance to review the different possible exceptions and places in the pipeline that can fail
• currently 113 rgb, 101 flir full fields for 2016-2017
  • need to make sure that 2017 summer flir is excluded from pipeline

[craig-willis]

On terra-clowder from @max-zilla a dump of the rulemonitor db with all the unique FF scans we have and their counts as of friday 3/16 /home/mburnet2/scans.csv.

[craig-willis]

Identified problem with fullfield process due to bin2tif (and likely ir) posting metadata after the files are uploaded. Tentative fix for this in bin2tif is providing more complete, though incomplete still, fullfields. First image generated using this method is 2/7/2018.

[robkooper]

What triggers the run, the metadata or the actual image? We should probably make sure people understand this is not atomic, and if they need both pieces they should make sure both are available, we probably should also have some sort of contract that we do things is the same order (i.e. always upload image then metadata) that way people can trigger of just metadata and not have to trigger of both metadata and image to make sure they know when both are ready.

[max-zilla]

Rulechecker is a switchboard for many extractors and triggers from a file being added to a dataset. Several of our extractors upload files before attaching metadata, so rulechecker would trigger before metadata was uploaded and respond by saying md is missing. If md is added as a final step consistently we could trigger on that, but not every extractor uploads metadata. I briefly discussed with Craig whether we just rework things for extractors to manually trigger downstream extractors explicitly without using rulechecker or a dataset file added/metadata added listener.

[craig-willis]

Update - RGB fullfields are (generally) being generated for the new season starting 5/1. We are missing fullfields for 5/13 and 5/14, likely due to connection error in globus-monitor (need to monitor this component). We are also missing fullfields for 5/2 and 5/4, likely due to other pipeline failures.

After manually triggering, there is an entry in the fullfield extractor log for 5/2:

2018-05-14 19:38:21,572 INFO    : extractor - [5aece09d4f0c026b45768bab] RGB GeoTIFFs - 2018-05-02__09-56-56-643 - SKIP: /home/extractor/sites/ua-mac/Level_1/fullfield/2018-05-02/fullfield_L1_ua-mac_2018-05-02_rgb_stereovis_ir_sensors_fullfield_sorghum6_shade_may2018.vrt already exists; ending process

So the VRT already exists but the TIF hasn't been generated.

In the case of 5/4, we have over 99% coverage in rgb-geotiff, but rulechecker reports otherwise:

2018-05-14 18:58:51,541 INFO    : extractor - [5af020434f0c026b4591c975] RGB GeoTIFFs - 2018-05-04__09-50-20-762 - found 5978/9284 (64.39%) necessary geotiffs (184 in this scan)
2018-05-14 18:58:51,541 INFO    : extractor - [5af020434f0c026b4591c975] RGB GeoTIFFs - 2018-05-04__09-50-20-762 - SKIP: 80 tolerance percentage not met for fullfield.

[craig-willis]

Lookin in detail at the missing files for recent dates

2018-05-12
raw_data/stereoTop/2018-05-12 has 9089 directories
Level_1/rgb_geotiff/2018-05-12 has 9088 directories
2018-05-12__09-27-13-683 is missing

TIMESTAMP=2018-05-12__09-27-13-683
curl -s "https://terraref.ncsa.illinois.edu/clowder/api/datasets?key=$CLOWDER_KEY&&exact=true&title=stereoTop%20-%20$TIMESTAMP"
[]

Raw dataset doesn't exist in Clowder.

2018/5/11
raw/stereoTop = 9100
Level_1/rgb_geotiff = 9072
28 missing datasets

TIMESTAMP=2018-05-10__13-30-51-481
curl -s "https://terraref.ncsa.illinois.edu/clowder/api/datasets?key=$CLOWDER_KEY&&exact=true&title=stereoTop%20-%20$TIMESTAMP"
[]

Raw dataset doesn't exist in Clowder. Of the 28 datasets, 27 do not exist in Clowder. This suggests that there is a failure in registering the initial raw data.

[craig-willis]

We've determined that this is likely caused by the somewhat rare case that files in a dataset are split across two transfers this is now tracked in #446

[max-zilla]

I just deployed new fullfield with higher threshold. Once we see those, we can close this.
#447

[craig-willis]

We are missing fullfields for 5/7, 5/25, and 5/26. I've resubmitted all stereotop datasets to rule checker for these dates and we have near complete coverage on filesystem (rgb_geotiff) but the counts in rulemonitor do not match. See https://docs.google.com/spreadsheets/d/1emUW6xu9i0zMh1jUupgqPmIIz_IVjDv-DZWEKbZNCz8/.

I'd like to keep this open until we know why this is happening.

[craig-willis]

Causes identified were related to outages -- zero byte files on disk. Created new issue.

#469