Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Hyperspectral 2018 Level_1 data generation #591

Open
max-zilla opened this issue Jun 18, 2019 · 6 comments
Open

Hyperspectral 2018 Level_1 data generation #591

max-zilla opened this issue Jun 18, 2019 · 6 comments
Assignees
@max-zilla @Paheding
Labels
kind/bug sensor/hyperspectral issues relating to VNIR and SWIR scanners
Milestone
TERRA Sprint - Au...

Comments

@max-zilla
Copy link
Contributor

max-zilla commented Jun 18, 2019 •
edited by dlebauer
Loading

We need Level_1 outputs from 2018 using existing code to recalibrate the new algorithm.

It seems the cst_cnv_trg.nc file has trouble with the new band count - the new algorithm doesn't use this file but we need to use it with existing pipeline on 2018 data then recalibrate based on that output.

Is it possible to avoid relying on this file, or generating an updated version w/ different number of bands?
@max-zilla max-zilla added sensor/hyperspectral issues relating to VNIR and SWIR scanners kind/bug labels Jun 18, 2019
@max-zilla max-zilla added this to the TERRA Sprint - June 2019 milestone Jun 18, 2019
@dlebauer
Copy link
Member

Moving this to a separate comment because it addresses the specific question

Is it possible to ... generate an updated version w/ different number of bands?

Yes!

The NCO command for subsetting the file to the first 939 wavelengths uses the ncks function

ncks -O -d wavelength,0,938 cst_cnv_trg.nc foo.nc

This creates a new file called ‘foo.nc’ that contains only 939 wavelengths. To explain each part:

  • ncks : use the nco kitchen sink operators to output a new
  • -O output a new file
  • -d wavelength,0,938
    • subset the dimension named wavelength
    • starting at the first value (index 0) and ending at the 939th (index 938)
  • cst_cnv_trg.nc # file to read in
  • foo.nc # file to read out

To check that the output file is of the correct dimensions, dump the meatdata header and print just the dimension defintions:

ncdump -h foo.nc | grep -A 5 dimensions

Dump out the actual values of wavelength along the wavelength dimension:

ncdump -v wavelength cst_cnv_trg.nc

note that the wavelength variable is both a dimension with integer indices 0:938 (one dimension of the hyperspectral data cube) and a variable with values and units of wavelengths in nm divided into 939 intervals in the range [389,1000].

@max-zilla
Copy link
Contributor Author

Added the following code:
https://github.com/terraref/extractors-hyperspectral/blob/reprocessing2018/hyperspectral/hyperspectral_workflow.sh#L593

# Subset raw NC file bands
    att_sub=${fl_in[$fl_idx]/_raw/_raw_sub.nc}
    att_sub="${att_in}.subset.tmp"
    cmd_subset="ncks -O -d wavelength,0,938 ${att_in} ${att_sub}"
    eval ${cmd_subset}
    if [ $? -ne 0 ] || [ ! -f ${att_sub} ]; then
		printf "${spt_nm}: ERROR Failed to subset raw data bands. Debug this:\n${cmd_subset}\n"
		exit 1
	    fi # !err
    att_in=${att_sub}

...however it didn't seem to resolve error:

Terraref hyperspectral data workflow invoked with:
./hyperspectral_workflow.sh -d 1 -h --new_clb_mth -i /home/extractor/sites/ua-mac/raw_data/VNIR/2018-08-18/2018-08-18__11-47-47-403/aabc02a5-2fe1-4fdd-b26f-b87bf50c24fe_raw -o /home/extractor/sites/ua-mac/Level_1/vnir_netcdf/2018-08-18/2018-08-18__11-47-47-403/vnir_netcdf_L1_ua-mac_2018-08-18__11-47-47-403.nc
Hyperspectral workflow scripts in directory /home/extractor
NCO version 4.7.9 from directory /home/extractor/anaconda2/bin
Intermediate/temporary files written to directory /tmp
Final output stored in directory /home/extractor/sites/ua-mac/Level_1/vnir_netcdf/2018-08-18/2018-08-18__11-47-47-403
Input #00: /home/extractor/sites/ua-mac/raw_data/VNIR/2018-08-18/2018-08-18__11-47-47-403/aabc02a5-2fe1-4fdd-b26f-b87bf50c24fe_raw
trn(in)  : /home/extractor/sites/ua-mac/raw_data/VNIR/2018-08-18/2018-08-18__11-47-47-403/aabc02a5-2fe1-4fdd-b26f-b87bf50c24fe_raw
trn(out) : /tmp/terraref_tmp_trn.nc.pid25.fl00.tmp
ncks -O --hdr_pad=10000 --no_tmp_fl --trr_wxy=939,1600,2212 --trr typ_in=NC_USHORT --trr typ_out=NC_USHORT --trr ntl_in=bil --trr ntl_out=bsq --trr_in=/home/extractor/sites/ua-mac/raw_data/VNIR/2018-08-18/2018-08-18__11-47-47-403/aabc02a5-2fe1-4fdd-b26f-b87bf50c24fe_raw /home/extractor/hyperspectral_dummy.nc /tmp/terraref_tmp_trn.nc.pid25.fl00.tmp
att(in)  : /tmp/terraref_tmp_trn.nc.pid25.fl00.tmp.subset.tmp
att(out) : /tmp/terraref_tmp_att.nc.pid25.fl00.tmp
ncatted -O --gaa terraref_script=hyperspectral_workflow.sh --gaa workflow_invocation="'./hyperspectral_workflow.sh -d 1 -h --new_clb_mth -i /home/extractor/sites/ua-mac/raw_data/VNIR/2018-08-18/2018-08-18__11-47-47-403/aabc02a5-2fe1-4fdd-b26f-b87bf50c24fe_raw -o /home/extractor/sites/ua-mac/Level_1/vnir_netcdf/2018-08-18/2018-08-18__11-47-47-403/vnir_netcdf_L1_ua-mac_2018-08-18__11-47-47-403.nc'" --gaa terraref_hostname=hyperspectral-gn4nd --gaa terraref_version=4.7.9 -a "Conventions,global,o,c,CF-1.5" -a "Project,global,o,c,TERRAREF" --gaa history="Mon Jul 22 15:30:10 UTC 2019: ./hyperspectral_workflow.sh -d 1 -h --new_clb_mth -i /home/extractor/sites/ua-mac/raw_data/VNIR/2018-08-18/2018-08-18__11-47-47-403/aabc02a5-2fe1-4fdd-b26f-b87bf50c24fe_raw -o /home/extractor/sites/ua-mac/Level_1/vnir_netcdf/2018-08-18/2018-08-18__11-47-47-403/vnir_netcdf_L1_ua-mac_2018-08-18__11-47-47-403.nc" /tmp/terraref_tmp_trn.nc.pid25.fl00.tmp.subset.tmp /tmp/terraref_tmp_att.nc.pid25.fl00.tmp
jsn(in)  : /home/extractor/sites/ua-mac/raw_data/VNIR/2018-08-18/2018-08-18__11-47-47-403/aabc02a5-2fe1-4fdd-b26f-b87bf50c24fe_raw
jsn(out) : /tmp/terraref_tmp_jsn.nc.pid25
python /home/extractor/hyperspectral_metadata.py dbg=yes fmt=4 ftn=no /home/extractor/sites/ua-mac/raw_data/VNIR/2018-08-18/2018-08-18__11-47-47-403/aabc02a5-2fe1-4fdd-b26f-b87bf50c24fe_raw /tmp/terraref_tmp_jsn.nc.pid25.fl00.tmp
ERROR: nco_put_vara() failed to nc_put_vara() variable "cst_cnv_trg_nw"
nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco_put_vara()
nco_err_exit(): ERROR Error code is -57. Translation into English with nc_strerror(-57) is "NetCDF: Start+count exceeds dimension bound"
nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE)

will need someone's help with this.

@max-zilla
Copy link
Contributor Author

SWIR 2017 error:

./hyperspectral_workflow.sh -d 1 -h --new_clb_mth \
-i /home/extractor/sites/ua-mac/raw_data/SWIR/2017-04-16/2017-04-16__11-50-46-707/c6079666-b686-4481-9a4f-0663f5f43a6a_raw \
-o /home/extractor/sites/ua-mac/Level_1/swir_netcdf/2017-04-16/2017-04-16__11-50-46-707/swir_netcdf_L1_ua-mac_2017-04-16__11-50-46-707.nc

Terraref hyperspectral data workflow invoked with:
./hyperspectral_workflow.sh -d 1 -h --new_clb_mth -i /home/extractor/sites/ua-mac/raw_data/SWIR/2017-04-16/2017-04-16__11-50-46-707/c6079666-b686-4481-9a4f-0663f5f43a6a_raw -o /home/extractor/sites/ua-mac/Level_1/swir_netcdf/2017-04-16/2017-04-16__11-50-46-707/swir_netcdf_L1_ua-mac_2017-04-16__11-50-46-707.nc
Hyperspectral workflow scripts in directory /home/extractor
NCO version 4.7.9 from directory /home/extractor/anaconda2/bin
Intermediate/temporary files written to directory /tmp
Final output stored in directory /home/extractor/sites/ua-mac/Level_1/swir_netcdf/2017-04-16/2017-04-16__11-50-46-707
Input #00: /home/extractor/sites/ua-mac/raw_data/SWIR/2017-04-16/2017-04-16__11-50-46-707/c6079666-b686-4481-9a4f-0663f5f43a6a_raw
trn(in)  : /home/extractor/sites/ua-mac/raw_data/SWIR/2017-04-16/2017-04-16__11-50-46-707/c6079666-b686-4481-9a4f-0663f5f43a6a_raw
trn(out) : /tmp/terraref_tmp_trn.nc.pid36.fl00.tmp
ncks -O --hdr_pad=10000 --no_tmp_fl --trr_wxy=273,384,2726 --trr typ_in=NC_USHORT --trr typ_out=NC_USHORT --trr ntl_in=bil --trr ntl_out=bsq --trr_in=/home/extractor/sites/ua-mac/raw_data/SWIR/2017-04-16/2017-04-16__11-50-46-707/c6079666-b686-4481-9a4f-0663f5f43a6a_raw /home/extractor/hyperspectral_dummy.nc /tmp/terraref_tmp_trn.nc.pid36.fl00.tmp
att(in)  : /tmp/terraref_tmp_trn.nc.pid36.fl00.tmp
att(out) : /tmp/terraref_tmp_att.nc.pid36.fl00.tmp
ncatted -O --gaa terraref_script=hyperspectral_workflow.sh --gaa workflow_invocation="'./hyperspectral_workflow.sh -d 1 -h --new_clb_mth -i /home/extractor/sites/ua-mac/raw_data/SWIR/2017-04-16/2017-04-16__11-50-46-707/c6079666-b686-4481-9a4f-0663f5f43a6a_raw -o /home/extractor/sites/ua-mac/Level_1/swir_netcdf/2017-04-16/2017-04-16__11-50-46-707/swir_netcdf_L1_ua-mac_2017-04-16__11-50-46-707.nc'" --gaa terraref_hostname=hyperspectral-2ssvf --gaa terraref_version=4.7.9 -a "Conventions,global,o,c,CF-1.5" -a "Project,global,o,c,TERRAREF" --gaa history="Mon Jul 15 13:00:47 UTC 2019: ./hyperspectral_workflow.sh -d 1 -h --new_clb_mth -i /home/extractor/sites/ua-mac/raw_data/SWIR/2017-04-16/2017-04-16__11-50-46-707/c6079666-b686-4481-9a4f-0663f5f43a6a_raw -o /home/extractor/sites/ua-mac/Level_1/swir_netcdf/2017-04-16/2017-04-16__11-50-46-707/swir_netcdf_L1_ua-mac_2017-04-16__11-50-46-707.nc" /tmp/terraref_tmp_trn.nc.pid36.fl00.tmp /tmp/terraref_tmp_att.nc.pid36.fl00.tmp
jsn(in)  : /home/extractor/sites/ua-mac/raw_data/SWIR/2017-04-16/2017-04-16__11-50-46-707/c6079666-b686-4481-9a4f-0663f5f43a6a_raw
jsn(out) : /tmp/terraref_tmp_jsn.nc.pid36
python /home/extractor/hyperspectral_metadata.py dbg=yes fmt=4 ftn=no /home/extractor/sites/ua-mac/raw_data/SWIR/2017-04-16/2017-04-16__11-50-46-707/c6079666-b686-4481-9a4f-0663f5f43a6a_raw /tmp/terraref_tmp_jsn.nc.pid36.fl00.tmp
mrg(in)  : /tmp/terraref_tmp_jsn.nc.pid36.fl00.tmp
mrg(out) : /tmp/terraref_tmp_att.nc.pid36.fl00.tmp
ncks -A -C -v wavelength,x,y /tmp/terraref_tmp_jsn.nc.pid36.fl00.tmp /tmp/terraref_tmp_att.nc.pid36.fl00.tmp
clb(in)  : /tmp/terraref_tmp_att.nc.pid36.fl00.tmp
clb(out) : /tmp/terraref_tmp_clb.nc.pid36.fl00.tmp
ncap2 --no_cll_mth -O --hdr_pad=10000 -v -S /home/extractor/hyperspectral_calibration_new.nco /tmp/terraref_tmp_att.nc.pid36.fl00.tmp /tmp/terraref_tmp_clb.nc.pid36.fl00.tmp
ncap2: WARNING prs_cls::ncap_var_init(): Unable to find variable cst_cnv_trg_nw in /tmp/terraref_tmp_att.nc.pid36.fl00.tmp or /tmp/terraref_tmp_clb.nc.pid36.fl00.tmp
nco_dfl_case_nc_type_err(): ERROR switch(nctype) statement fell through to default case, which is illegal, because each type should have a well-defined action. This error may be triggered by using an NCO built with only netCDF3 functionality to examine a netCDF4 dataset that contains a new atomic type (e.g., NC_INT64).
HINT: Configure/build NCO with --enable-netCDF4. Exiting...
nco_err_exit(): ERROR Short NCO-generated message (usually name of function that triggered error): nco_dfl_case_nc_type_err()
nco_err_exit(): ERROR Error code is 0. This indicates an error occurred in NCO code or in a system call, not in the netCDF layer.
nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE)
hyperspectral_workflow.sh: ERROR Failed to calibrate data in ncap2. Debug this:
ncap2 --no_cll_mth -O  --hdr_pad=10000 -v -S /home/extractor/hyperspectral_calibration_new.nco /tmp/terraref_tmp_att.nc.pid36.fl00.tmp /tmp/terraref_tmp_clb.nc.pid36.fl00.tmp

@max-zilla
Copy link
Contributor Author

SWIR 2018 error:

./hyperspectral_workflow.sh -d 1 -h --new_clb_mth \
-i /home/extractor/sites/ua-mac/raw_data/SWIR/2018-08-29/2018-08-29__14-57-20-076/51bec282-a013-4811-b9dd-c1628afe34ab_raw \
-o /home/extractor/sites/ua-mac/Level_1/swir_netcdf/2018-08-29/2018-08-29__14-57-20-076/swir_netcdf_L1_ua-mac_2018-08-29__14-57-20-076.nc

Terraref hyperspectral data workflow invoked with:
./hyperspectral_workflow.sh -d 1 -h --new_clb_mth -i /home/extractor/sites/ua-mac/raw_data/SWIR/2018-08-29/2018-08-29__14-57-20-076/51bec282-a013-4811-b9dd-c1628afe34ab_raw -o /home/extractor/sites/ua-mac/Level_1/swir_netcdf/2018-08-29/2018-08-29__14-57-20-076/swir_netcdf_L1_ua-mac_2018-08-29__14-57-20-076.nc
Hyperspectral workflow scripts in directory /home/extractor
NCO version 4.7.9 from directory /home/extractor/anaconda2/bin
Intermediate/temporary files written to directory /tmp
Final output stored in directory /home/extractor/sites/ua-mac/Level_1/swir_netcdf/2018-08-29/2018-08-29__14-57-20-076
Input #00: /home/extractor/sites/ua-mac/raw_data/SWIR/2018-08-29/2018-08-29__14-57-20-076/51bec282-a013-4811-b9dd-c1628afe34ab_raw
trn(in)  : /home/extractor/sites/ua-mac/raw_data/SWIR/2018-08-29/2018-08-29__14-57-20-076/51bec282-a013-4811-b9dd-c1628afe34ab_raw
trn(out) : /tmp/terraref_tmp_trn.nc.pid23.fl00.tmp
ncks -O --hdr_pad=10000 --no_tmp_fl --trr_wxy=275,384,12268 --trr typ_in=NC_USHORT --trr typ_out=NC_USHORT --trr ntl_in=bil --trr ntl_out=bsq --trr_in=/home/extractor/sites/ua-mac/raw_data/SWIR/2018-08-29/2018-08-29__14-57-20-076/51bec282-a013-4811-b9dd-c1628afe34ab_raw /home/extractor/hyperspectral_dummy.nc /tmp/terraref_tmp_trn.nc.pid23.fl00.tmp
att(in)  : /tmp/terraref_tmp_trn.nc.pid23.fl00.tmp
att(out) : /tmp/terraref_tmp_att.nc.pid23.fl00.tmp
ncatted -O --gaa terraref_script=hyperspectral_workflow.sh --gaa workflow_invocation="'./hyperspectral_workflow.sh -d 1 -h --new_clb_mth -i /home/extractor/sites/ua-mac/raw_data/SWIR/2018-08-29/2018-08-29__14-57-20-076/51bec282-a013-4811-b9dd-c1628afe34ab_raw -o /home/extractor/sites/ua-mac/Level_1/swir_netcdf/2018-08-29/2018-08-29__14-57-20-076/swir_netcdf_L1_ua-mac_2018-08-29__14-57-20-076.nc'" --gaa terraref_hostname=hyperspectral-w1jbx --gaa terraref_version=4.7.9 -a "Conventions,global,o,c,CF-1.5" -a "Project,global,o,c,TERRAREF" --gaa history="Mon Jul 22 18:55:01 UTC 2019: ./hyperspectral_workflow.sh -d 1 -h --new_clb_mth -i /home/extractor/sites/ua-mac/raw_data/SWIR/2018-08-29/2018-08-29__14-57-20-076/51bec282-a013-4811-b9dd-c1628afe34ab_raw -o /home/extractor/sites/ua-mac/Level_1/swir_netcdf/2018-08-29/2018-08-29__14-57-20-076/swir_netcdf_L1_ua-mac_2018-08-29__14-57-20-076.nc" /tmp/terraref_tmp_trn.nc.pid23.fl00.tmp /tmp/terraref_tmp_att.nc.pid23.fl00.tmp
jsn(in)  : /home/extractor/sites/ua-mac/raw_data/SWIR/2018-08-29/2018-08-29__14-57-20-076/51bec282-a013-4811-b9dd-c1628afe34ab_raw
jsn(out) : /tmp/terraref_tmp_jsn.nc.pid23
python /home/extractor/hyperspectral_metadata.py dbg=yes fmt=4 ftn=no /home/extractor/sites/ua-mac/raw_data/SWIR/2018-08-29/2018-08-29__14-57-20-076/51bec282-a013-4811-b9dd-c1628afe34ab_raw /tmp/terraref_tmp_jsn.nc.pid23.fl00.tmp
Traceback (most recent call last):
  File "/home/extractor/hyperspectral_metadata.py", line 780, in <module>
    main()
  File "/home/extractor/hyperspectral_metadata.py", line 776, in main
    testCase.writeToNetCDF(file_input, file_output, " ".join((file_input, file_output)), format, flatten, debug)
  File "/home/extractor/hyperspectral_metadata.py", line 156, in writeToNetCDF
    tempVariable[...] = translate_time(subdata)
  File "/home/extractor/hyperspectral_metadata.py", line 624, in translate_time
    timeSplit  = date(year=timeUnpack.tm_year, month=timeUnpack.tm_mon,
AttributeError: 'NoneType' object has no attribute 'tm_year'
hyperspectral_workflow.sh: ERROR Failed to parse JSON metadata. Debug this:
python /home/extractor/hyperspectral_metadata.py dbg=yes fmt=4 ftn=no /home/extractor/sites/ua-mac/raw_data/SWIR/2018-08-29/2018-08-29__14-57-20-076/51bec282-a013-4811-b9dd-c1628afe34ab_raw /tmp/terraref_tmp_jsn.nc.pid23.fl00.tmp

@max-zilla
Copy link
Contributor Author

@Paheding https://github.com/terraref/extractors-hyperspectral/blob/reprocessing2018/hyperspectral/hyperspectral_workflow.sh

@max-zilla
Copy link
Contributor Author

https://github.com/terraref/extractors-hyperspectral/pull/50/files

new PR here that is still being finalized, result of our first two hackathons. Should be able to support VNIR new camera, SWIR new + old cameras.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@max-zilla max-zilla

@Paheding Paheding

Labels
kind/bug sensor/hyperspectral issues relating to VNIR and SWIR scanners
Projects
None yet
Milestone
TERRA Sprint - August 2019
Development

No branches or pull requests
3 participants
@dlebauer @max-zilla @Paheding