README.org

Table of Contents

• Sentinel-3 (S3) Albedo Processing Pipeline
  • Fetch S3 OLCI & SLSTR products
  • Process with SNAP
  • pySICE
  • Mosaic
  • Outputs
• Debugging & Testing
• Development Environment
  • Bash
  • Python
  • Parallel
  • GRASS
  • GPT
  • SNAP
• Misc Notes & Code Snippets
  • Generate ice mask
  • Footprint
  • Regional masks

Sentinel-3 (S3) Albedo Processing Pipeline

1. Fetch S3 OLCI & SLSTR products
2. Process OLCI & SLSTR using SNAP GPT
3. Process using pySICE
4. Build daily mosaic

The steps above are encapsulated in ./S3_wrapper.sh.

In more detail:

Fetch S3 OLCI & SLSTR products

• Download OLCI EFR and SLSTR RBT products for a specified day with ./dhusget_wrapper.sh.

Process with SNAP

• This step is a combination of running GPT on ./S3.xml with ./S3_proc.sh.
• Inputs: OLCI and SLSTR scenes
• Outputs: A folder OLCI timestamp

pySICE

• Run ./sice.py passing in one of the folders generated in the previous step.
• WARNING: This step is slow, and may take > 24 hours.

Mosaic

• As implemented in ./dm.sh and ./dm.grass.sh, for each day
  • Combine all the files from to form a mosaic
  • Mask out clouds, then…
  • When scenes overlap, use minimum SZA

Outputs

File Name	Description	Units
grain_diameter		mm
snow_specific_surface_area		m2kg-1
ndsi	normalized difference snow index
ndbi	normalized difference blue ice index
auto_cloud	in development
sza	sun zenith angle
vza	viewing zenith angle
saa	sun azimuth angle
vaa	viewing azimuth angle
albedo_bb_planar_nir	near infrared broadband planar albedo	700-2400 nm
albedo_bb_planar_sw	shortwave broadband planar albedo	300-2400 nm
albedo_bb_planar_vis	visible broadband planar albedo	300-700 nm
albedo_bb_spherical_nir	near infrared broadband spherical albedo	700-2400 nm
albedo_bb_spherical_sw	shortwave broadband spherical albedo	300-2400 nm
albedo_bb_spherical_vis	visible broadband spherical albedo	300-700 nm
Oa01..21_reflectance	top of the atmosphere reflectance (OLCI bands 1 through 21)
rBRR_01..21	bottom of the atmosphere surface reflectance (OLCI bands 1 through 21)
albedo_spectral_planar_01..21	spectral planar albedo (OLCI bands 1 through 21)
diagnostic_retrieval	See next table.

Diagnostic Code	Description
0	clean snow
1	polluted snow
6	polluted snow for which r0 was calculated and not derived from observations
7	polluted snow of calculated spherical albedo in bands 1 and 2 >0.98 reprocessed as clean snow
100	sza>75, no retrival
102	TOA reflectance at band 21 < 0.1, no retrieval
104	grain_diameter < 0.1, no retrieval, potential cloud flag
-n	impossible to solve polluted snow albedo equation at band n

Debugging & Testing

Change the year and day for loops in ./S3_wrapper.sh to one year and one day. E.g.

for year in 2017; do # one year
  for doy in 227 180; do # two example days
  # loop contents
  done
done

Development Environment

This work is developed with the following software versions

Bash

bash --version

GNU bash, version 4.4.20(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Python

Run the following using ./SICE.yml in order to set up an identical Anaconda Python environment.

conda env create -f SICE.yml

And then run conda activate SICE to activate the SICE environment.

ResolvePackageNotFound error can be raised. In that case, run conda env export --no-builds > environment.yml instead.

Parallel

parallel --version

GNU parallel 20161222
Copyright (C) 2007,2008,2009,2010,2011,2012,2013,2014,2015,2016
Ole Tange and Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
GNU parallel comes with no warranty.

Web site: http://www.gnu.org/software/parallel

When using programs that use GNU Parallel to process data for publication
please cite as described in 'parallel --citation'.

GRASS

grass --version 2>&1

GRASS GIS 7.4.0

Geographic Resources Analysis Support System (GRASS) is Copyright,
1999-2018 by the GRASS Development Team, and licensed under terms of the
GNU General Public License (GPL) version >=2.
 
This GRASS GIS 7.4.0 release is coordinated and produced by
the GRASS Development Team with contributions from all over the world.

This program is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
General Public License for more details.

GPT

~/local/snap/bin/gpt --diag

SNAP Release version 7.0
SNAP home: /home/kdm/local/snap/bin//..
SNAP debug: null
SNAP log level: null
Java home: /home/kdm/local/snap/jre
Java version: 1.8.0_202
Processors: 8
Max memory: 18.7 GB
Cache size: 1024.0 MB
Tile parallelism: 8
Tile size: 512 x 512 pixels

To configure your gpt memory usage:
Edit snap/bin/gpt.vmoptions

To configure your gpt cache size and parallelism:
Edit .snap/etc/snap.properties or gpt -c ${cachesize-in-GB}G -q ${parallelism} 

SNAP

SNAP extensions used in this toolchain are listed in SNAP_extensions.txt.

Misc Notes & Code Snippets

Generate ice mask

• Use the icemask from BedMachine v3

grass72 -c EPSG:3413 ./Gtmp
r.in.gdal input=NetCDF:~/data/Greenland/Morlighem_2017/BedMachineGreenland-2017-09-20.nc:mask output=icemask

g.region raster=icemask
g.region res=500 -ap
g.region zoom=icemask

d.mon start=wx0
d.erase
d.rast icemask

r.mapcalc "mask = if(icemask == 4, null(), icemask)" --o
d.rast mask
g.region zoom=mask

r.out.gdal -c -m input=mask output=mask.tif type=Byte createopt=COMPRESS=DEFLATE --o
exit
trash Gtmp

Buffered ice mask

grass -c mask.tif ./Gtmp
r.in.gdal input=mask.tif output=mask
# 50 cells = 25 km @ 500 m
r.mapcalc "ice = if(mask == 2, 1, null())"
r.grow input=ice output=ice_grow radius=50 new=1

r.null ice_grow null=100
r.clump input=ice_grow output=clumps
r.stats -c clumps sort=asc
for ID in $(r.stats -c clumps sort=asc | head -n7 | cut -d" " -f1); do
  r.mapcalc "ice_grow = if(clumps == ${ID}, 1, ice_grow)" --o
done
r.null ice_grow setnull=100

r.out.gdal input=ice_grow output=ice_mask_buffer.tif format=GTiff type=Byte createopt="COMPRESS=DEFLATE"
exit
trash Gtmp

Footprint

Get GL outline by drawing in Google Earth, export KML, then:

ogrinfo -al GL_outline.kml  | grep LINESTRING | sed s/\ 0//g

Regional masks

Regional masks stored in the masks folder are clips from the ESA global Land Cover products (download here). Masks were downloaded for 2018 and are available at 300m and 1km resolutions ({region}_300m.tif and {region}_1km.tif, respectively). A description of the 22 ESA global Land Cover products can be found here.