This repository has been archived by the owner on Sep 3, 2018. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 7
/
README
130 lines (92 loc) · 5.8 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
This repository provides material used for the paper: <add a link here>
Overview
========
The repository contains scripts and Python Notebooks which can be used to perform a comparison of
1) Raster water masks derived from Landsat satellite imagery
2) Vector datasets representing hydrography (OpenStreetMap or Surface Hydrology for Australia)
3) Drainage networks derived from 30m SRTM
Some of the scripts make use of Google Earth Engine (GEE).
Some of the scripts require https://github.com/gena/ee-runner to run GEE JavaScript API in a command-line.
Dirctories
==========
bin/ .......................... programs used during conversion
gdalcopyprj.cmd ............. copies gographic projection from one raster file to another
gdalcopyprj.py
osmconvert.exe .............. converts OpenStreetMap pbf binary files to o5m and osm format
osmfilter.exe ............... filter OpenStreetMap files
config/ ....................... config files use by some of the scripts
osmconf.ini ................. used by osmconvert and osmfilter
water.osmfilter.params ...... combines tags used to extract hydrography from OpenStreetMap files (polylines and polygons)
data/ ......................... input files
OpenStreetMap/
SurfaceHydrology/ ........... major rivers extracted from http://www.ga.gov.au
grid.kml .................... regular grid used to aggregate results and to parallelize the analysis
HydroSHEDS_Catchment_MD.* ... HydroSHEDS cachment boundaries for Murray-Darling River Basin
HydroSHEDS_rivers_MD.* ...... HydroSHEDS drainage network
graphics/ ..................... images comparing generated water mask and OSM with WOfS and Surface Hydrology
<images>
notebooks/ .................... Python Notebooks used to perform different steps of the analysis
ConvertWaterHistogramsAndThresholds.ipynb
ExplodeShpPolylines.ipynb
ExplodeShpPolylinesSH.ipynb
HAND.ipynb
HydroBASINS (fix).ipynb
NDWI_thresholds.ipynb
OSM water validation using LANDSAT (Murray & Darling).ipynb
SHP2KML with fix.ipynb
TileCatchmentsToGrid.ipynb
UploadToFusionTable.ipynb
results/ ...................... some of the results generated during analysis
water_and_results_examples/
water.zip
authoritative/ .............. screenshots comparing L8 watermask to some of the authoritative datastes
src/
javascript/
download_SRTM.js ........... GEE script used to download SRTM data clipped by catchment boundaries
download_water.js .......... GEE
training.js ................ training dataset, generated within GEE
javascript-ee-playground/ ......... GEE Playground scripts
OSM_automatic_NDWI.js ..........
OSM_compute_buffers.js ......... computes distances between rivers using Goodchilds's method
OSM_compute_buffers_export.js ..
OSM_stat_per_catchment.js ...... generates aggregated statistics
python/
generate-hydro-datasets.py .......... script used to deleniate drainage network and compute HAND
generate-hydro-datasets-runner.py ... wraps the above script to avoid out-of-memory probles
scripts/
convert.cmd .................... script used to extract hydrography from the OSM files
web/ .......................... source code of the Google App Engine website (http://osm-water.appspot.com)
httplib2
media
oauth2client
static
app.yaml
config_web.py
index.html
README.md
server.py
six.py
AuthoritativeDatasets.docx ...... early comparisons to the authoritative datasets in Australia
Tutorial
========
The overall analysis is performed in a number of steps, starting with the download of all required datasets, such as:
* HydroBASINS
* OpenStreetMap extracts, relevant for the study area
Additional step include preparation of the regular grid file (grid.kml) used to parallelize the analysis.
The actual steps required for the analysis are quite time consuming and include:
1. Extraction of the hydrography from the the OSM files (scripts/convert.cmd)
Produces two shapefiles (polylines and polygons) representing hydrography
2. Split linear hydrography features into small polylines in order to perform the analysis (notebooks/ExplodeShpPolylines.ipynb)
3. Upload results of 2. to Google Fusion Table (UploadToFusionTable.ipynb)
4. Select required catchment boundaries from HydroBASINS. Catchment size should be smaller than current Google Earth Engine download limit!
5. Upload catchments to the Google Fusion Table
6. Prepare regular grid file (grid.kml) and upload to Google Fusion Table, the resulting asset id needs to be updated in the OSM_automatic_NDWI.js script
7. Download SRTM extracts clipped by catchment boundaries (src/javascript/download_SRTM.js)
8. Deleniate drainage network for every sub-catchment DEM from 6. and compute HAND dataset (and all other related hydrological dataset) using python/generate-hydro-datasets-runner.py
9. Convert results of 7. to regular grid tiles and upload to Google Earth Engine as raster assets.
10. Adjust src/javascript/download_water.js do use the new HAND, grid (a copy of src/javascript-ee-playground/OSM_automatic_NDWI.js).
11. Prepare training set for GEE Playground scripts (10.) by performing script runs for a number of tiles, repeat until the results are good for hilly areas,
For flat areas and sufficient cloud-free images the script should produce good results without any changes
12. Detect surface water mask for every grid tile using src/javascript/download_water.js. src/javascript/templates provides example on how to parallelize water detection.
13. Upload (re-tile if necessart) the resulting water mask to Google Earth Engine
14. Compute positional and thematic accuracy using src/javascript-ee-playground/OSM_compute_buffers_export.js and src/javascript-ee-playground/OSM_stat_per_catchment.js