Week 1 check ins

[eeholmes]

SST NOAA OISST AVHRR-only 0.25 deg grid

https://github.com/UW-Upwelling-Project/nhw21-projects-upwelling

[minhphan03]

bounding_box = '60,5,80,25' # W,S,E,N for India

lon1, lat1 = 60, 5 # west
lon2, lat2 = 80, 25

[minhphan03]

Research paper on possible factors of coastal upwelling in Indian Ocean: https://www.frontiersin.org/articles/10.3389/fmars.2022.950733/full

Three possible factors: chlorophyll-A density, sea surface temperature, and sea level anomaly

[eeholmes]

[eeholmes]

[eeholmes]

Wind

[minhphan03]

Hey @eeholmes now that the paper mentioned using monthly winds instead of daily winds is it recommended that we should collect the data that way, too? If so, the chlorophyll data from MODIS may actually work...

[eeholmes]

Wednesday updates

Eli

• Still working on the dask hub. Making progress.
• It seems that s3 access is possible with the correct url https://docs.2i2c.org/data/cloud/#reading-data This works

import xarray as xr
ds = xr.open_dataset("s3://mur-sst/zarr/", engine="zarr", storage_options={"anon": True})

It is in the AWS Public Data program. https://registry.opendata.aws/mur/

• or this is Luis' new earthdata package which use https if not on AWS I think https://nasa-openscapes.github.io/earthdata-cloud-cookbook/external/Sea_level_rise_notebook.html

More tutorials on NASA Earthdata

https://nasa-openscapes.github.io/earthdata-cloud-cookbook/
This shows how to access the data wo S3 but with https using https://pypi.org/project/earthaccess/
https://nasa-openscapes.github.io/earthdata-cloud-cookbook/external/Sea_level_rise_notebook.html
More examples using earthdata
https://github.com/betolink/earthaccess-gallery/tree/main/notebooks

[eeholmes]

Plan for tomorrow: Get the hub going with a bigger VM

[NaNa7Miiii]

Dataset in EARTHDATASEARCH: MUR-JPL-L4-GLOB-v4.1.
Variables Contain: analysed_sst, analysis_error, dt_1km_data, mask, sea_ice_fraction, sst_anomaly. It can be a good starting point to see the effects of only SST on coastal upwelling. To remove noises (months that coastal upwelling are not active), if we are studying the Arabian Sea, we can filter the date to be June-September.

My plan:
(1) Model Selection: choosing a pre-trained CNN model that has been trained on a large image dataset (ImageNet). The model can be efficientnet / resnet / VGG, etc. Given those models are trained on natural objects instead of remote sensing data, I am going to remove the softmax layer and only use the model for feature extraction.

(2) Dimensionality Reduction: using techniques such as PCA to reduce the dimensionality of the feature vectors.

(3) Clustering: Since we are doing unsupervised learning, we will then group the data with a clustering algorithm. In this case, we can try both K-Means and hierarchical Clustering with method="complete", while Dr.Eli mentions later is better since it allows different group sizes. We can adjust the # of centroids to see effects, but based on previous works, the smaller # of centroids has better results.

For later work, we may (1) apply different variables, as minh described, sea level anomaly, wind, and chlorophyll-A density.
(2) focus not only on the South East Arabian Sea but also Sudan
(3) Model Accuracy Benchmark

[NaNa7Miiii]

https://colab.research.google.com/drive/1NneZ3mzLwAHGs3vanwCihEgEMsZtacis?usp=sharing
https://colab.research.google.com/drive/1RRPV4heT9PEhFTGxBGRRrTVMLfgGT2Jf?usp=sharing

[eeholmes]

Automatic upwelling detection: https://uw-upwelling-project.github.io/imageryML/

[eeholmes]

Howard's 2021 write up SAFS-Varanasi-Internship/Summer-2021#45

[minhphan03]

Minh's Thursday report

Chlorophyll-a concentration dataset:

I found this dataset online through an article published on Frontiers in Marine Science of some scientists working on a merged dataset from various sources, including SeaWiFS, MODIS, MERIS, VIIRS, OLCI. Missing data in Nov 2002.

Advantages:

• Clean dataset
• Has been tested/validated

Disadvantages:

• Have to download individual files by month (no cloud access)

Copernicus ERA5 dataset:

Advantages:

• Many popular variables: u and v components of wind (10m, 100m), temperature, sea surface temperature
• Temporal and spatial selection

Disadvantages:

• Limited data request size
• Downloads only -- no cloud access. Download speed is slow. I downloaded wind components from 2016-2022 with netcdf4 file type for 3.1GB and over an hour of download time.

ERDDAP NOAA CHL datasets (will try to download sample data, but requires lots of space):

Have lots of different file types.

• SeaWiFS: https://coastwatch.pfeg.noaa.gov/erddap/info/erdSW1chlamday/index.html
• MODIS-Aqua: https://coastwatch.pfeg.noaa.gov/erddap/info/erdMH1chlamday/index.html

General ideas

Supervised learning: downsampling data (pivoted tables by lat/lon) or images to daily/monthly average (depending on whether we want consistency between variables) and provide labels for data using the time variable. One model for each variable to pick up special features...?
Unsupervised learning: group feature images by time, then apply models to perform k-means, clustering, etc.

[eeholmes]

surface currents: OSCAR_L4_OC_FINAL_V2.0
https://search.earthdata.nasa.gov/search/granules/collection-details?portal=podaac-cloud&p=C2098858642-POCLOUD&pg%5B0%5D%5Bv%5D=f&pg%5B0%5D%5Bgsk%5D=-start_date&g=G2207151749-POCLOUD&q=wind&ff=Available%20in%20Earthdata%20Cloud&tl=1686871520.511%213%21%21&zoom=0

[eeholmes]

https://coastwatch.pfeg.noaa.gov/erddap/griddap/erdMH1chlamday.nc?chlorophyll%5B(2022-05-16T00:00:00Z):1:(2022-05-16T00:00:00Z)%5D%5B(25):1:(5)%5D%5B(60):1:(80)%5D

[eeholmes]

Link to notebook that downloads files
https://github.com/UW-Upwelling-Project/nhw21-projects-upwelling/blob/main/notebooks/06-Multi-year-onpremise-download.ipynb

[eeholmes]

@minhphan03 Here is a summary (table at bottom) of various "reanalysis" datasets. These datasets combine the information from many datasets to get estimates of variables (like SST, and sea surface height, or wind or pressure) from all the available data.
https://climatedataguide.ucar.edu/climate-data/atmospheric-reanalysis-overview-comparison-tables

Page on the ERA5 reanalysis https://rda.ucar.edu/datasets/ds633.0/

I found the ERA5 data on Open Data AWS https://registry.opendata.aws/ecmwf-era5/ That means you can load it easily. You won't run into the S3 credentials problems. There is a Jupyter Notebook tutorial on the Open Data AWS page.

[eeholmes]

@NaNa7Miiii The ERA5 data set has a lot of variables of interest. This might save a lot of time because we could (hopefully) just use the ERA5 data set. It needs to be checked if it has good coastal SST information. Some of these datasets do not have good coastal resolution. It'll be necessary to compare to the OI (optimal interpolation) data set because that one was designed to be good at the coast.

[NaNa7Miiii]

Jiarui Thursday Report:
(1) Digested Howard's code and automatic detection method , created a python version of it:https://colab.research.google.com/drive/1FZ6fvA-QH7LUzhItYhGL1mvQGEYawTv9?usp=sharing, plan to apply this automatic detection on SST images in terms of labeling. My current Python version can show points of upwelling in an image, while for labeling it's just True or False.

[eeholmes]

geopandas to create a buffer along a coastline https://stackoverflow.com/questions/74709468/wrong-points-buffer-using-geopandas/74783942#74783942

[eeholmes]

https://uw-upwelling-project.github.io/imageryML/articles/data.html

[minhphan03]

Minh's Friday report

Potential datasets on marine nutrient

Corpernicus Marine Science: https://data.marine.copernicus.eu/product/OCEANCOLOUR_GLO_BGC_L4_MY_009_104/download?dataset=cmems_obs-oc_glo_bgc-plankton_my_l4-multi-4km_P1M; for monthly data we have overall chlorophyll-a mass concentration and dinophytes/greenalgae/diatoms/haptophytes/microphytoplankton/... concentration.

Explore current datasets:

ERA5 dataset: promising but weird rendering due to nan values during peak months? (white area)

chlorophyll.3.mp4

Hawaii EPRAC 3D wind source

Cannot recognize land/ocean area (monthly data average, perhaps?)

2000-09 (10m)

2000-02 (10m)

Next steps

Exploring AWS CLI on jupyter notebook to access data on ERA5 reanalysis https://aws.amazon.com/cli/

[minhphan03]

Notebook on accessing s3 bucket: https://github.com/planet-os/notebooks/blob/master/aws/era5-s3-via-boto.ipynb

[NaNa7Miiii]

Jiarui Friday Report
(1) worked on automatic detection and add the buffer functions to it: https://dhub.opensci.live/user/nana7miiii/lab/tree/test.ipynb
(2) test the sst data from the ERA5 dataset:
https://dhub.opensci.live/user/nana7miiii/lab/tree/data_test.ipynb

[eeholmes]

Eli's Friday Report

I found the ERA5 data on Open Data AWS https://registry.opendata.aws/ecmwf-era5/ That means you can load it easily. You won't run into the S3 credentials problems. There is a Jupyter Notebook tutorial on the Open Data AWS page.

I contacted Azure support and set up a meeting to learn how to set up a drive for our data.