Week 5 Check Ins

[NaNa7Miiii]

[NaNa7Miiii]

Jiarui Monday CheckIn:

• Integrate my code into 3 notebooks and put them into my jhub shared folder
• Attempt to train a larger model using ERA5's SST data from 197901 (when the ERA5 data starts) to 202305. The training session begins at 1 pm and is still running. I will save it as a checkpoint file so anyone can reload it and continue to train or fine-tune it without trainning from scratch.
• I have also made a file using the LSTM model for prediction. Since I have used the LSTM model for detection, and we are interested in using the past SST data to predict future upwelling, I have tested the LSTM model on an international passenger dataset to see its performance:

# Normalize the dataset
scaler = MinMaxScaler(feature_range=(0, 1))
dataset = scaler.fit_transform(df)

# Split into train and test sets
train_size = int(len(dataset) * 0.67)
test_size = len(dataset) - train_size
train, test = dataset[0:train_size,:], dataset[train_size:len(dataset),:]

# Create TimeseriesGenerator object
look_back = 3
train_data_gen = TimeseriesGenerator(train, train, length=look_back, batch_size=1) 
test_data_gen = TimeseriesGenerator(test, test, length=look_back, batch_size=1)

# Create LSTM model
model = Sequential()
model.add(LSTM(4, activation='relu', input_shape=(look_back, 1)))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')

# Train the model
model.fit_generator(train_data_gen, epochs=50)

# Make predictions
train_predict = model.predict_generator(train_data_gen)
test_predict = model.predict_generator(test_data_gen)

# Invert predictions
train_predict = scaler.inverse_transform(train_predict)
trainY = scaler.inverse_transform([np.squeeze(train)])
test_predict = scaler.inverse_transform(test_predict)
testY = scaler.inverse_transform([np.squeeze(test)])

# Calculate root mean squared error
trainScore = math.sqrt(mean_squared_error(trainY[0][look_back:], train_predict[:,0]))
print('Train Score: %.2f RMSE' % (trainScore))
testScore = math.sqrt(mean_squared_error(testY[0][look_back:], test_predict[:,0]))
print('Test Score: %.2f RMSE' % (testScore))

We can customize the variable 'look_back' to do predictions.

[minhphan03]

Continue writing notebooks

[eeholmes]

Eli

• Working on Zarr tutorial for R

  • https://www.r-bloggers.com/2022/09/reading-zarr-files-with-r-package-stars/
  • https://bioconductor.org/packages/release/bioc/html/Rarr.html

• Some Zarr Python tutorials

  • https://www.youtube.com/watch?v=unGL07trSjA
  • https://github.com/zarr-developers/tutorials/blob/main/zarr_cloud_native_geospatial_2022.ipynb
  • https://mesowest.utah.edu/html/hrrr/zarr_documentation/html/zarr_HowToDownload.html

• CMIP6

  • https://wcrp-cmip.org/cmip-phase-6-cmip6/
  • https://wcrp-cmip.github.io/WGCM_Infrastructure_Panel//cmip6.html
  • https://www.sciencedirect.com/science/article/abs/pii/S0967064523000103
  • jupyter notebook example with CMIP6 https://github.com/zarr-developers/tutorials/blob/main/zarr_cloud_native_geospatial_2022.ipynb
  • https://oceanhealthindex.org/news/cmip_2_what_is_this/

• Bathymetry

[minhphan03]

Start date for our datasets:

• ERA5: 1979 (wind, sea temp, sst) https://github.com/planet-os/notebooks/blob/master/aws/era5-pds.md
• Globcolour: 9/1997 (chlorophyll-a) https://data.marine.copernicus.eu/product/OCEANCOLOUR_GLO_BGC_L4_MY_009_104/description. This is the most expansive one available for chlorophyll-a data, as others in ERDDAP do not have older data: https://coastwatch.pfeg.noaa.gov/erddap/griddap/index.html?page=1&itemsPerPage=1000 (search for 'Chlorophyll-a')
• OSCAR: 1993 (currents) https://search.earthdata.nasa.gov/search/granules/collection-details?portal=podaac-cloud&p=C2098858642-POCLOUD&pg[0][v]=f&pg[0][gsk]=-start_date&g=G2207151749-POCLOUD&q=wind&ff=Available%20in%20Earthdata%20Cloud&tl=1686871520!3!!&lat=-0.5625&zoom=0
• SEALEVEL_GLO_PHY: 1993 (sea level) https://data.marine.copernicus.eu/product/SEALEVEL_GLO_PHY_L4_MY_008_047/download?dataset=cmems_obs-sl_glo_phy-ssh_my_allsat-l4-duacs-0.25deg_P1D

[eeholmes]

Eli Tuesday

• Working on CMIP tutorial
• Thinking about tutorials in general: See https://nasa-openscapes.github.io/earthdata-cloud-cookbook/tutorials/

Example project

• create CMIP downscaled projects of SST and .... for Indian Ocean. https://assets.researchsquare.com/files/rs-905960/v1/06cd648a-721e-4291-84b3-e9e3565b1ce8.pdf?c=1673471650. I wonder why they didn't use OM4 which is 0.25 deg resolution?

[NaNa7Miiii]

Jiarui Tuesday CheckIn

• Have finished building heatmaps, please check the HEATMAP.ipynb file in my notebook folder for more details.
• Playing Around with the direction data: (the data starts from 199709 to 202012, as well as the speed data), attempts to put the data inside the LSTM model.

[NaNa7Miiii]

• Since yesterday we've discussed about the true value and predicted value of my LSTM model fed by SST data, I have made a dataframe that contains three columns: time, actual_val, predicted_val, and saved the dataframe to a .csv file called predictions.csv in my shared folder.

The percentage of the actual label being different from the predicted value is around 1.9%, indicating the model accuracy is solid

[minhphan03]

Wednesday check in:

• Finished writing notebooks
• Downloaded additional data from 9/1997-12/1999

[eeholmes]

@minhphan03 Some more zarr/xarray ideas for tutorial https://earth-env-data-science.github.io/lectures/xarray/xarray-part2.html

[NaNa7Miiii]

Jiarui Wed Checkin:

• Added youtube videos related to image classification models to my notebook
• Continue LSTM model benchmarking

[minhphan03]

Dask arrays vs NumPy arrays

https://tutorial.dask.org/02_array.html

"In other words, Dask Array implements a subset of the NumPy ndarray interface using blocked algorithms, cutting up the large array into many small arrays. This lets us compute on arrays larger than memory using all of our cores. We coordinate these blocked algorithms using Dask graphs."

"By default, Dask Array uses the threaded scheduler in order to avoid data transfer costs"

https://docs.dask.org/en/stable/why.html

"To avoid excess memory use, Dask is good at finding ways to evaluate computations in a low-memory footprint when possible by pulling in chunks of data from disk, doing the necessary processing, and throwing away intermediate values as quickly as possible. This lets analysts perform computations on moderately large datasets (100GB+) even on relatively low-power laptops."

[minhphan03]

Minh's Thursday check-in:

• Cleaning up space
• Modify & working on tutorials

[eeholmes]

• Data tutorial:
  • reading in the data
  • slicing
  • indexing
  • creating new variables. Show how to create a new variable wind speed variable (sqrt(u^2 + v^2) and direction
• Plotting the data
  • grouping, resampling plots_
  • raster plots
  • adding points to a raster plot
  • add a coastline to a raster plot
• Making bathymetry plots, specifically add contour lines
  • colormap for ocean https://matplotlib.org/cmocean/
  • download the fine scale bathymetry, make a contour map, add raster image on top
  • fine scale bathymetry. Not sure if helpful. wrong grid but maybe useful for contour lines https://coastwatch.pfeg.noaa.gov/erddap/griddap/usgsCeSrtm30v6.graph
  • https://www.pygmt.org/v0.3.0/tutorials/contour-map.html
  • one could spend a life time here -> https://docs.generic-mapping-tools.org/latest/gallery.html
  • probably not useful but pretty https://towardsdatascience.com/creating-beautiful-topography-maps-with-python-efced5507aa3
  • https://www.earthinversion.com/utilities/pygmt-high-resolution-topographic-map-in-python/

[eeholmes]

Eli Thursday checkin

• I am working on fixing the xarray resample problem. Turns out installation of zarr-eosdis-store is causing a downgrade of xarray to 0.21 which is 2 years old. I don't think we use this. Since I will be updating the docker image for dhub are there any modules that I need to add to our image? @minhphan03 @NaNa7Miiii Anything that you had to install with pip install?
• Added pip install imbalanced-learn

[NaNa7Miiii]

Jiarui Thursday Checkin:
Yesterday's result:

Result after using Dr.Eli's normalization method:

The current accuracy is:

With the new way of doing normalization(-mean), the median, especially that of data where actual_val being 0, has changed a lot in a good way. This proves the direction is correct. I am currently increasing the epoch num from 50 to 100, and dataset used will be 2000-202305 instead of 2021 to 202305. In such way, the model will be larger and hopefully the accuracy will be better.

[NaNa7Miiii]

@eeholmes Sorry Dr.Eli, the model is still running (probably the epoch is too large for the large dataset). I will keep my computer running during the night and see what's going to happen tomorrow morning. If the model didn't finish running tomorrow morning, I will retrain it and keep the epoch num to 50.

[NaNa7Miiii]

The model is still running. Please shut down the jhub sever and I will retrain it later with lower epoch num @eeholmes .

[eeholmes]

Hi Jiarui, It'll be a few more hours before I get to this so we can let it run some more. Eli

…

On Fri, Jul 14, 2023 at 8:43 AM Jiarui Yu ***@***.***> wrote: The model is still running. Please shut down the jhub sever and I will retrain it later with lower epoch num @eeholmes <https://github.com/eeholmes> . — Reply to this email directly, view it on GitHub <#10 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AATNSOQMLD7H35H3YETNWMTXQFSKXANCNFSM6AAAAAA2FFCI6U> . You are receiving this because you were mentioned.Message ID: <UW-Upwelling-Project/Summer-2023/repo-discussions/10/comments/6451106@ github.com>

[NaNa7Miiii]

Jiarui Friday CheckIn

• Will continue to train a smaller model
• made an image classification tutorial