Week 4 Check-in (short week)

[eeholmes]

Eli

• Set up https://github.com/Hackweek-ITCOocean for prepping our tutorials.
  • Add Minh and Jiarui and discuss how to create new tutorials
• Contacted OHW to get the participation of Minh (virtual) and Jiarui (in-person) fixed.

Notes

• AGU 2020 tutorials https://github.com/pangeo-gallery/osm2020tutorial

[minhphan03]

Wednesday's update:

• Downloaded, cleaned, and synthesized all variables

[eeholmes]

Try different chunking on Zarr files to see what has faster access.

https://discourse.pangeo.io/t/understanding-optimal-zarr-chunking-scheme-for-a-climatology/2335

https://discourse.pangeo.io/t/netcdf-to-zarr-best-practices/1119/13

[minhphan03]

https://discourse.pangeo.io/t/processing-large-too-large-for-memory-xarray-datasets-and-writing-to-netcdf/1724/3 appending by time, chunk of every 100 days.

[NaNa7Miiii]

Jiarui Wed & Thurs Check-In:

Summarized all the data from 1979-01 to 2023-05 into .nc files in this format:

The SST Diff is the difference between each pair (a point on the 100km buffer along the bounding box and its paired (nearest) point on the 300km buffer along the coastal line), we can apply a flexible threshold on those differences to get labels. Here is an example:

import xarray as xr
import pandas as pd

# Open the dataset and convert it to a pandas DataFrame
temp = xr.open_dataset('shared/sst_data/201407_sst_diff.nc')
temp_df = temp.to_dataframe()

# Convert the 'time' column to date only format
temp_df.index = pd.to_datetime(temp_df.index).date

# Reset index to create 'time' column
temp_df = temp_df.reset_index()

# Create a new column 'entry_id' which enumerates each entry within each date
temp_df['entry_id'] = temp_df.groupby('time').cumcount()

# Filter dataframe for rows where 'entry_id' is 22, 23 or 24
filtered_df = temp_df[temp_df['entry_id'].isin([22,23,24])]

# Create a label column - set to 1 if SST_Diff > threshold, else set to 0
threshold = 1
filtered_df['label'] = np.where(filtered_df['SST_Diff'] > threshold, 1, 0)

# Group by 'time' and check if all 'label' in a group is 1, if so, set group_label to 1, else 0
group_labels = filtered_df.groupby('time')['label'].apply(lambda x: 1 if x.sum() == 3 else 0)

Visualization on heatmap (This example uses data from 1979-01)

[NaNa7Miiii]

Jiarui Thursday Checkin:

• Updated the threshold data, and saved the files into the shared/sst_diff folder
• Created a tutorial for a demo using LSTM to do training, saved it as shared/jiarui-notebooks/Demo_LSTM.ipynb
• Used all of the data from 2010-2020 to do training / testing for the LSTM model

[minhphan03]

Finished 1 tutorial (NASA Earthdata downloader)
Working on the next (ERDDAP downloader)
Added metadata and rechunk everything

[eeholmes]

Friday

Eli

• Met with Jim C at 2i2c re setting up a JupyterHub for the INCOIS hackweek.
• Sent some emails trying to get funding

Minh

• Worked on ERDDAP tutorial

Jiarui

• Worked on tutorial for image classification
• worked on the threshold

[eeholmes]

Look into dask-tensorflow

• https://ml.dask.org/keras.html
• https://matthewrocklin.com/blog/work/2017/02/11/dask-tensorflow