Week 6 Check-in

[minhphan03]

---

[minhphan03]

Ideas for OceanHackWeek topics:

Indian Ocean warming to trends of Arctic Oscillation (idea: https://www.nature.com/articles/s41612-022-00279-x):

• "Momentum flux convergence by stationary waves due to the Indian Ocean warming plays positive trend of arctic oscillation, shown by a poleward shift of zonal-mean zonal winds"
• "...reduced propagating wave activity flux over North Pacific (Ocean?) due to warming also plays a role in polar vortex strengthening..."

Eli: Can also look at North Atlantic Oscillation and the Indian Ocean Dipole. Ask if + and - AO and NAO values are associated with different patterns of SST or wind or SSH (sea level anomaly) are associated. So rather than looking at average SST across the Arabian Sea, use image classification (or image clustering) to look at whether + or - AO (NAO) are associated with particular patterns.

Taking a look into relationship between wave and sea surface temperature (https://www.mdpi.com/2072-4292/15/7/1948):

• "By the seasonal variation, the interrelation of the currents, sea level anomaly, and significant wave heights (SWHs) were strong in the Indian Ocean..."
• " In the strong current areas, the distribution of the sea level was consistent with the SWHs. The monthly variation of SWHs, currents, sea surface elevation, and sea level anomalies revealed that the upward trends of SWHs and sea level anomalies were consistent from 1993–2015 over the global ocean. In the Indian Ocean, the SWHs were obviously influenced by the SST and sea surface wind stress."
• Variables to look for: significant wave heights, SLA, types of waves (breaking waves, nonbreaking waves)
  
• Breaking waves: ordinary (non-breaking) water waves are conventionally considered as separated-phase flows, in which the free-surface separates the gas and liquid components completely as shown in figure 1a. For breaking waves as mentioned in §1, the water may be entrapped with air pockets or entrained air bubbles when the crest starts to curl... At the same time, water droplets may also be ejected into the air as spray as illustrated in figure 1b.

Eli: I think one would be hard.

Observing trends between East and West Indian Ocean...? (https://www.climate.rocksea.org/research/indian-ocean-warming/)

• Can we predict the trend of SST (or any other climate variables) of one area based on data from the other one?
• "Being the warmest among the major oceans, the Indian Ocean plays a critical role in regulating the mean climate and variability of the Asian monsoon, as well as the dynamics over the tropics."
• "Our analysis of SSTs during 1901-2012 reveals a more gravid picture of the Indian Ocean warming. The focus shifts to the relatively cool western Indian Ocean. We find that the western tropical Indian Ocean has been warming for more than century at a rate faster than any other region of the tropical oceans..."

Eli: Could expand to other variables and ask if those have changed too in east versus west.

Asian Monsoon trends (https://www.mpg.de/18355039/0228-wisy-indian-ocean-warming-could-weaken-summer-monsoon-rainfall-in-south-asia-9347732-x):

• Predicting trends / anomaly detection
• " From a historical perspective, fluctuations in monsoon rainfall have been linked to the rise and fall of civilizations in the Indian subcontinent..."
• "Solar radiation is often considered the primary influencer of the Indian Summer Monsoon’s intensity, with elevated solar radiation increasing humidity, wind circulation, and ultimately precipitation..." -> variables: solar radiation, humidity, wind speed and pattern detection, precipitation (land vs ocean)
• "To identify the main driver of the monsoon’s rainfall during warm climate states, the researchers compared available reconstructions of past sea surface temperature from the Indian Ocean and found that the equatorial and tropical regions were 1.5-2.5 C warmer..."
• "when the Indian Ocean’s surface temperature rose in the past, monsoon rainfall would decrease on land and increase in the sea above the Bay of Bengal..." -> rise in rainfall in the ocean would lead to reduction of rainfall in the land? (a project on comparison would be nice)

Eli: I think one would be hard.

Rainfall in India (https://unacademy.com/content/mppsc/study-material/geography/rainfall-in-india/)

• Two types of monsoon rainfall
• Rainfall TYPE detection/classification? (Rainfall Detection and Rainfall Rate Estimation Using Microwave Attenuation, Classification of Rainfall Types Using Parsivel Disdrometer and S-Band Polarimetric Radar in Central Korea)
• Classifying types of rainfalls in Indian Ocean / India:
  West coast India's rainfall is becoming more convective: https://www.nature.com/articles/s41612-022-00258-2...
  "Model studies predict that the geographic distribution of precipitation will change in response to global warming1,2. In the tropics, these changes have been reflected as’wet-gets-wetter’ (increased rainfall in presently rainy regions)3,4 or’warmer-gets-wetter’ (increased rainfall where the rise in sea surface temperature (SST) exceeds the mean surface warming)4,5 pattern."
• Indian Ocean SST and Indian Summer Rainfall: Predictive Relationships and Their Decadal Variability
  "Using monthly gridded (4° × 4°) SST data from the Global Sea-Ice and Sea Surface Temperature (GISST) dataset that spans 1945–94, the correlation fields between the All-India Rainfall Index (AIRI) and SST fields over the tropical Indian Ocean are calculated. In the boreal fall and winter preceding the summer Indian monsoon, SST throughout the tropical Indian Ocean correlates positively with subsequent monsoon rainfall. Negative correlation occurs between SST and the AIRI in the subsequent autumn in the northern Indian Ocean only."

Eli: I think one would be hard.

Indian Ocean Dipole (IOD)/Indian Nino (?)

• Linked with SST -> prediction/classification/...?
• Affecting Monsoon:
  "Positive IOD winds over the Indian Ocean blow from east to west (from Bay of Bengal towards Arabian Sea) result in the Arabian Sea (western Indian Ocean near African Coast) being much warmer and eastern Indian Ocean around Indonesia becoming colder and dry. In the negative dipole year, reverse happens making Indonesia much warmer and rainier."
  

Eli: Could be added to idea #1.

[minhphan03]

Solar radiation, ocean surface wind and SST relationship (idea: Role of absorbed solar radiation on Indian Ocean surface temperature: A case study for calm winds using satellite data)

• Variables/keywords: mixed layer depth, sst, absorbed solar radiation at ocean surface
• Case study: MONEX-79 (During MONEX-79 an opportunity has arisen to examine a case study of a depression which occurred over the East Central Arabian Sea on 16 June 1979 with its mean position 13°N and 71 °E).
• "It could be thus concluded that in order to understand and predict SST changes, the variations in the absorbed solar radiation are an important input to be monitored." -> model?

[NaNa7Miiii]

Hi Minh,
I just have a question about the time range for the final-sample-appending.zarr. When I load the zarr file, here is what I got:

I notice the time range is from 1993 to 2001, is that our final time range? Thank you!

[minhphan03]

@NaNa7Miiii you should use the INDIAN_OCEAN_025GRID_DAILY.zarr in the data folder. I just updated it yesterday.

[eeholmes]

Ocean climate change in the Indian Ocean versus Bay of Bengal

Use GDFL-CM4 model forecasts (CMIP6), see notebook in eli-notebooks,

• How is chlorophyl, SST, wind and precipitation projected to change
• Eastern versus Western Arabian Sea. Bay of Bengal versus Arabian Sea.

Eddy and Gyres detection. Work with an oceanographer.

Machine-learning image to create ocean forecasts

SST, chlorophyll or wind or .... How far ahead can one forecast and how well?

• How well can you predict and how many days (or hours) ahead can you predict?
• Are there regions where you can predict better than others?

Species Distribution Modeling.

The project that Eli is bringing. You have locations (lat/lon/day) of, say, fish. Fit model fish = f(variables at lat/lon/time) --> a map of where you are likely to find fish.

• Tutorials to do this using your IO.zarr data set.

[NaNa7Miiii]

Jiarui Monday Checkin

I have trained 4 models, each takes half a day in the jhub environment for training. I will post the results (acc) for these four models, all of them have been stored as checkpoint files in my notebook, one can quickly load them and do fine-tunings:

Actions Applied for All of These Models:*
1. SMOTE
Since data with label 1 is less than data with label 0, we conduct over-samplings here to balance the dataset.

smote = SMOTE(random_state=42)
sst_data_res, labels_res = smote.fit_resample(sst_data_flat, all_labels_reshaped)

2. Masking
In our SST data, we have a lot of nan values in our bounding box that stands for the land part. We don't want to care about those regions, but a model cannot accept nan values contained in the input. Hence, we need to turn those nan values to 0.0 and use
model.add(Masking(mask_value=0., input_shape=(X_train.shape[1], 1))) to let our model ignore them.

1. LSTM_201501-202305.h5 (Tested on 2015-2023 All SST Data)
Architecture:

LSTM(100, activation='tanh', return_sequences=True)
LSTM(50, activation='tanh')
Dense(32, activation='relu')

Accuracy:
actual_val = 0 & predicted_val > 0.5: 290/2610, acc = 88.89%
actual_val = 1 & predicted_val > 0.5: 294/345, acc = 85.21%
actual_val = 0 & predicted_val > 0.7: 184/2610, acc = 92.95%
acutal_val = 1 & predicted_val > 0.7: 268/345, acc = 77.68%

2. LSTM(NEW)201501-202305.h5 (Tested on 2015-2023 All SST Data)
Architecture:

LSTM(150, activation='tanh', return_sequences=True, kernel_regularizer=l1_l2(l1=1e-5, l2=1e-4))
LSTM(100, activation='tanh', return_sequences=True, kernel_regularizer=l1_l2(l1=1e-5, l2=1e-4)）
LSTM(50, activation='tanh', kernel_regularizer=l1_l2(l1=1e-5, l2=1e-4)
Dense(64, activation='relu', kernel_regularizer=l1_l2(l1=1e-5, l2=1e-4)
Dropout(0.2)

Accuracy:
actual_val = 0 & predicted_val > 0.5: 961/2610, acc = 63.18%
actual_val = 1 & predicted_val > 0.5: 345/345, acc = 100%
actual_val = 0 & predicted_val > 0.73: 516/2610, acc = 80.23%
actual_val = 1 & predicted_val > 0.73: 303/345, acc = 87.83%

3. LSTM_202001-202305.h5 (Tested on 2015-2023 All SST Data)
Architecture: (same architecture above)
Accuracy:
actual_val = 0 & predicted_val > 0.5: 387/2610, acc = 85.17%
actual_val = 1 & predicted_val > 0.5: 245/345, acc = 71.01%
actual_val = 0 & predicted_val > 0.7: 320/2610, acc = 87.74%
actual_val = 1 & predicted_val > 0.7: 231/345, acc = 66.96%

4.LSTM(Stopped)_202001-202305.h5 (Tested on 2020-2023 All SST Data)
Architecture (use same layers above)

LSTM(150, activation='tanh', return_sequences=True, kernel_regularizer=l1_l2(l1=1e-5, l2=1e-4))
LSTM(100, activation='tanh', return_sequences=True, kernel_regularizer=l1_l2(l1=1e-5, l2=1e-4)）
LSTM(50, activation='tanh', kernel_regularizer=l1_l2(l1=1e-5, l2=1e-4)
Dense(64, activation='relu', kernel_regularizer=l1_l2(l1=1e-5, l2=1e-4)
Dropout(0.2)

optimizer = Adam(learning_rate=0.001)

classes = np.unique(y_train)
class_weights = class_weight.compute_sample_weight('balanced', y=y_train)
class_weight_dict = dict(zip(classes, class_weights))

early_stopping = EarlyStopping(monitor='val_loss', patience=5)

Accuracy:
actual_val = 0 & predicted_val > 0.5: 375/1088, acc = 65.53%
actual_val = 1 & predicted_val > 0.5: 101/101, acc = 100%
actual_val = 0 & predicted_val > 0.77: 335/1088, acc =69.21%
actual_val = 1 & predicted_val > 0.77:100/101, acc =99%

I use a learning_rate of 0.001 here to adjust the speed of convergence for the gradient descent algorithm, and the way class_weights work. "Class weights" and "SMOTE" are both methods to handle unbalanced category data, but they are different in details:

Class weights: This is a way to handle unbalanced data by adjusting the class weights. For a few categories, we can give them larger weights, and for the majority, smaller weights. The purpose of this is to make the model focus more on the minority categories. When the model is trained, a loss function is computed based on these weights, and misclassification of the minority categories is given a greater penalty.

SMOTE (Synthetic Minority Over-sampling Technique): this is an oversampling method that attempts to solve the imbalance problem by synthesizing new minority category samples to increase the number of minority category samples. Specifically, for minority category samples, SMOTE finds its nearest neighbor samples and then generates new samples between these samples.

I also used the early stopping mechanism to prevent overfitting: when the loss on the validation set does not improve for a number of consecutive rounds (in this model, PATIENCE is set to 5 rounds), training is stopped early.

I have added a progress bar for training a model:
Please do from tqdm.keras import TqdmCallback
and then add this callback to the model, such as

tqdm_callback = TqdmCallback(verbose=1)
model.fit(X_train, y_train, epochs=50, verbose=0, callbacks=[tqdm_callback])

In this way, the training session won't be a black box to us, and we can clearly observe and monitor the progress.

As you can see, the model accuracy data above shows tradeoff. If we want to prevent (decrease) the false negative rate, the false positive will be increased.

[minhphan03]

I'm a little bit confused on the accuracy. Are the acc value based on predictions on the validation set, and the fractions predictions on the test set?

[NaNa7Miiii]

the accuracy is based on all the datasets on the time range (trained + test).

[NaNa7Miiii]

The test acc only will be the output after the model finishes its training, or you can observe the current test acc through the progress bar.

[minhphan03]

Minh's Tuesday check-in:

New variable interest: salinity

• During the dry season (January–May), upwelling results in ocean surface water which is cool (~20–25°C), has high salinity (~34 ppt), is rich in nutrients, and supports plankton blooms. During the rainy season (May–December), the absence of upwelling results in surface waters with higher temperatures (~28–29°C), lower salinity (~30 ppt), and less nutrient and phytoplankton availability (D'Croz & O'Dea, 2007)

• Based on observations from two Argo floats that drifted and stayed around Sumatra, we investigated ocean temperature and salinity variations during several coastal upwelling events observed in 2013–2017... The observation data show intraseasonal-scale subsurface temperature cooling events with significant upward displacements of the thermocline and high-salinity water

• Download with MOTU API failed, attempt with OPeNDAP instead. Streaming and downloading is extremely slow (2 mins/monthly file * 12 months / year * 27 years ~ aprox 6hrs) STILL PROCESSING. In the process of writing a small tutorial on utilizing this protocol.

Added topography into final dataset & fix some metadata issues.

Uploaded SRTM30+ bathymetry data to jhub (around 1hr) + sliced + exported.

[NaNa7Miiii]

Jiarui Tuesday CheckIn:

1. Working on another ML architecture to improve accuracy. The previous architecture I was using was RNN's layers, which may not perform excellently in this task, so I am using CNN's layers now.
2. Comming up with an idea of the hackweek:

we could do fine-tunings for datasets related to fish in India based on pretrained CNN models. Those models were trained using the ImageNet dataset consisting of general natural objects, thus they may lack performance when detecting a specific type of fish, so we can conduct fine-tunings on those models and benchmark the results. Maybe we can also build a simple app for that.

[NaNa7Miiii]

@eeholmes Hi Dr.Eli, just wonder do we have meetings this week? Minh and I joined the meeting today and we've read each other's posts.

[NaNa7Miiii]

Jiarui Wed check-in

1. SST Model Finished (Using EfficientNetB0 Architecture):

base_model = EfficientNetB0(include_top=False, weights=None, input_shape=(processed_data.shape[1], processed_data.shape[2], 1))
I have also replaced the top classification layer with my own layers:

x = base_model.output
x = tf.keras.layers.GlobalAveragePooling2D()(x)
x = Dense(1024, activation='relu')(x)
predictions = Dense(1, activation='sigmoid')(x)
model = Model(inputs=base_model.input, outputs=predictions)

Here is the result from the model compiling:

• The test accuracy is around 99.11%, which is very high compared to those previous models using LSTM layers.
• This model training session is very fast, it takes around ten minutes, while the previous LSTM models would half a day.

Let's dive into the distribution:
The model is trained on 80% of data between 2020-01-01 to 2023-05-31, here are the predictions made by the model for all of the data in that time range (train set + test set):

Here is the histogram:

• I have made a notebook related to this model, this is the path to that notebook: shared/jiarui-notebooks/04_EfficientNet.ipynb.

2. Start testing a wind model using Minh's Zarr dataset (direction) and the same model architecture above:
The dataset seems to have a time range of [1993-01-01, 2001-12-31], and I found for some dates the matrix tends to be full of NaNs (both for sst and direction, haven't tested other variables yet), but the rest is enough for model training.
Here is the model result:

Again, the model is using the EfficientNet architecture, so the training session is fast (less than 20 mins), so feel free to reproduce the results (the file is in my shared folder, with the path shared/jiarui-notebooks/wind_model.ipynb).
Here are the distributions for all the data in the range [1993-01-01, 2001-12-31]:

• Notice: my previous SST model takes the daily data at 9AM for each day, while this Zarr file takes the daily data at 0:00 for each day. Thus, for the wind model, I made a new folder that stores the SST differences at each day's 0:00 from 1979-01 to 2023-05. I have made corresponding changes in the wind_model.ipynb file, this is just an explanation for the different paths to the labels files.

[minhphan03]

@NaNa7Miiii I'm looking at the wind_model.ipynb at the moment and curious at the output for dates that have only NaNs. Why are there so many days with NaNs? I checked out one of the dates and it definitely has some numbers in it, so can you double check?

I double-checked with the big final zarr file we have and there's no missing data there, so you should be good to go.

[NaNa7Miiii]

@minhphan03 Sorry in that loop I didn't consider my skip ranges

skip_ranges = [pd.date_range(start='1997-02-01', end='1997-02-28'), 
                          pd.date_range(start='2001-11-01', end='2001-11-30')]

(This is just a range of dates where I don't have labels)
I have re-defined the func:

I found we still have some NaN values.

Am I using the right Zarr file for testing? The file I am currently using is shared/minh-notebooks/demonstrated data/final-sample-appending.zarr. Thank you!

[minhphan03]

You should use the big zarr file in all caps in the data folder. Go back to the big folder, then data: cd ../data/INDIAN_OCEAN_025GRID_DAILY.zarr/

[NaNa7Miiii]

oh that file is cool, thank you so much!

[eeholmes]

@eeholmes Load opencv-python

Actually I don't see where Jiarui is using this so will hold off for now.

[minhphan03]

Minh's Wednesday update

I apologize @eeholmes @NaNa7Miiii for my absence at the meeting today. My phone died while I was sleeping last night and the alarm did not work, so I overslept.

I added mixed ocean layer depth and salinity into our final dataset. For the former variable, I read this paper and thought it was interesting:" Using observational climatologies for four EBUSs, we diagnose that the Peruvian upwelling system is the only one to reveal that intense upwelling coincides with deep mixed layers. " -> Can we confirm this theory using ML?

This paper reveals that "[In Moroccan Atlantic coast], the deepest MLD occur in December and February, it coincides with a weak activity of upwelling. During April, June and August, the mixed layer is shallow, accompanied with a high activity of upwelling filament. In October, the mixed layer was deeper following a persistent activity of upwelling." -> Is this also the case for Indian Ocean upwelling zones?

[minhphan03]

Minh's Thursday update

• attempt to work on putting OPeNDAP with ERDDAP server over Python but I got IP address blocked (in blacklist) for some reasons. Emailed them to ask about the issue. Need some time waiting for the response as I want to test the data for the ERDDAP downloader.

[minhphan03]

Update: emailed [email protected], who is in charge of ERDDAP but sender blocked... Waiting from [email protected] itself.

[NaNa7Miiii]

Jiarui Thursday Checkin:

• Have constructed the speed model using the zarr file. I have stored my code in the notebook wind_new.ipynb in my shared folder. The code is very user-friendly and is encapsulated into functions. So if you change the variable from speed to direction, you can have a direction model then.
• Made a plan for the ocean hack week:
  hackweek_jiarui_plan.pdf

[minhphan03]

Minh's Friday checkin:

• Finish downloading all data for CMIP6 at all time ranges. Expect to start working on building datasets next week.
• Some tutorials?

[NaNa7Miiii]

Jiarui Friday CheckIn:
Trained a speed model using Minh's dataset shared/data/INDIAN_OCEAN_025GRID_DAILY.zarr
Trained a two-channel model using wind_dir and wind_speed data

[minhphan03]

Apologies @eeholmes @NaNa7Miiii my home wifi connection ran into some issues and it’s too slow for me to join using my computer. I couldn't figure out how to fix it, so I guess I’ll restart the router to see if anything changes. My phone data plan is limited in VN so I don’t think I can set up a personal hotspot for the laptop either.

As for the progress, I’ll continue working on building the CMIP6 dataset. Let me know if anything happens, and apologies again for the late notice.

[eeholmes]

For Jiarui

• Try to fill in the missing months for diff_sst
• Save the SST and SST_diff dataframe into a csv file
• In your Jupyter Notebooks, keep working on cleaning up the code and adding documentation.