Week 8 CheckIn

[NaNa7Miiii]

[NaNa7Miiii]

Monday Checkin:

• work on the forecasting transformer model architecture:

def transformer_model(input_shape, num_heads=4, feed_forward_dim=256):
    inputs = layers.Input(shape=input_shape)
    
    # Flatten the spatial dimensions
    x_flattened = layers.TimeDistributed(layers.Flatten())(inputs)
    
    # Transformer block
    x_transformed = layers.MultiHeadAttention(num_heads=num_heads, key_dim=input_shape[-1])(x_flattened, x_flattened)
    
    # Reshape x_transformed back to its original spatial dimensions
    x_reshaped = layers.TimeDistributed(layers.Reshape((input_shape[1], input_shape[2])))(x_transformed)
    x_normalized = layers.LayerNormalization()(x_reshaped + inputs)
    x_normalized_flattened = layers.TimeDistributed(layers.Flatten())(x_normalized)
    
    ffn_output = layers.Dense(feed_forward_dim, activation="relu")(x_normalized_flattened)
    ffn_output = layers.Dense(input_shape[-1])(ffn_output)
    
    x = layers.LayerNormalization()(ffn_output)
    
    # Reduce the time dimension
    x = layers.GlobalAveragePooling1D()(x)
    
    # Decoding: Ensure the Dense layer outputs the exact number of units for reshaping
    x = layers.Dense(input_shape[1] * input_shape[2])(x)
    outputs = layers.Reshape((input_shape[1], input_shape[2]))(x)
    
    return tf.keras.models.Model(inputs=inputs, outputs=outputs)

model = transformer_model((5, 149, 181))

Epoch 20:

Epoch 20/20
21/21 [==============================] - 11s 502ms/step - loss: 0.1299 - mae: 0.2021 - val_loss: 0.3383 - val_mae: 0.2975

[NaNa7Miiii]

Tuesday Checkin:

• Adjust the Transformer model

[eeholmes]

@NaNa7Miiii Remember to post your project pitch to the Slack channel. It's helpful for the team to see that sooner than later, esp befor the all-hands.

[eeholmes]

this channel https://app.slack.com/client/TUFFMB3EE/C05JKG7FE9E

[NaNa7Miiii]

Hi Dr.Eli, Sure! I have posted the pitch on the channel. I didn't mention the sample code because I am still working on the Transformer model, but I mentioned the benchmark things.

…

On Tue, Aug 1, 2023 at 4:10 PM Eli Holmes - NOAA ***@***.***> wrote: this channel https://app.slack.com/client/TUFFMB3EE/C05JKG7FE9E <https://urldefense.com/v3/__https://app.slack.com/client/TUFFMB3EE/C05JKG7FE9E__;!!K-Hz7m0Vt54!g1SAdPeImMcNQzAk_0Gdsw0X9WDS9-t5vJt1E8NAAodx9GuUl52XN0Qy0MJzc3OZhW2xiQCr6LRV-oo-ltHM_zIX$> — Reply to this email directly, view it on GitHub <https://urldefense.com/v3/__https://github.com/UW-Upwelling-Project/Summer-2023/discussions/13*discussioncomment-6610557__;Iw!!K-Hz7m0Vt54!g1SAdPeImMcNQzAk_0Gdsw0X9WDS9-t5vJt1E8NAAodx9GuUl52XN0Qy0MJzc3OZhW2xiQCr6LRV-oo-ltJ4JKF6$>, or unsubscribe <https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AXI254K3WETLW26CFFFWZSDXTGEHXANCNFSM6AAAAAA265ENLQ__;!!K-Hz7m0Vt54!g1SAdPeImMcNQzAk_0Gdsw0X9WDS9-t5vJt1E8NAAodx9GuUl52XN0Qy0MJzc3OZhW2xiQCr6LRV-oo-ltiX4k_i$> . You are receiving this because you were mentioned.Message ID: <UW-Upwelling-Project/Summer-2023/repo-discussions/13/comments/6610557@ github.com>

[NaNa7Miiii]

Wed Checkin:

• Adjusts the transformer model
• Made a draft code for a 3D CNN model

[eeholmes]

Some papers on remote-sensing image forecasting

https://ieeexplore.ieee.org/abstract/document/7752890
https://www.nature.com/articles/s41598-020-79148-7
A review: https://arxiv.org/abs/1906.04928

CNN. Convolutional neural networks (CNN) is a class of
deep, feed-forward artificial neural networks that are applied
to analyze visual imagery. A typical CNN model usually
contains the following layers as shown in Fig. 7: the input
layer, the convolutional layer, the pooling layer, the fullyconnected layer and the output layer. The convolutional layer
will determine the output of neurons of which are connected
to local regions of the input through the calculation of the
scalar product between their weights and the region connected
to the input volume. The pooling layer will then simply
perform downsampling along the spatial dimensionality of the
given input to reduce the number of parameters. The fullyconnected layers will connect every neuron in one layer to
every neuron in the next layer to learn the final feature vectors
for classification. It is in principle the same as the traditional
multi-layer perceptron neural network (MLP). Compared with
traditional MLPs, CNNs have the following distinguishing
features that make them achieve much generalization on vision
problems: 3D volumes of neurons, local connectivity and
shared weights. CNN is designed to process image data. Due to
its powerful ability in capturing the correlations in the spatial
domain, it is now widely used in m

[eeholmes]

Ideas to explore (later):

1. Convert daily data to monthly averages
2. Predict next month's image from the last 1-12 monthly images
3. Questions
   • Does the forecast improve with more months of data
   • Is the forecast of next month with the model better than just using last's month's image

[NaNa7Miiii]

Thursday Checkin:
New Arch for Efficientnet + Transformer:

def build_model(input_shape=(5, 149, 181, 1), num_heads=4, dropout_rate=0.1, regularizer=tf.keras.regularizers.l2(0.001)):
    inputs = Input(shape=input_shape)
    
    masked_inputs = Masking(mask_value=0.0)(inputs)
    
    # TimeDistributed EfficientNet
    x = TimeDistributed(EfficientNetB0(include_top=False, weights=None, pooling='avg'))(inputs)

    transformer_block = MultiHeadSelfAttention(embed_size=x.shape[-1], num_heads=num_heads)
    x = transformer_block(x)
    
    x = GlobalAveragePooling1D()(x)

    # Decode the features
    x = Dense(149 * 181)(x)
    x = Reshape((149, 181, 1))(x)

    # Transpose Convolution layers
    x = Conv2DTranspose(32, (3,3), strides=(1,1), padding='same')(x)
    x = Conv2DTranspose(1, (3,3), strides=(1,1), padding='same')(x)
    
    outputs = x

    model = Model(inputs=inputs, outputs=outputs)
    return model

model = build_model(input_shape=(5, 149, 181, 1))
model.compile(optimizer="adam", loss="mse")

This step x = TimeDistributed(EfficientNetB0(include_top=False, weights=None, pooling='avg'))(inputs) would apply EfficientnetB0 to every timestep of the input data for feature extraction. I tested the model using different params, here is the progress for 4pm (it's not finish training yet), but you can see the val_mae tends to decrease in a fast pace:

[NaNa7Miiii]

Hi @eeholmes, May I ask if the jhub crashed? I cannot open it from my side and it says service unavailable

[NaNa7Miiii]

nevermind, it came back when I did some refreshes! Thank you!

[NaNa7Miiii]

The models I have so far reach a validation MAE of 0.3, but the null mae is around 0.15, still trying to do fine-tunings

[NaNa7Miiii]