Prediction failure with training models generated with icenet<0.2.7

[bnubald]

The following issue needs accounting for in the prediction scripts:

icenet-ai/icenet#236

Error copied across from this issue:

[10-04-24 12:19:04 :DEBUG   ] - Creating converter from 3 to 5
Traceback (most recent call last):
  File "/data/hpcdata/users/jambyr/miniconda3/envs/icenet/bin/icenet_predict", line 33, in <module>
    sys.exit(load_entry_point('icenet', 'console_scripts', 'icenet_predict')())
  File "/data/hpcdata/users/jambyr/icenet/icenet2/icenet/model/predict.py", line 198, in main
    predict_forecast(
  File "/data/hpcdata/users/jambyr/icenet/icenet2/icenet/model/predict.py", line 66, in predict_forecast
    network.load_weights(network_path)
  File "/data/hpcdata/users/jambyr/miniconda3/envs/icenet27/lib/python3.8/site-packages/keras/src/utils/traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/data/hpcdata/users/jambyr/miniconda3/envs/icenet27/lib/python3.8/site-packages/keras/src/backend.py", line 4361, in _assign_value_to_variable
    variable.assign(value)
ValueError: Cannot assign value to variable ' conv2d_8/kernel:0': Shape mismatch.The variable shape (3, 3, 368, 736), and the assigned value shape (737, 368, 3, 3) are incompatible.

This needs a legacy flag (-l) added as an argument for the following statement if trying to run prediction with a training model generated using icenet<0.2.7:

icenet-pipeline/ensemble/template/icenet_predict.sh.j2

Line 39 in 3db92a0

icenet_predict -v -n {{ run.arg_filter_factor }} $PREDICT_STATIC_ARGS \{% if run.arg_testset == "true" %}

Issue is caused by a change in rounding in convolution operation in v0.2.7.