Unable to publish data from MPC when dayahead has been saved to file

[RudolfRendier]

Describe the bug
I run a dayahead (publish_prefix=dh_) and save it to file.
If I run MPC afterwards it'll try to publish from those files and will fail.
It looks like it finds those files in the /entities-folder, but they don't have the correct prefix.
It throws this error, and doesn't publish the results from 'opt_res_latest.csv'

2024-11-07 14:08:27,397 - web_server - INFO - Status: Optimal
2024-11-07 14:08:27,405 - web_server - INFO - Total value of the Cost function = -0.51
2024-11-07 14:08:29,686 - web_server - INFO -  >> Sending rendered template table data
2024-11-07 14:08:34,657 - web_server - INFO - Passed runtime parameters: {'publish_prefix': 'mpc_'}
2024-11-07 14:08:34,657 - web_server - INFO -  >> Setting input data dict
2024-11-07 14:08:34,657 - web_server - INFO - Setting up needed data
2024-11-07 14:08:34,677 - web_server - INFO -  >> Publishing data...
2024-11-07 14:08:34,696 - web_server - INFO - Publishing data to HASS instance
2024-11-07 14:09:28,706 - web_server - ERROR - Exception on /action/publish-data [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 1473, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 882, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 880, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 865, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/emhass/src/emhass/web_server.py", line 343, in action_call
    _ = publish_data(input_data_dict, app.logger)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/emhass/src/emhass/command_line.py", line 813, in publish_data
    opt_res = pd.concat(opt_res_list, axis=1)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pandas/core/reshape/concat.py", line 372, in concat
    op = _Concatenator(
         ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pandas/core/reshape/concat.py", line 429, in __init__
    raise ValueError("No objects to concatenate")
ValueError: No objects to concatenate
2024-11-07 14:09:38,387 - web_server - INFO - Passed runtime parameters: {'publish_prefix': 'all'}
2024-11-07 14:09:38,387 - web_server - INFO -  >> Setting input data dict
2024-11-07 14:09:38,387 - web_server - INFO - Setting up needed data
2024-11-07 14:09:38,407 - web_server - INFO -  >> Publishing data...
2024-11-07 14:09:38,407 - web_server - INFO - Publishing data to HASS instance
2024-11-07 14:09:41,607 - web_server - INFO - Successfully posted to sensor.dh_p_pv_forecast = 13
2024-11-07 14:09:41,965 - web_server - INFO - Successfully posted to sensor.dh_unit_prod_price = 0.055
2024-11-07 14:09:42,247 - web_server - INFO - Successfully posted to sensor.dh_unit_load_cost = 0.2664
2024-11-07 14:09:42,507 - web_server - INFO - Successfully posted to sensor.dh_optim_status = Optimal
2024-11-07 14:09:42,977 - web_server - INFO - Successfully posted to sensor.dh_p_deferrable1 = 0
2024-11-07 14:09:43,246 - web_server - INFO - Successfully posted to sensor.dh_total_cost_fun_value = -1.77
2024-11-07 14:09:43,537 - web_server - INFO - Successfully posted to sensor.dh_p_load_forecast = 367.29
2024-11-07 14:09:43,787 - web_server - INFO - Successfully posted to sensor.dh_p_grid_forecast = 354.29
2024-11-07 14:09:44,066 - web_server - INFO - Successfully posted to sensor.dh_p_batt_forecast = 0
2024-11-07 14:09:44,307 - web_server - INFO - Successfully posted to sensor.dh_soc_batt_forecast = 86
2024-11-07 14:09:44,557 - web_server - INFO - Successfully posted to sensor.dh_p_deferrable0 = 0
2024-11-07 14:09:44,656 - web_server - INFO -  >> Sending rendered template table data

To Reproduce
Run a dayahead with the following parameters, I guess the first two are the most relevant
{"entity_save": True, "publish_prefix": "dh_", "battery_maximum_state_of_charge": 1, "battery_minimum_state_of_charge": 0.05, "battery_target_state_of_charge": 0.86, "battery_discharge_power_max": 400, "battery_charge_power_max": 350, "weather_forecast_cache_only": True, "pv_power_forecast": [111, 58, 13, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 32, 97, 199, 318, 501, 728, 857, 912, 903, 857, 747, 634, 498, 357, 223], "load_cost_forecast": [0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26402, 0.26402, 0.26402, 0.26402, 0.26402, 0.26402, 0.26402, 0.26402, 0.26402, 0.26402, 0.26402, 0.26402, 0.26402, 0.26402, 0.26402, 0.26402, 0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26644]}

Then run the MPC:
{"publish_prefix": "mpc_", "prediction_horizon": 18, "alpha": 0.25, "beta": 0.75, "battery_discharge_power_max": 400, "battery_charge_power_max": 350, "prod_price_forecast": [0.055, 0.055, 0.055, 0.055, 0.055, 0.055, 0.055, 0.055, 0.055, 0.055, 0.055, 0.055, 0.055, 0.055, 0.055, 0.055, 0.055, 0.055], "soc_final": 0.6, "soc_init": 0.86, "number_of_deferrable_loads": 2, "operating_hours_of_each_deferrable_load": [0, 0], "pv_power_forecast": [111, 58, 13, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], "load_power_forecast": [446, 228, 338, 367, 395, 354, 482, 620, 1014, 1056, 807, 491, 476, 611, 478, 365, 181, 184], "load_cost_forecast": [0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26644, 0.26402, 0.26402]}

Then publish the results from the MPC:
{"publish_prefix": "mpc_"}

I was experimenting with this save_entity-parameter and noticed a typo in the documentation, that has me thinking it was a string "true" from the second example.

# RUN dayahead, with optimization_time_step=30 (default), prefix=dh_, save entity
curl -i -H 'Content-Type:application/json' -X POST -d '{"entity_save": true, "publish_prefix":"dh_"}' http://localhost:5000/action/dayahead-optim
# RUN MPC, with optimization_time_step=5, prefix=mpc_, save entity
curl -i -H 'Content-Type:application/json' -X POST -d '{"entity_save": true", 'optimization_time_step':5,"publish_prefix":"mpc_"}' http://localhost:5000/action/naive-mpc-optim

Expected behavior
I was hoping to see the results from the MPC, in variables prefixed with mpc_ in Home Assistant
This works when I also use entity_save for the MPC.

Screenshots
N/A/

Home Assistant installation type

• Home Assistant OS

Your hardware

• OS: HA OS
• Architecture: amd64

EMHASS installation type

• Add-on

Additional context
None

[GeoDerp]

Thanks for the detailed issue @RudolfRendier. There is likely a bug as it should use opt_res when it can't find a file. Will look into this later.