Show pre-results (without need for simulation run) on results page

[henhuy]

Closes #60

TODOS:

• Activate pre-results in results dropdown
• Store simulation parameters server-side (pre-results ID)
• deliver pre-results ID when requesting backend
• Rewrite calculations to use only data from datapackage and pre-results parameters instead of simulation results (see TODOS below)

Calculations which have to be refactored:

• electricity_demand_per_municipality_2045
• heat_demand_per_municipality_2045

[henhuy]

Hey @nesnoj,

I just wanted to refactor electricity and heat demand calculation for 2045 (as currently this is calculated from simulation results instead of datapackage), but I stumble upon following lines.

epp-app/digiplan/map/calculations.py

Lines 358 to 361 in 8400d03

	sector_shares = pd.DataFrame(
	{sector: demands_per_sector[sector]["2022"] / demands_per_sector[sector]["2022"].sum() for sector in mappings},
	)
	demand = sector_shares * demand.values

Do you know why we multiply values from 2045 with shares from 2022?? Or is it a bug?
IMO, the shares are set up by user via GUI and should already be considered in demand sequences via hooks.
Cannot explain those lines - maybe you can enlighten me? THX

[nesnoj]

Yes, because 2022 is the reference which will be scaled according to the slider settings (relative changes), cf. my commit message for this line. Also, I changed the hooks accordingly in the same commit those days. If I recall this correctly, there was an error in the calculation before..
But maybe I didn't fully understand the consideration in the hooks then - could you adjust this (back) and check if the passed energy amount to the esys equals the actual demand scaled by the slider settings?

PS: BTW, this is done for heat the very same way.. (line 359)

[henhuy]

• Implemented pre results for heat and electricity (including capita results)
• Activated chart results per default

Note:

• Choropleths show many black areas - this is due to missing data for most municipalities (as data is from digiplan, but municipalities are already present for empowerplan)
• Same goes for population dependent results, as population is from digiplan and does not fit
• Chart results must be marked as not ready yet if simulation is still runnning -> New PR for @bmlancien?
• Results dropdown elements depending on simulation are disabled -> no spinner next to option shown...

[henhuy]

Please review @nesnoj
As said above, we could move pre-result design to extra issue if necessary

[nesnoj]

Thx @henhuy!

While haven't checked every line, it looks plausible to me. Some notes and questions:

• The deactivation of dropdown elements is great
• I do not see any changes in the result charts yet
• When I select an available entry from dropdown while simulation is running (e.g. heat demand) the choropleth dowsn't show up and logs say TypeError: can't multiply sequence by non-int of type 'float'
• I extended the list PRE_RESULTS for the menu in menu.js to cover more results (actually all of the current entries can be precalculated)
• On the NaN/black areas: Data is available (some are dummy though) on the WAM-Server
• (Why did you implement the pre-results as model? Design-driven decision?)

• Chart results must be marked as not ready yet if simulation is still runnning -> New PR for @bmlancien?

Yes, will create..

[henhuy]

I do not see any changes in the result charts yet

Will move this to extra issue

When I select an available entry from dropdown while simulation is running (e.g. heat demand) the choropleth dowsn't show up and logs say TypeError: can't multiply sequence by non-int of type 'float'

This works on my side. Do you have different data? (I'm using data from server)

I extended the list PRE_RESULTS for the menu in menu.js to cover more results (actually all of the current entries can be precalculated)

I have to refactor calculations in order to not use simulation results. Maybe we have to discuss this for some calculations, as I'm not sure ALL selections can be pre-calculated...

On the NaN/black areas: Data is available (some are dummy though) on the WAM-Server

No more black areas on my side. Downloaded data from server.

(Why did you implement the pre-results as model? Design-driven decision?)

Internally, the map stores the pre-result ID and requests choropleths/popups/charts from backend using this ID. Alternative would be to store all form inputs of the user and query them within request. Maybe I switch to this, as waiting for pre-result ID takes too long IMO

[henhuy]

Should be ready now.
All results in dropdown are now available before finish of simulation run.
Also removed pre-results model.

When I select an available entry from dropdown while simulation is running (e.g. heat demand) the choropleth dowsn't show up and logs say TypeError: can't multiply sequence by non-int of type 'float'

Is this still happening on your side?

Please review again. THX

[nesnoj]

Should be ready now. All results in dropdown are now available before finish of simulation run. Also removed pre-results model.

When I select an available entry from dropdown while simulation is running (e.g. heat demand) the choropleth dowsn't show up and logs say TypeError: can't multiply sequence by non-int of type 'float'

Is this still happening on your side?

Please review again. THX

No, but

  File "/app/digiplan/map/views.py", line 196, in get_charts
    {lookup: charts.CHARTS[lookup](user_settings=map_state).render() for lookup in lookups},
  File "/app/digiplan/map/views.py", line 196, in <dictcomp>
    {lookup: charts.CHARTS[lookup](user_settings=map_state).render() for lookup in lookups},
  File "/app/digiplan/map/charts.py", line 117, in __init__
    super().__init__()
  File "/app/digiplan/map/charts.py", line 28, in __init__
    self.chart_data = chart_data if chart_data is not None else self.get_chart_data()
  File "/app/digiplan/map/charts.py", line 492, in get_chart_data
    future_data = calculations.energies_per_municipality_2045(self.user_settings).sum().astype(float) * 1e-3
  File "/venv/lib/python3.9/site-packages/pandas/core/generic.py", line 6643, in astype
    new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors)
  File "/venv/lib/python3.9/site-packages/pandas/core/internals/managers.py", line 430, in astype
    return self.apply(
  File "/venv/lib/python3.9/site-packages/pandas/core/internals/managers.py", line 363, in apply
    applied = getattr(b, f)(**kwargs)
  File "/venv/lib/python3.9/site-packages/pandas/core/internals/blocks.py", line 758, in astype
    new_values = astype_array_safe(values, dtype, copy=copy, errors=errors)
  File "/venv/lib/python3.9/site-packages/pandas/core/dtypes/astype.py", line 237, in astype_array_safe
    new_values = astype_array(values, dtype, copy=copy)
  File "/venv/lib/python3.9/site-packages/pandas/core/dtypes/astype.py", line 182, in astype_array
    values = _astype_nansafe(values, dtype, copy=copy)
  File "/venv/lib/python3.9/site-packages/pandas/core/dtypes/astype.py", line 133, in _astype_nansafe
    return arr.astype(dtype, copy=True)
ValueError: could not convert string to float: ''

[henhuy]

No, but

Ah yes! I did one change to your data.
Could you change "bioenergy": "none" to "bioenergy": null and try again?
As otherwise string "none" is causing this error.
And would it be possible to change it in your pipeline as well if it works?

[nesnoj]

Works out!