diff --git a/desenvolupadors/2023-08-28-alta_planta/index.html b/desenvolupadors/2023-08-28-alta_planta/index.html index 626a1c3..e6cb181 100644 --- a/desenvolupadors/2023-08-28-alta_planta/index.html +++ b/desenvolupadors/2023-08-28-alta_planta/index.html @@ -1606,10 +1606,19 @@

Afegir planta per meteologicaAfegir planta al script de SolarGIS

S'ha de modificar el codi de plantmonitor per incloure la nova planta. Podeu trobar un exemple de modificacions github

Amb la MR aprovada i fusionada, s'ha de fer git pull des de el servidor de producció i reiniciar el servei de plantmonitor.

+
ssh someone@somewhere.lan
+source /home/<someone>/Envs/plantmonitor/bin/activate
+cd /home/<someone>/somenergia/plantmonitor
+git pull
+python -m scripts.solargis --help
+python -m scripts.solargis [OPTIONS] DBAPI SOLARGIS_API_KEY FROM_DATE TO_DATE [PLANT_IDS]
+sudo supervisorctl restart plantmonitor
+
+

Aqui, someone és el nom de l'usuari que té accés a somewhere.lan, i PLANT_IDS són els ids de les plantes que vau afegir a l'API de SolarGIS amb la PR anterior. Demaneu accés a somewhere.lan a l'equip de dades.

La API de SolarGIS requereix un token d'accés i es crida remotament amb supervisord, amb codi al repositori de plantmonitor.

L'API de SolarGIS és SOAP

-

L'API es del tipus SOAP i no REST. És a dir, demana un XML amb les dades de la planta.

+

L'API es del tipus SOAP i no REST. És a dir, demana un XML amb les dades de la planta. Podeu trobar informació sobre com cridar-la a la seva documentació d'usuari.

Dades requerides per aparells

Afegir Comptadors a l'ERP

diff --git a/search/search_index.json b/search/search_index.json index 3341094..7e3cfd7 100644 --- a/search/search_index.json +++ b/search/search_index.json @@ -1 +1 @@ -{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Sobre Som Energia Jardiner","text":"

Documentaci\u00f3 per eines desenvolupades en el context del projecte somenergia-jardiner de Som Energia.

"},{"location":"#repositoris","title":"Repositoris","text":""},{"location":"#accessos-directes","title":"Accessos directes","text":""},{"location":"adr/2022-10-11-dbt_workflow_meter_alarms/","title":"Com processem les alarmes en DBT?","text":""},{"location":"adr/2022-10-11-dbt_workflow_meter_alarms/#context-and-problem-statement","title":"Context and Problem Statement","text":"

Alarmes decomptadors venen en batches que poden venir molt tard.

"},{"location":"adr/2022-10-11-dbt_workflow_meter_alarms/#decision-drivers","title":"Decision Drivers","text":"

unsupported ara mateix

"},{"location":"adr/2022-10-11-dbt_workflow_meter_alarms/#considered-options","title":"Considered Options","text":"
  1. Calcular l'alarma de les \u00faltimes N hores (rang definit per l'alarma)
  2. Calcular per cada row si en aquell moment hi havia alarma o no

pseudocode-ish idea: Energy last 24h is 0 -> alarm

select sum(energy) from select lag(energy, partition by device order by date desc) from (select * from registry where device1 and date > last_date_1 or device2 and date > last_date_2) group

on last_date = select max(date) from processed group by device

  1. Union amb jinja de 2

select '' as lectura_date, '' as device, '' as alarm

calculated_alarms:

{% foreach row in meters%} UNION opci\u00f3 2 select max(energy) > 0 from registry where device = row['device'] and date > row['date'] - 24h

select from {{ this }} order by date desc limit 1

materialitzada: select max(date) from calculated_alarms group by device

  1. incremental de dbt + foreach device

{{ config(materialized='incremental') }}

select date, max(energy) > 0 from registry --> caldr\u00e0 un lag o algo

from raw_app_data.events

{% if is_incremental() %} where event_time > (select max(event_time) from {{ this }})

quan entri un device nou hem de comprovar que where event_time > NULL torna totsels resultats

source

"},{"location":"adr/2022-10-11-dbt_workflow_meter_alarms/#decision-outcome","title":"Decision Outcome","text":"

Chosen option: triem l'opci\u00f3 2. per suportar calcular alarmes en el passat

"},{"location":"adr/2022-10-11-dbt_workflow_meter_alarms/#positive-consequences","title":"Positive Consequences","text":""},{"location":"adr/2022-10-11-dbt_workflow_meter_alarms/#negative-consequences","title":"Negative Consequences","text":""},{"location":"adr/2022-10-14-meter_alarm_logic/","title":"Meter alarm logic","text":"

Technical Story: [description | ticket/issue URL]

"},{"location":"adr/2022-10-14-meter_alarm_logic/#context-and-problem-statement","title":"Context and Problem Statement","text":"

Les alarmes de comptadors s'han de notificar.

"},{"location":"adr/2022-10-14-meter_alarm_logic/#decision-drivers","title":"Decision Drivers","text":"

coberts: * moxa ens arriben les lectures cada 24h * hi ha comptadors sense lectures durant temps * la notificaci\u00f3 diuen que la volen un cop al dia al llegir del moxa

"},{"location":"adr/2022-10-14-meter_alarm_logic/#considered-options","title":"Considered Options","text":""},{"location":"adr/2022-10-14-meter_alarm_logic/#decision-outcome","title":"Decision Outcome","text":""},{"location":"adr/2022-10-14-meter_alarm_logic/#positive-consequences","title":"Positive Consequences","text":""},{"location":"adr/2022-10-14-meter_alarm_logic/#negative-consequences","title":"Negative Consequences","text":"

podem

"},{"location":"adr/2022-10-14-meter_alarm_logic/#pros-and-cons-of-the-options","title":"Pros and Cons of the Options","text":""},{"location":"adr/2022-10-14-meter_alarm_logic/#guardar-taula-i-fer-diff","title":"guardar taula i fer diff","text":"

ho farem amb un pandas/sql f\u00e0cil per no posar dbt a producci\u00f3 encara

--> ho acabar\u00e0 fent dbt a mig termini

"},{"location":"adr/2022-10-14-meter_alarm_logic/#links","title":"Links","text":""},{"location":"adr/2022-10-14-spines/","title":"Spines","text":"

Technical Story: [description | ticket/issue URL]

"},{"location":"adr/2022-10-14-spines/#context-and-problem-statement","title":"Context and Problem Statement","text":"

[Podem fer un spine amb un left join sense condici\u00f3 contra plant i meter per tenir el carret buit i despr\u00e9s fer un left join amb les lectures

dbt_spine left join plant left join meter on plant.id = meter.plant

"},{"location":"adr/2022-10-14-spines/#decision-drivers","title":"Decision Drivers","text":""},{"location":"adr/2022-10-14-spines/#considered-options","title":"Considered Options","text":""},{"location":"adr/2022-10-14-spines/#decision-outcome","title":"Decision Outcome","text":"

Chosen option: \"[option 1]\", because [justification. e.g., only option, which meets k.o. criterion decision driver | which resolves force force | \u2026 | comes out best (see below)].

"},{"location":"adr/2022-10-14-spines/#positive-consequences","title":"Positive Consequences","text":""},{"location":"adr/2022-10-14-spines/#negative-consequences","title":"Negative Consequences","text":""},{"location":"adr/2022-10-14-spines/#pros-and-cons-of-the-options","title":"Pros and Cons of the Options","text":""},{"location":"adr/2022-10-14-spines/#option-1","title":"[option 1]","text":"

[example | description | pointer to more information | \u2026]

"},{"location":"adr/2022-10-14-spines/#option-2","title":"[option 2]","text":"

[example | description | pointer to more information | \u2026]

"},{"location":"adr/2022-10-14-spines/#option-3","title":"[option 3]","text":"

[example | description | pointer to more information | \u2026]

"},{"location":"adr/2022-10-14-spines/#links","title":"Links","text":""},{"location":"adr/2022-10-26-real-time_alarms/","title":"Real time alarms","text":"

Technical Story: [description | ticket/issue URL]

"},{"location":"adr/2022-10-26-real-time_alarms/#context-and-problem-statement","title":"Context and Problem Statement","text":"

Necessitem alertes d'incid\u00e8ncies dels diferents dispositius de les plantes. Cada dispositiu t\u00e9 especificitats: granularitat temporal de les lectures, lag natural en al c\u00e0rrega de lectures i casu\u00edstiques concretes que es consideren errors.

Dispositius: - Meter ip Granularitat lectura: 1 hora Cada quan arriben les lectures: 20 min (per\u00f2 ERP sincronitza cada 2 hores) Valors historics updatable: No actualment (a l'erp s\u00ed) Poden apar\u00e8ixer lectures antigues: No actualment (a l'erp s\u00ed)

Casu\u00edstica: Meter \u00e9s variable enla cad\u00e8ncia d'enviar lectures (sovint tarda a reconnectar)

"},{"location":"adr/2022-10-26-real-time_alarms/#decision-drivers","title":"Decision Drivers","text":""},{"location":"adr/2022-10-26-real-time_alarms/#considered-options","title":"Considered Options","text":""},{"location":"adr/2022-10-26-real-time_alarms/#decision-outcome","title":"Decision Outcome","text":"

Hem decidit atacar la opci\u00f3 HI i veure si podem reduir a 5 minuts d'execuci\u00f3 amb incremental. Probablement no ser\u00e0 possible i necessitarem un fast-track (opci\u00f3 RL) per les alertes real-time.24h

Aquest fast track s'executar\u00e0 cincminutalment o podem veure si es podria fer en streaming EL -> T -> N(otificaci\u00f3) Ser\u00e0 exclusiu per alertes i amb poca consolidaci\u00f3 a base de dades i no revisitar\u00e0 el passat. Nom\u00e9s NOW()-ish.

"},{"location":"adr/2022-10-26-real-time_alarms/#positive-consequences","title":"Positive Consequences","text":""},{"location":"adr/2022-10-26-real-time_alarms/#negative-consequences","title":"Negative Consequences","text":""},{"location":"adr/2022-10-26-real-time_alarms/#pros-and-cons-of-the-options","title":"Pros and Cons of the Options","text":""},{"location":"adr/2022-10-26-real-time_alarms/#hi-calcular-la-taula-historica","title":"HI - Calcular la taula hist\u00f2rica","text":"

[example | description | pointer to more information | \u2026] 

DAG:\n(production_inverter) -> (alarm) -> (alrm2)\n\nwith ( Taula production_inverter (registry gapfilled, clean, alarms )) as foo\n\ntime     | inverter_id | energy | n_readings | alarm_no_reading | alarm_zero_daylight\n10:00:00      16           34          1            FALSE               FALSE\n09:55:00      16         0/NULL        0            TRUE                NULL\n09:50:00      16            0          1            FALSE               TRUE\n\nselect last row of foo where alarm = TRUE\n
"},{"location":"adr/2022-10-26-real-time_alarms/#rt-calcular-una-taula-real-time-i-una-historica-mes-lenta","title":"RT - Calcular una taula real-time i una hist\u00f2rica m\u00e9s lenta","text":"

[example | description | pointer to more information | \u2026]

RT:

real-time, per les alarmes que necessiten notificaci\u00f3 immediata, i per cada tipus d'alarma 

with foo as (\nselect\n    count(energy),\n    max(energy)\nfrom registry\nwhere NOW() - device.alarm_offset < time\ngroup by device\n)\nselect\n now() as time,\n count = 0 as no_reading_alarm,\n max = 0 as no_energy_alarm\nfrom foo\n
notify OK/KO en aquella execuci\u00f3 fent un diff de la taula anterior com ara

hist\u00f2ric: com tenim ara, al final del dia o d'una setmana, re-calcular les alarmes hist\u00f2ricament

"},{"location":"adr/2022-10-26-real-time_alarms/#st-trobar-la-manera-de-fer-stream-enlloc-de-batch","title":"ST- Trobar la manera de fer stream enlloc de batch","text":""},{"location":"adr/2022-10-26-real-time_alarms/#links","title":"Links","text":""},{"location":"adr/2023-06-13-obt_plantes/","title":"Pipeline de dades revisited","text":"

Technical Story:

Com ser\u00e0 la obt? horaria calculada un cop cada hora? I quan passem a comptadors 15-minutals, segurem amb dbt run 15-minutal? Quina frescor de les dades cal? --> posem-ho a un adr diferent

"},{"location":"adr/2023-06-13-obt_plantes/#context-and-problem-statement","title":"Context and Problem Statement","text":"

Si les dades cincminutals s\u00f3n horaries, la obt de tots els indicadors tamb\u00e9?

Passarem a comptadors 15minutals en algun moment proper.

"},{"location":"adr/2023-06-13-obt_plantes/#decision-drivers","title":"Decision Drivers","text":""},{"location":"adr/2023-06-13-obt_plantes/#considered-options","title":"Considered Options","text":""},{"location":"adr/2023-06-13-obt_plantes/#decision-outcome","title":"Decision Outcome","text":"

Chosen option: \"[option 1]\", because [justification. e.g., only option, which meets k.o. criterion decision driver | which resolves force force | \u2026 | comes out best (see below)].

"},{"location":"adr/2023-06-13-obt_plantes/#positive-consequences","title":"Positive Consequences","text":""},{"location":"adr/2023-06-13-obt_plantes/#negative-consequences","title":"Negative Consequences","text":""},{"location":"adr/2023-06-13-obt_plantes/#pros-and-cons-of-the-options","title":"Pros and Cons of the Options","text":""},{"location":"adr/2023-06-13-obt_plantes/#option-1","title":"[option 1]","text":"

[example | description | pointer to more information | \u2026]

"},{"location":"adr/2023-06-13-obt_plantes/#option-2","title":"[option 2]","text":"

[example | description | pointer to more information | \u2026]

"},{"location":"adr/2023-06-13-obt_plantes/#option-3","title":"[option 3]","text":"

[example | description | pointer to more information | \u2026]

"},{"location":"adr/2023-06-13-obt_plantes/#links","title":"Links","text":""},{"location":"adr/2023-06-13-pipeline_dades_llargues/","title":"Pipeline de dades revisited","text":"

Technical Story:

https://trello.com/c/rmCTOJrH/205-decidir-com-farem-lobt-i-el-cincminutal-r%C3%A0pid

Passem a una macrotaula de dades llargues clau-valor per a les dades cincminutals?

Com ser\u00e0 la obt? horaria calculada un cop cada hora? I quan passem a comptadors 15-minutals, segurem amb dbt run 15-minutal? Quina frescor de les dades cal? --> posem la discussi\u00f3 de la obt a un adr diferent

"},{"location":"adr/2023-06-13-pipeline_dades_llargues/#context-and-problem-statement","title":"Context and Problem Statement","text":"

Ara que sabem molt m\u00e9s dbt i que incorporarem un prove\u00efdor de dades cincminutals amb dades llargues que deprecar\u00e0 l'esquema antic, ens plantegem si haur\u00edem de basar en una macrotaula de dades llargues per les dades cincminutals.

"},{"location":"adr/2023-06-13-pipeline_dades_llargues/#decision-drivers","title":"Decision Drivers","text":""},{"location":"adr/2023-06-13-pipeline_dades_llargues/#considered-options","title":"Considered Options","text":"
  1. normalitzar dset i fer un multisource normalitzat com est\u00e0 ara
  2. desnormalitzar lo que hi ha ara i fer un multisource com ens envia dset
  3. fer una pseudo-obt llarga a partir de dset i fer le multisource al mart
"},{"location":"adr/2023-06-13-pipeline_dades_llargues/#decision-outcome","title":"Decision Outcome","text":"

Chosen option: opci\u00f3 3

"},{"location":"adr/2023-06-13-pipeline_dades_llargues/#negative-consequences","title":"Negative Consequences","text":""},{"location":"adr/2023-06-13-pipeline_dades_llargues/#pros-and-cons-of-the-options","title":"Pros and Cons of the Options","text":""},{"location":"adr/2023-06-13-pipeline_dades_llargues/#option-1","title":"[option 1]","text":"

[example | description | pointer to more information | \u2026]

"},{"location":"adr/2023-06-13-pipeline_dades_llargues/#option-2","title":"[option 2]","text":"

[example | description | pointer to more information | \u2026]

"},{"location":"adr/2023-06-13-pipeline_dades_llargues/#option-3","title":"[option 3]","text":"

[example | description | pointer to more information | \u2026]

"},{"location":"adr/2023-06-13-pipeline_dades_llargues/#links","title":"Links","text":""},{"location":"adr/2023-09-12-redash_vs_superset/","title":"Pipeline de dades revisited","text":""},{"location":"adr/2023-09-12-redash_vs_superset/#context-and-problem-statement","title":"Context and Problem Statement","text":"

En el seu moment vam triar redash perqu\u00e8 podia tenir apis com a sources i el dev \u00e9s molt m\u00e9s c\u00f2mode perqu\u00e8 no est\u00e0 tant layeritzat (dataset/query - chart - dashboard).

A la pr\u00e0ctica no hem fet servir mai la lectura directa de les apis perqu\u00e8 les aut\u00e8ntificacions no estaven suportades, ens agradava tenir una c\u00f2pia, t'obligava a treballar en queries de l'sqlite de redash.

A m\u00e9s a m\u00e9s, redash delega la gesti\u00f3 d'usuaris a la db, cosa que implica que no pots limitar que un usuari pugui veure uns valors d'una query i no uns altres, per exemple filtrar per plantes la producci\u00f3, els permisos es fan a nivell de db.

La macrofase de plantmonitor i l'evoluci\u00f3 de l'equip de dades cap a superset amb els altres equips ens porta a replantejar-nos-ho.

Iniciem un spike de superset per a fer els nous dashboards.

"},{"location":"adr/2023-09-12-redash_vs_superset/#decision-drivers","title":"Decision Drivers","text":""},{"location":"adr/2023-09-12-redash_vs_superset/#considered-options","title":"Considered Options","text":"
  1. superset
  2. redash
  3. metabase -- en Diego comenta que en la versi\u00f3 free no tens la capa sem\u00e0ntica
"},{"location":"adr/2023-09-12-redash_vs_superset/#decision-outcome","title":"Decision Outcome","text":"

Chosen option: ?

"},{"location":"adr/2023-09-12-redash_vs_superset/#positive-consequences","title":"Positive Consequences","text":""},{"location":"adr/2023-09-12-redash_vs_superset/#negative-consequences","title":"Negative Consequences","text":""},{"location":"adr/2023-09-12-redash_vs_superset/#pros-and-cons-of-the-options","title":"Pros and Cons of the Options","text":""},{"location":"adr/2023-09-12-redash_vs_superset/#redash","title":"redash","text":"

\ud83d\udc4d

mustache

definir parametres a nivell de default de charts, de chart especific o dashboard

\ud83d\udc4e

no apte per heavy duty (cache regular)

"},{"location":"adr/2023-09-12-redash_vs_superset/#superset","title":"superset","text":"

\ud83d\udc4d

cohesi\u00f3 cooperativa -- tota la cooperativa fa servir superset

\ud83d\udc4e

No pots fixar filtres a nivell de chart al dashboard, cal fer-ne una c\u00f2pia. Implica que si volem canviar com s'ensenya un chart de planta, els haurem de canviar tots un a un.

El jinja \u00e9s funky. No \u00e9s clar com fer servir el \"filter_values\" a dins d'una query. Les queries din\u00e0miques s\u00f3n molt menys directes de fer que al redash.

"},{"location":"adr/template/","title":"Template","text":"

^-- [short title of solved problem and solution]

Technical Story: [description | ticket/issue URL]

"},{"location":"adr/template/#context-and-problem-statement","title":"Context and Problem Statement","text":"

[Describe the context and problem statement, e.g., in free form using two to three sentences. You may want to articulate the problem in form of a question.]

"},{"location":"adr/template/#decision-drivers","title":"Decision Drivers","text":""},{"location":"adr/template/#considered-options","title":"Considered Options","text":""},{"location":"adr/template/#decision-outcome","title":"Decision Outcome","text":"

Chosen option: \"[option 1]\", because [justification. e.g., only option, which meets k.o. criterion decision driver | which resolves force force | \u2026 | comes out best (see below)].

"},{"location":"adr/template/#positive-consequences","title":"Positive Consequences","text":""},{"location":"adr/template/#negative-consequences","title":"Negative Consequences","text":""},{"location":"adr/template/#pros-and-cons-of-the-options","title":"Pros and Cons of the Options","text":""},{"location":"adr/template/#option-1","title":"[option 1]","text":"

[example | description | pointer to more information | \u2026]

"},{"location":"adr/template/#option-2","title":"[option 2]","text":"

[example | description | pointer to more information | \u2026]

"},{"location":"adr/template/#option-3","title":"[option 3]","text":"

[example | description | pointer to more information | \u2026]

"},{"location":"adr/template/#links","title":"Links","text":""},{"location":"deprecated/TODOs/","title":"Views de plantmonitor a migrar a dbt","text":"

~ marca les que segurament no cal fer, al gust

view_availability.sql\n~view_q_hourly_debug.sql~\nview_q_monthly.sql\n~view_cos_phi_hourly_debug.sql~\nview_cos_phi_monthly.sql\nview_expected_power.sql\nview_hd_daily.sql\nview_ht_daily.sql\nview_ht_monthly.sql\nview_pr_hourly.sql\nview_target_energy.sql\n
"},{"location":"deprecated/TODOs/#agregacions","title":"Agregacions","text":"

No cal fer-les perqu\u00e8 son agraupacions que segurament farem servir m\u00e8triques de dbt tipus:

view_inverter_energy_daily.sql\nview_inverter_energy_monthly.sql\nview_meter_export_energy_daily.sql\nview_meter_export_energy_monthly.sql\nview_average_yearly_production_plant.sql\nview_satellite_irradiation.sql\n

A m\u00e9s a m\u00e9s les m\u00e8triques de dbt resolen el problema seg\u00fcent:

sum(a/x) -- horaria

sum(a/x) + sum(b/y) ... mensual wrong

sum((a+b+c)/(x+y+z)) -- mensual correct

"},{"location":"deprecated/TODOs/#views-per-limitar-lacces-a-la-info-dels-tecnics","title":"views per limitar l'acc\u00e9s a la info dels t\u00e8cnics","text":"

No cal fer-ho, limitarem l'acc\u00e9s a la wide table o farem servir els permisos del superset o ja veurem

view_plants_energes.sql\nview_plants_energetica.sql\nview_plants_ercam.sql\nview_plants_exiom.sql\nview_meterregistry_energetica.sql\nview_inverter_energy_daily_exiom.sql\nview_inverter_intensity_energes.sql\nview_inverter_intensity_ercam.sql\nview_inverter_intensity_exiom.sql\nview_inverter_power_energes.sql\nview_inverter_power_ercam.sql\nview_inverter_power_exiom.sql\nview_inverterregistry_energes.sql\nview_inverterregistry_ercam.sql\nview_inverterregistry_exiom.sql\nview_inverter_temperature_exiom.sql\nview_sensorirradiationregistry_energes.sql\nview_sensorirradiationregistry_ercam.sql\n
"},{"location":"deprecated/TODOs/#no-shan-de-fer-obsoletes","title":"no s'han de fer (obsoletes)","text":"
zero_inverter_power_at_daylight.sql\nzero_sonda_irradiation_at_daylight.sql\n
"},{"location":"desenvolupadors/2023-02-08-novu_notifications/","title":"Novu helpers and quickstart","text":""},{"location":"desenvolupadors/2023-02-08-novu_notifications/#swagger-api","title":"Swagger API","text":"

Novu api

"},{"location":"desenvolupadors/2023-02-08-novu_notifications/#overview","title":"overview","text":"

Features in novu get to the api waaaay before the get to the UI, so keep it at hand since most of the changes (adding removing subscribers, topics, etc.) will be done via python, react or postman via api rest calls.

"},{"location":"desenvolupadors/2023-02-08-novu_notifications/#helpers","title":"helpers","text":"

We have a handful of scripts to interact with novu via airflow or command-line scripts/notify_alert.py and scripts/novu_interface.py.

The notify_alert is ran by airflow to periodically check the alarm tables and trigger a notification on change.

novu_interface is more of a command-line util to do novu stuff manually, like adding subscribers or topics.

For example, to add a topic of a plant we'd do

$ python -m scripts.novu_interface --help\n\nUsage: python -m scripts.novu_interface [OPTIONS] URL API_KEY TOPIC_KEY\n                                         TOPIC_NAME\n\nUsage: python -m scripts.novu_interface add-topic [OPTIONS] URL API_KEY\n                                                   TOPIC_KEY TOPIC_NAME\n\n\u256d\u2500 Arguments \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502 *    url             TEXT  [default: None] [required]                  \u2502\n\u2502 *    api_key         TEXT  [default: None] [required]                  \u2502\n\u2502 *    topic_key       TEXT  [default: None] [required]                  \u2502\n\u2502 *    topic_name      TEXT  [default: None] [required]                  \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n\u256d\u2500 Options \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502 --help          Show this message and exit.                            \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n

$ python -m scripts.novu_interface add-topic {novu_base_url} {api_key} {topic_key} {topic_name}\n$ python -m scripts.novu_interface add-topic http://moll3.somenergia.lan:3000/v1 {api_key} asomada-topic-1 \"asomada general topic\"\n
and to get the topics

$ python -m scripts.novu_interface get-topic {novu_base_url} {api_key} {topic_key} {topic_name}\n\n2023-01-27 16:39:29,080 [INFO] {\"page\":0,\"totalCount\":1,\"pageSize\":10,\"data\":[\n    {\"_id\":\"63d3ef030d600f1df1a3829e\",\"_environmentId\":\"6349479f3043d373c3d5fc07\",\"_organizationId\":\"6349479f3043d373c3d5fc02\",\"key\":\"asomada-topic-1\",\"name\":\"asomada general topic\",\"subscribers\":[]}\n]}\n

Let's add some subcribers to it:

$ python -m scripts.novu_interface add-subscriber http://moll3.somenergia.lan:3000/v1 {api_key} pol_recipient\n$ python -m scripts.novu_interface add-topic-subscriber http://moll3.somenergia.lan:3000/v1 {api_key} asomada-topic-1 pol_recipient\n
"},{"location":"desenvolupadors/2023-06-05-download_solargis_manually/","title":"Manually download a solargis reading","text":"

To download a specific day you just need to call

python -m scripts.solargis <dbapi> <apikey> 2023-05-13 2023-05-13 1 2 3 4\n 5 6 7 9 13 14 15 16 17 22 40\n

See python -m scripts.solargis --help for info on each parameter (essentially time range and plant id list) and the wiki or the pyxis server for credentials.

Note that to download one day you have to put the same from and to date. The from is start day and to date is end day. The script DOES NOT check for duplicates. You'll have to clean up yourself using the query time if you made a mistake.

"},{"location":"desenvolupadors/2023-06-10-overview-devs/","title":"Overview for devs","text":""},{"location":"desenvolupadors/2023-06-10-overview-devs/#quickstart","title":"quickstart","text":"

To fully deploy the dbt models to the production schema you can do

dbt build -s tag:jardiner,tag:legacy --store-failures --full-refresh --target prod\n

You will need to have the prod profile in your ~/.dbt/profiles.yml file.

"},{"location":"desenvolupadors/2023-06-10-overview-devs/#current-state-of-the-project","title":"Current State of the project","text":""},{"location":"desenvolupadors/2023-06-10-overview-devs/#pieces","title":"Pieces","text":""},{"location":"desenvolupadors/2023-06-10-overview-devs/#repos","title":"repos","text":"

Plantmonitor

Fa d'ORM de les plantes, cont\u00e9 la api i \u00e9s codi de les raspberrypis. A m\u00e9s a m\u00e9s tamb\u00e9 t\u00e9 el codi d'ingesta d'apis de tercers.

Warning

La meitat del plantmonitor acabar\u00e0 deprecated pel prove\u00efdor de dades

Plant Reader

Lectura remota de les plantes. Actualment nom\u00e9s Asomada es llegeix remotament i es guarda a plant_lake.

Acabar\u00e0 deprecated pel prove\u00efdor de dades.

Actualment tamb\u00e9 allotja la lectura de la API del prove\u00efdor de dades.

Jardiner

dbt de les plantes. Actualment tot views.

"},{"location":"desenvolupadors/2023-06-10-overview-devs/#data-sources","title":"Data sources","text":""},{"location":"desenvolupadors/2023-06-10-overview-devs/#visualization","title":"Visualization","text":""},{"location":"desenvolupadors/2023-06-10-overview-devs/#project-tructure","title":"Project tructure","text":""},{"location":"desenvolupadors/2023-06-10-overview-devs/#actual","title":"Actual","text":"
flowchart LR\n\ndw[db/plants]\n\nplants -- pull 5' \\n plant_reader/dags --> lake[db/plant_lake]\nlake -- push 5' \\n airbyte --> dw\n\nplants -- pull 5' \\n plantmonitor/main.py --> rPIs -- push 5'  --> dw\nplants -- pull 2h/12h \\n meter \\n ERP's import_tm_data_click.py --> ERP -- pull 20'\u00ba\\n meter data --> dw\nSAT -- pull daily\u00ba\\n irr/expected energy --> dw\nMETEO <-- pull daily\u00ba\\n meter data +  irr/kWh forecast --> dw\nplants -- pull 15'\\n plant_reader/dags --> PLANT -- pull 15'\\n devices data --> db/plant_lake\n\ndw -- dbt views --> prod --> alarms\nprod --> alerts\nprod --> datasets

\u00ba: plantmonitor does it. Update rate defined at conf/startup_configuration.py and each task is run as a function.

"},{"location":"desenvolupadors/2023-06-10-overview-devs/#planned-scheme","title":"Planned scheme","text":"
flowchart LR\n\ndw[db/plants]\n\nplants -- pull 2h/12h \\n meter \\n import_tm_data_click.py --> ERP -- pull 20' \u00ba\\n meter data --> dw\nSAT -- pull daily\\n irr/expected energy --> dw\nMETEO <-- pull daily\\n meter data +  irr/kWh forecast --> dw\nplants -- pull 15' --> PLANT -- pull 15'\\n devices data --> dw\n\n\ndw -- dbt fast --> fast[dbt_prod/fast_] --> union\ndw -- dbt --> obt --> alarms\nobt --> union\nunion -- notify.py --> alerts\nobt --> datasets

Despr\u00e9s de l'ADR d'estructura de pipeline

See Roadmap

"},{"location":"desenvolupadors/2023-07-10-dbt_cmds_utils/","title":"dbt commands","text":"
  1. Generar i servir la documentaci\u00f3 per a poder veure el directed graph.
$ dbt docs generate --project-dir dbt_jardiner --target pre\n\n$ dbt docs serve --project-dir dbt_jardiner --target pre --port 8010\n
  1. Pujar tots els CSVs de seeds
dbt seed --project-dir dbt_jardiner\n
  1. Podem passar variables per command line (en comptes de definir-les a dbt_project.yml i fer-les accesibles per diferents resources):

dbt run --vars '{\"variable\":\"value\"}'\n
Per fer testing amb un csv determinat:

dbt run --project-dir project_x --target testing --vars '{\"test_sample\":\"nom_del_csv\"}' -m model_x\n
"},{"location":"desenvolupadors/2023-07-13-codi_reciclable/","title":"Reusuable sqls","text":""},{"location":"desenvolupadors/2023-07-13-codi_reciclable/#dbt-sql-for-rolling-window","title":"dbt sql for rolling window","text":"
{%- set plant_alarm_ranges = dbt_utils.get_query_results_as_dict(ref('plant_alarm_ranges')) -%}\n\n{% for plant in plant_alarm_ranges %}\n\n  {% if var('plants_code') == 'SomEnergia_Riudarenes_ZE' %}\n      {%- set time_to_sum = '1 hour' -%}\n  {% elif var('plants_code') == 'SomEnergia_Matallana' %}\n      {%- set time_to_sum = '4 hour' -%}\n  {% else %}\n          {{ ref('accounts') }}   accounts\n  {% endif %}\n\n  SELECT\n      *,\n      SUM(export_energy_wh) OVER (\n          PARTITION BY plant_name\n          ORDER BY \"time\"\n          RANGE BETWEEN interval {{time_to_sum}} PRECEDING AND CURRENT ROW\n      ) as export_energy_wh_sum\n  FROM (\n    select * from {{ref('meter_registry_cleaned')}} where codename = {{plants_code}}\n  ) as meter_registry_cleaned_current_plant\n  UNION ALL\n\n{% endfor %}\n
"},{"location":"desenvolupadors/2023-08-28-alta_planta/","title":"Context","text":"

Tenim un pipe de transformaci\u00f3 de dades que acaba en una serie de dashboards de superset que mostra dades de plantes des de el pipe de transformaci\u00f3 de dades arribant del nostre prove\u00efdor de dades.

"},{"location":"desenvolupadors/2023-08-28-alta_planta/#protocol-dalta-duna-planta-fotovoltaica","title":"Protocol d'alta d'una planta fotovoltaica","text":"

Per afegir una nova planta a la visualitzaci\u00f3, cal seguir un protocol documentat al Formulari d\u2019inscripci\u00f3 de noves plantes a Superset

El equip de gestio d'actius ha d'omplir aquest formulari amb les dades de la planta i ens ho han de comunicar per a que puguem seguir amb el proc\u00e9s nosaltres.

Obres en curs amb jardiner-admin

Aquesta tasca est\u00e0 en proc\u00e9s de ser portada al jardiner-admin. Per tant, aquest protocol pot canviar en el futur.

Necessites permisos d'administrador

Aquesta tasca \u00e9s per a desenvolupadors que treballen amb el pipe de transformaci\u00f3 de dades. Si no ets un desenvolupador, no cal que facis aquesta tasca.

Aqui sota ens dediquem a documentar t\u00e9cnicament els passos a seguir des de la perspectiva de desenvolupament.

"},{"location":"desenvolupadors/2023-08-28-alta_planta/#checklist-general-per-it","title":"Checklist general per IT","text":""},{"location":"desenvolupadors/2023-08-28-alta_planta/#descripcio-de-dades","title":"Descripci\u00f3 de Dades","text":""},{"location":"desenvolupadors/2023-08-28-alta_planta/#fitxer-mapeig-de-senyals","title":"Mapeig de senyals de Planta","text":"

Per cada planta, GdA genera fitxers amb informaci\u00f3 de la planta per la seva gesti\u00f3. Aquests fitxers contenen informaci\u00f3 de senyals, aparells, plantes, etc. per\u00f2 sense el context del pipe de Jardiner. Com IT, portem aquesta informaci\u00f3 al fitxer de mapeig de senyals per a que el pipe de Jardiner pugui fer integrar les dades de la planta.

Un exemple d'aquest fitxer pot ser aquest.

"},{"location":"desenvolupadors/2023-08-28-alta_planta/#fitxer-dades-fixes-de-planta","title":"Dades fixes de planta","text":"

Aquest fitxer cont\u00e9 dades fixes de la planta, com ara la pot\u00e8ncia de la planta, la seva ubicaci\u00f3, etc. Es trova a google drive i es transforma amb airbyte aqui

"},{"location":"desenvolupadors/2023-08-28-alta_planta/#fitxer-info-api-solargis","title":"Info API SolarGIS","text":"

GdA ha de crear una entrada per la planta nova al fitxer excel per aix\u00f2

"},{"location":"desenvolupadors/2023-08-28-alta_planta/#fitxer-rendiment-de-planta","title":"Rendiment de planta","text":"

T\u00e9 un nom tipus \"C\u00e0lcul Rendiment de Planta <nom planta>\". El podeu buscar al cercador de drive.

"},{"location":"desenvolupadors/2023-08-28-alta_planta/#fitxer-produccio-actual-de-planta","title":"Objectius de producci\u00f3 de planta","text":"

Es troba al fitxer de google sheets aqui i es transforma amb airbyte aqui.

"},{"location":"desenvolupadors/2023-08-28-alta_planta/#fitxer-unificacio-noms-projectes-generacio","title":"Unificaci\u00f3 Noms Projectes Generaci\u00f3","text":"

Cont\u00e9 els llistat de noms unificats en un sol fitxer. Ves a Unificaci\u00f3 Noms Projectes Generaci\u00f3.

"},{"location":"desenvolupadors/2023-08-28-alta_planta/#fitxer-modelos-contadores-inversores-scadas","title":"Modelos contadores_inversores_SCADAS","text":"

Modelos contadores_inversores_SCADAS

"},{"location":"desenvolupadors/2023-08-28-alta_planta/#descripcio-de-tasques","title":"Descripci\u00f3 de tasques","text":""},{"location":"desenvolupadors/2023-08-28-alta_planta/#afegir-mapeig-airbyte","title":"Afegir dades al mapeig d'airbyte","text":"

Tenim un document on consolidem relacions entre UUIDs de senyals, aparells i plantes que inserim amb airbyte

Aquest mapeig es una traducci\u00f3 del fitxer de mapeigs.

Un cop heu acabat de fer el mapeig, cal engegar la tasca de sync de l'airbyte d'uuids per que les dades arribin a la base de dades. Heu de tenir especial cura d'omplir les columnes updated_at i aquelles que posen UUIDs, ja que amb elles se creuen les dades dins del pipe.

update_at s'actualitza automagicament

Al fitxer de google sheets hi ha un script que actualitza la columna updated_at quan hi ha hagut un canvi a nivell de fila. Es molt limitat i heu de supervisar que s'ha actualitzat correctament, ja que de vegades google sheets no ho fa. Podeu mirar a Extensions > Apps Script per veure el codi.

UUID s\u00f3n versi\u00f3 4

Els senyals porten UUID4 (han de ser versio 4) \u00fanics al mapeig de la planta indicada. Es poden generar UUID4 amb https://www.uuidgenerator.net/ O ben v\u00e9 amb un script de python que podeu afegir a ~/.bash_aliases:

#\u00b7function\u00b7to\u00b7genenerate\u00b7random\u00b7uuid4\u00b7using\u00b7python$\nfunction pyuuid4() {\npython -c \"import uuid$\nfor _ in range(${N:-1}):\n    print(uuid.uuid4())\" ; }\n

i el podeu executar amb N=<n> pyuuid4 per a generar n UUID4.

$ N=10 pyuuid4\nbf98707e-688c-471c-8700-9ffeaf436531\n8f6bfcc0-48bf-4b7c-895f-0b2efa37dd63\nee5dcfd1-80ba-48fb-800d-5bc8c04d217f\nd32b0a23-c479-4ae9-bb54-43390fedf8e4\neced945e-7a91-4451-bbd5-974a82c9c0d4\nfbacbf3d-23f6-4361-b5ec-b48b390d9c09\nb9c4ea66-672e-4dca-ad08-fddacdfb3627\nae3e242e-9a57-4359-a72b-313a72b48348\n9e11a595-c32c-42e1-8549-a5161c0ea624\n172061ed-96b4-4aa3-b5bd-0c66e850383b\n
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#afegir-meteologica-plantmonitor","title":"Afegir planta per meteologica amb plantmonitor","text":"

Qu\u00e8 \u00e9s meteologica?

meteologica \u00e9s un servei que ens permet tenir dades meteorol\u00f2giques de les plantes fotovoltaiques. Aquest servei es comunica amb plantmonitor per a obtenir dades de les plantes.

Qu\u00e8 \u00e9s plantmonitor?

Tots els scripts en aquest apartat fan refer\u00e8ncia al repositori de github plantmonitor. \u00c9s un projecte previ a jardiner que ens permet monitoritzar plantes fotovoltaiques quan l'obtenci\u00f3 de dades la gestionem nosaltres.

A plantmonitor tenim un script que ens permet afegir plantes noves. Aquest script s'ha de cridar amb un fitxer yaml que contingui les dades de la planta.

S'ha de crear la planta a la base de dades tant a la raspberrypi com a plantmonitor (no calen els dispositius, que els crea autom\u00e0ticament). Haureu d'afegir plantlocation, plantparameters i moduleparameters. Trobareu un exemple a plantmonitor

Un cop teniu el fitxer yaml, per exemple data/plant-nova.yaml, podeu cridar l'script addPlant.py amb el seg\u00fcent comandament:

PLANTMONITOR_MODULE_SETTINGS='conf.settings.prod' python addPlant.py data/plant-asomada.yaml\n

Haureu de tenir un fitxer .env.prod amb les variables d'entorn necess\u00e0ries per a que l'script funcioni correctament. Mireu a la documentaci\u00f3 de plantmonitor per a m\u00e9s informaci\u00f3.

De moment manualment cal afegir els registres de plantestimatedmonthlyenergy. L'any \u00e9s irrellevant. Si tenim hist\u00f2rics d'energia objectiu, els podeu afegir.

INSERT INTO\n public.plantestimatedmonthlyenergy(plantparameters, \"time\", monthly_target_energy_kwh)\nVALUES\n (33, '2022-01-01 00:00:00+01', 473630),\n (33, '2022-02-01 00:00:00+01', 453370),\n (33, '2022-03-01 00:00:00+01', 592040),\n (33, '2022-04-01 00:00:00+02', 617000),\n (33, '2022-05-01 00:00:00+02', 667210),\n (33, '2022-06-01 00:00:00+02', 648940),\n (33, '2022-07-01 00:00:00+02', 703030),\n (33, '2022-08-01 00:00:00+02', 644500),\n (33, '2022-09-01 00:00:00+02', 574580),\n (33, '2022-10-01 00:00:00+02', 524110),\n (33, '2022-11-01 00:00:00+01', 446490),\n (33, '2022-12-01 00:00:00+01', 442060);\n
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#afegir-solargis","title":"Afegir planta al script de SolarGIS","text":"

S'ha de modificar el codi de plantmonitor per incloure la nova planta. Podeu trobar un exemple de modificacions github

Amb la MR aprovada i fusionada, s'ha de fer git pull des de el servidor de producci\u00f3 i reiniciar el servei de plantmonitor.

La API de SolarGIS requereix un token d'acc\u00e9s i es crida remotament amb supervisord, amb codi al repositori de plantmonitor.

L'API de SolarGIS \u00e9s SOAP

L'API es del tipus SOAP i no REST. \u00c9s a dir, demana un XML amb les dades de la planta.

"},{"location":"desenvolupadors/2023-08-28-alta_planta/#dades-requerides-per-aparells","title":"Dades requerides per aparells","text":""},{"location":"desenvolupadors/2023-08-28-alta_planta/#afegir-comptadors-a-lerp","title":"Afegir Comptadors a l'ERP","text":"

Les lectures a l'ERP s\u00f3n necessaries mentre metelogica faci servir les lectures de comptadors que plantmonitor copia de l'ERP. Tamb\u00e9 s\u00f3n necess\u00e0ries per al generationkwh. Per aix\u00f2 cal agafar les dades de Modelos contadores_inversores_SCADAS i afegir-les a l'erp Infrastructura -> Registradors -> tots els registradors.

"},{"location":"desenvolupadors/2023-08-28-alta_planta/#serveis-de-somenergia-addicionals","title":"Serveis de SomEnergia addicionals","text":""},{"location":"desenvolupadors/2023-08-28-alta_planta/#nagios","title":"nagios","text":"

Preguntar a Gesti\u00f3 d'Actius si encara fan servir el nagios. Si si, afegir-la al nagios o modificar-la en cas de canvi d\u2019ip p\u00fablica

"},{"location":"desenvolupadors/2023-08-28-alta_planta/#opendata","title":"opendata","text":"

Cal afegir la info de la nova planta a taula literal que es fa servir a somenergia-opendata/som_opendata/queries/plantpower.sql. Un cop aquesta info estigui a plantmonitor, l\u2019opendata podr\u00e0 agafar-la directament per sql i no caldr\u00e0 fer aquest pas.

"},{"location":"desenvolupadors/2023-08-28-alta_planta/#generationkwh","title":"generationkwh","text":"

Si la planta est\u00e0 inclosa al generation, \u00e9s important afegir-la a l\u2019erp abans de la mitja nit del dia de posada a producci\u00f3. Si no, cal descartar c\u00e0lculs de drets del generation per a que incloguin els resultats. Si, a m\u00e9s, els drets s\u2019han comen\u00e7at a gastar (15 dies de marge de facturaci\u00f3), cal reperfilar els nous drets amb la producci\u00f3 afegida perqu\u00e8 no superi els drets ja atorgats.

La nova planta es crea fent servir l\u2019script a somenergia-generationkwh/scripts/genkwh_plants.py

Un exemple complert de com afegir una planta amb aquest script el podeu trobar a https://github.com/Som-Energia/somenergia-generationkwh/blob/master/scripts/genkwh_migrate_plantmeter_1_7_0_newplant.sh

Es recomana provar primer a un ERP de testing o local, i, despr\u00e9s aplicar-ho a producci\u00f3.

"},{"location":"desenvolupadors/2023-08-28-alta_planta/#reperfilacio-de-dades","title":"Reperfilaci\u00f3 de dades","text":"

Un exemple d\u2019script de migraci\u00f3 en la que vam haver de reperfilar: https://github.com/Som-Energia/somenergia-generationkwh/blob/master/scripts/genkwh_migrate_plantmeter_1_7_2_fontivsolarfix.sh

"},{"location":"desenvolupadors/2024-01-11-incremental-dedup/","title":"Incremental Dedup","text":"

Exemple de com fer un incremental amb deduplicaci\u00f3 en dbt.

{{\n  config(\n    materialized = 'incremental',\n    unique_key = ['signal_uuid','ts'],\n    incremental_strategy = 'merge',\n    incremental_predicates = [\n      \"DBT_INTERNAL_DEST.queried_at > dateadd(day, -1, current_date)\"\n    ]\n  )\n}}\n\nwhere\n    ts < now() - interval '1 hour'  {#- select only freshly ingested rows #}\n\n    {% if is_incremental() -%}\n        and queried_at > coalesce((select max(queried_at) from {{ this }}), '1900-01-01') and queried_at > now() - interval '2 hour'\n    {%- endif %}\n
source senyal ts valor queried_at energia_comptador 03:00 80 +1d 04:00 energia_comptador 03:00 null 04:00 potencia_inversor 03:00 50 +1d 04:00 potencia_inversor 03:00 50 04:00

Son ara les 15:04. en la materialitzada tindrem

source senyal ts valor potencia_inversor 14:00 67 energia_comptador 14:00 null

o bien, la \u00faltima fila no hi ser\u00e0:

source senyal ts valor potencia_inversor 14:00 67"},{"location":"desenvolupadors/2024-01-11-incremental-dedup/#explicacio","title":"Explicaci\u00f3","text":"

where max(ts) = 14:00:

source senyal ts valor potencia_inversor 14:00 67 energia_comptador 14:00 null

where max(ts) - 24h:

source senyal ts valor queried_at energia_comptador 03:00 80 +1d 04:00 potencia_inversor 03:00 50 +1d 04:00 potencia_inversor 03:00 50 04:00 potencia_inversor 14:00 67 energia_comptador 14:00 null

unique (signal_uuid, ts):

source senyal ts valor queried_at energia_comptador 03:00 80 +1d 04:00"},{"location":"desenvolupadors/2024-01-11-incremental-dedup/#prova-practica","title":"prova pr\u00e0ctica","text":"

Abans d'executar l'incremental tenim aquest tres \u00faltims registres a materialized one hour late

group_name queried_at ts signal_code signal_device_type signal_device_uuid signal_frequency signal_id signal_is_virtual signal_last_ts signal_last_value signal_type signal_tz signal_unit signal_uuid signal_uuid_raw signal_value materialized_at SE_vallehermoso 2024-01-09 13:24:42.488123+01 2024-01-05 23:15:00+01 ce_eactexp meter 08bc8d88-4ea2-4ee9-b817-00e1f07debba 15 minutes 983894 false 2024-01-08 23:45:00+01 0 absolute Europe/Madrid kWh 646d1ec2-0ca2-4c22-9739-8161aafb224e 646d1ec2-0ca2-4c22-9739-8161aafb224e 0.0 2024-01-09 13:36:52.925381+01 SE_asomada 2024-01-09 13:24:42.488123+01 2024-01-05 23:00:00+01 s8102 meter c5ed9fab-e73c-40a5-acb3-113e22052fd6 5 minutes 1031060 false 2024-01-09 11:47:00+01 60.2 absolute Europe/Madrid V dd065f6a-4ded-41a8-98d3-ba239f9a2a47 dd065f6a-4ded-41a8-98d3-ba239f9a2a47 60.2 2024-01-09 13:36:52.925381+01 SE_tahal 2024-01-09 13:24:42.488123+01 2024-01-05 23:45:00+01 ce_ercapexp_er3 meter 5deb3780-02e2-4dcf-a6a7-de646991762c 15 minutes 983890 false 2024-01-08 23:45:00+01 0 absolute Europe/Madrid kVArh 31ae7d09-95db-447d-b5bf-358b056ef5bf 31ae7d09-95db-447d-b5bf-358b056ef5bf 0.0 2024-01-09 13:36:52.925381+01"},{"location":"desenvolupadors/2024-02-16-row-level-security/","title":"Row Level Security","text":""},{"location":"desenvolupadors/2024-02-16-row-level-security/#context","title":"Context","text":"

Tenim usuaris externs que necessiten mirar dades de plantes, per\u00f2 nom\u00e9s de les plantes que gestionen. Aix\u00f2 ho podem fer amb Row Level Security a Superset.

El proc\u00e9s \u00e9s el seg\u00fcent:

  1. Crear un rol Manteniment_<nom empresa> si no existeix, pel cas de noves plantes.
  2. Afegir una regla Row Level Security Pareu atenci\u00f3 al camp \"clause\", s'afegir\u00e0 com a condici\u00f3 where exclusiva (and) als datasets que seleccionem.
  3. Donar acc\u00e9s als charts d'un dashboard
  4. Afegir la o les usu\u00e0ries i assignar-los els rols [Gamma] i [Manteniment_<nom empresa>]

I ja ho tens. Ho he fet amb energes amb la clause nom_planta in (noms plantes)

"},{"location":"desenvolupadors/2024-02-16-row-level-security/#decisions","title":"Decisions","text":"

Hem decidit a 02/2024 que no val la pena propagar l'empresa de manteniment en el pipe per a cobrir aquest use case, ja que no cal afegir 5m aquestes metadadades per a un tema de visualitzaci\u00f3 i permisos. Ho gestionem en la capa de visualitzaci\u00f3.

Canvis d'empresa de manteniment

Canvis d'empresa de manteniment s'hauran de fer al Superset (i a les metadades de la planta, per coher\u00e8ncia).

"},{"location":"desenvolupadors/2024-02-16-row-level-security/#referencies","title":"Refer\u00e8ncies","text":""},{"location":"projecte/2023-06-03-macrofase%20roadmap/","title":"Macrofase Jardiner 2023","text":"

Document de Roadmap

"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#roadmap-overview","title":"Roadmap overview","text":""},{"location":"projecte/2023-06-03-macrofase%20roadmap/#flux-de-dades","title":"Flux de dades","text":"
flowchart LR\n\nrPIs -- push 5'  --> dw[db/plants]\nERP -- pull 20' \u00ba\\n meter data --> dw\nSAT -- pull daily\\n irr/expected energy --> dw\nMETEO <-- pull daily\\n meter data +  irr/kWh forecast --> dw\nPLANT -- pull 15'\\n devices data --> dw\n\ndw -- dbt views --> prod --> alarms\nprod --> alerts\nprod --> datasets
"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#fita-1","title":"Fita 1","text":"

Prioritat 1

Aprox 1-2 fases

Definition of Done (DoD): veure per tema

Cost en mitjos dies de l'equip sencer entre par\u00e8ntesis (N)

"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#dashboard-tecnics-de-plantes-critiques","title":"Dashboard t\u00e8cnics de plantes cr\u00edtiques","text":"

Pulir el dashboard de t\u00e8cnics per a Asomada i Llanillos.

DoD: TODO

"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#obtencio-de-dades-de-tots-els-equips-i-totes-les-plantes","title":"Obtenci\u00f3 de dades de tots els equips i totes les plantes","text":"

DoD:

"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#introduccio-al-jardiner","title":"Introducci\u00f3 al jardiner","text":"

Estat del codi i rep\u00e0s de l'estructura. Documentaci\u00f3 per devs.

"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#dt-obt-i-fast-lane","title":"[DT] OBT i fast-lane","text":"

Crear la obt cincminutal incremental i/o el fast-lane cincminutal i la obt di\u00e0ria.

Crear el live per late arriving facts de comptadors i apis.

En funcionament normal s\u00f3n delays de m\u00e0xim un dia. Amb incid\u00e8ncia poden ser 1-2 mesos m\u00e0xim. I en casos excepcionals, molt m\u00e9s (candidat a full-refresh manual o alguna altra cosa)

"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#aterracio-de-tot-plegat","title":"Aterraci\u00f3 de tot plegat","text":""},{"location":"projecte/2023-06-03-macrofase%20roadmap/#fita-2-dashboard-overview-plantes","title":"Fita 2: Dashboard Overview plantes","text":"

Prioritat 4

Disseny d'un Dashboard amb un llistat de totes les plantes amb els principals indicadors (mesures en temps real i acumulades i resum alarmes). Pendent fer una proposta per part de GA. Des d'aquest dashboard s'ha de poder accedir al Dashboard de cada planta per obtenir informaci\u00f3 detallada. Esborrany inicial falta versi\u00f3 actualitzada. (6)

Implementaci\u00f3 (9)

"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#fita-3-dashboard-planta","title":"Fita 3: Dashboard planta","text":"

Prioritat 5

Definir versi\u00f3 revisada del Dashboard de planta per a totes les plantes: Unifilar din\u00e0mic, dades fixes i reempla\u00e7ar alarmes, millorar la velociatat i treure el dataset de sota, afegir solargis + coses que puguin sortir despr\u00e9s (3)

Implementar versi\u00f3 revisada del Dashboard de planta per a totes les plantes: Unifilar din\u00e0mic, dades fixes i reempla\u00e7ar alarmes, millorar la velocitat i treure el dataset de sota, afegir solargis + coses que puguin sortir despr\u00e9s (3)

Dashboard Ringsted per plantes -- particularitats (4)

Dashboard Gasdtrup -- particularitats (4)

Dashboard de planta t\u00edpica (Alcolea, etc) (0)

"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#fita-4-dashboard-tecnics","title":"Fita 4: Dashboard t\u00e8cnics","text":"

Prioritat 6

Definir la versi\u00f3 final amb grafiques de Solar Gis, Alarmes i pot ser alarmes. Pendent fer una revisi\u00f3 de les funcionalitats per part de GA. Incloure un dashboard per cadascuna de les empreses de manteniment. Al dashboard unificat d'alarmes estat actual afegir des de quan est\u00e0 activada l'alarma

Implementar la versi\u00f3 final amb grafiques de Solar Gis, Alarmes i pot ser alarmes. Pendent fer una revisi\u00f3 de les funcionalitats per part de GA. Incloure un dashboard per cadascuna de les empreses de manteniment

Fer el Dashboard per a 8 empreses instal\u00b7ladores. Haur\u00edem de passar a Superset que t\u00e9 una millor gesti\u00f3 d'usuaris?

"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#fita-5-dashboard-ineco","title":"Fita 5: Dashboard INECO","text":"

Prioritat 7

Per definir

"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#fita-6-noves-queries","title":"Fita 6: Noves queries","text":"

Prioritat 8

"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#fita-7-dashboard-gurb","title":"Fita 7: Dashboard GURB","text":"

Prioritat 9

"},{"location":"projecte/2023-06-03-overview/","title":"Overview del projecte per a usuaris","text":"

Descripci\u00f3 del projecte des del punt de vista de l'usuari

"},{"location":"projecte/2023-06-03-overview/#context-de-naixement","title":"Context de naixement","text":""},{"location":"projecte/2023-06-03-overview/#peces","title":"Peces","text":""}]} \ No newline at end of file +{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Sobre Som Energia Jardiner","text":"

Documentaci\u00f3 per eines desenvolupades en el context del projecte somenergia-jardiner de Som Energia.

"},{"location":"#repositoris","title":"Repositoris","text":""},{"location":"#accessos-directes","title":"Accessos directes","text":""},{"location":"adr/2022-10-11-dbt_workflow_meter_alarms/","title":"Com processem les alarmes en DBT?","text":""},{"location":"adr/2022-10-11-dbt_workflow_meter_alarms/#context-and-problem-statement","title":"Context and Problem Statement","text":"

Alarmes decomptadors venen en batches que poden venir molt tard.

"},{"location":"adr/2022-10-11-dbt_workflow_meter_alarms/#decision-drivers","title":"Decision Drivers","text":"

unsupported ara mateix

"},{"location":"adr/2022-10-11-dbt_workflow_meter_alarms/#considered-options","title":"Considered Options","text":"
  1. Calcular l'alarma de les \u00faltimes N hores (rang definit per l'alarma)
  2. Calcular per cada row si en aquell moment hi havia alarma o no

pseudocode-ish idea: Energy last 24h is 0 -> alarm

select sum(energy) from select lag(energy, partition by device order by date desc) from (select * from registry where device1 and date > last_date_1 or device2 and date > last_date_2) group

on last_date = select max(date) from processed group by device

  1. Union amb jinja de 2

select '' as lectura_date, '' as device, '' as alarm

calculated_alarms:

{% foreach row in meters%} UNION opci\u00f3 2 select max(energy) > 0 from registry where device = row['device'] and date > row['date'] - 24h

select from {{ this }} order by date desc limit 1

materialitzada: select max(date) from calculated_alarms group by device

  1. incremental de dbt + foreach device

{{ config(materialized='incremental') }}

select date, max(energy) > 0 from registry --> caldr\u00e0 un lag o algo

from raw_app_data.events

{% if is_incremental() %} where event_time > (select max(event_time) from {{ this }})

quan entri un device nou hem de comprovar que where event_time > NULL torna totsels resultats

source

"},{"location":"adr/2022-10-11-dbt_workflow_meter_alarms/#decision-outcome","title":"Decision Outcome","text":"

Chosen option: triem l'opci\u00f3 2. per suportar calcular alarmes en el passat

"},{"location":"adr/2022-10-11-dbt_workflow_meter_alarms/#positive-consequences","title":"Positive Consequences","text":""},{"location":"adr/2022-10-11-dbt_workflow_meter_alarms/#negative-consequences","title":"Negative Consequences","text":""},{"location":"adr/2022-10-14-meter_alarm_logic/","title":"Meter alarm logic","text":"

Technical Story: [description | ticket/issue URL]

"},{"location":"adr/2022-10-14-meter_alarm_logic/#context-and-problem-statement","title":"Context and Problem Statement","text":"

Les alarmes de comptadors s'han de notificar.

"},{"location":"adr/2022-10-14-meter_alarm_logic/#decision-drivers","title":"Decision Drivers","text":"

coberts: * moxa ens arriben les lectures cada 24h * hi ha comptadors sense lectures durant temps * la notificaci\u00f3 diuen que la volen un cop al dia al llegir del moxa

"},{"location":"adr/2022-10-14-meter_alarm_logic/#considered-options","title":"Considered Options","text":""},{"location":"adr/2022-10-14-meter_alarm_logic/#decision-outcome","title":"Decision Outcome","text":""},{"location":"adr/2022-10-14-meter_alarm_logic/#positive-consequences","title":"Positive Consequences","text":""},{"location":"adr/2022-10-14-meter_alarm_logic/#negative-consequences","title":"Negative Consequences","text":"

podem

"},{"location":"adr/2022-10-14-meter_alarm_logic/#pros-and-cons-of-the-options","title":"Pros and Cons of the Options","text":""},{"location":"adr/2022-10-14-meter_alarm_logic/#guardar-taula-i-fer-diff","title":"guardar taula i fer diff","text":"

ho farem amb un pandas/sql f\u00e0cil per no posar dbt a producci\u00f3 encara

--> ho acabar\u00e0 fent dbt a mig termini

"},{"location":"adr/2022-10-14-meter_alarm_logic/#links","title":"Links","text":""},{"location":"adr/2022-10-14-spines/","title":"Spines","text":"

Technical Story: [description | ticket/issue URL]

"},{"location":"adr/2022-10-14-spines/#context-and-problem-statement","title":"Context and Problem Statement","text":"

[Podem fer un spine amb un left join sense condici\u00f3 contra plant i meter per tenir el carret buit i despr\u00e9s fer un left join amb les lectures

dbt_spine left join plant left join meter on plant.id = meter.plant

"},{"location":"adr/2022-10-14-spines/#decision-drivers","title":"Decision Drivers","text":""},{"location":"adr/2022-10-14-spines/#considered-options","title":"Considered Options","text":""},{"location":"adr/2022-10-14-spines/#decision-outcome","title":"Decision Outcome","text":"

Chosen option: \"[option 1]\", because [justification. e.g., only option, which meets k.o. criterion decision driver | which resolves force force | \u2026 | comes out best (see below)].

"},{"location":"adr/2022-10-14-spines/#positive-consequences","title":"Positive Consequences","text":""},{"location":"adr/2022-10-14-spines/#negative-consequences","title":"Negative Consequences","text":""},{"location":"adr/2022-10-14-spines/#pros-and-cons-of-the-options","title":"Pros and Cons of the Options","text":""},{"location":"adr/2022-10-14-spines/#option-1","title":"[option 1]","text":"

[example | description | pointer to more information | \u2026]

"},{"location":"adr/2022-10-14-spines/#option-2","title":"[option 2]","text":"

[example | description | pointer to more information | \u2026]

"},{"location":"adr/2022-10-14-spines/#option-3","title":"[option 3]","text":"

[example | description | pointer to more information | \u2026]

"},{"location":"adr/2022-10-14-spines/#links","title":"Links","text":""},{"location":"adr/2022-10-26-real-time_alarms/","title":"Real time alarms","text":"

Technical Story: [description | ticket/issue URL]

"},{"location":"adr/2022-10-26-real-time_alarms/#context-and-problem-statement","title":"Context and Problem Statement","text":"

Necessitem alertes d'incid\u00e8ncies dels diferents dispositius de les plantes. Cada dispositiu t\u00e9 especificitats: granularitat temporal de les lectures, lag natural en al c\u00e0rrega de lectures i casu\u00edstiques concretes que es consideren errors.

Dispositius: - Meter ip Granularitat lectura: 1 hora Cada quan arriben les lectures: 20 min (per\u00f2 ERP sincronitza cada 2 hores) Valors historics updatable: No actualment (a l'erp s\u00ed) Poden apar\u00e8ixer lectures antigues: No actualment (a l'erp s\u00ed)

Casu\u00edstica: Meter \u00e9s variable enla cad\u00e8ncia d'enviar lectures (sovint tarda a reconnectar)

"},{"location":"adr/2022-10-26-real-time_alarms/#decision-drivers","title":"Decision Drivers","text":""},{"location":"adr/2022-10-26-real-time_alarms/#considered-options","title":"Considered Options","text":""},{"location":"adr/2022-10-26-real-time_alarms/#decision-outcome","title":"Decision Outcome","text":"

Hem decidit atacar la opci\u00f3 HI i veure si podem reduir a 5 minuts d'execuci\u00f3 amb incremental. Probablement no ser\u00e0 possible i necessitarem un fast-track (opci\u00f3 RL) per les alertes real-time.24h

Aquest fast track s'executar\u00e0 cincminutalment o podem veure si es podria fer en streaming EL -> T -> N(otificaci\u00f3) Ser\u00e0 exclusiu per alertes i amb poca consolidaci\u00f3 a base de dades i no revisitar\u00e0 el passat. Nom\u00e9s NOW()-ish.

"},{"location":"adr/2022-10-26-real-time_alarms/#positive-consequences","title":"Positive Consequences","text":""},{"location":"adr/2022-10-26-real-time_alarms/#negative-consequences","title":"Negative Consequences","text":""},{"location":"adr/2022-10-26-real-time_alarms/#pros-and-cons-of-the-options","title":"Pros and Cons of the Options","text":""},{"location":"adr/2022-10-26-real-time_alarms/#hi-calcular-la-taula-historica","title":"HI - Calcular la taula hist\u00f2rica","text":"

[example | description | pointer to more information | \u2026] 

DAG:\n(production_inverter) -> (alarm) -> (alrm2)\n\nwith ( Taula production_inverter (registry gapfilled, clean, alarms )) as foo\n\ntime     | inverter_id | energy | n_readings | alarm_no_reading | alarm_zero_daylight\n10:00:00      16           34          1            FALSE               FALSE\n09:55:00      16         0/NULL        0            TRUE                NULL\n09:50:00      16            0          1            FALSE               TRUE\n\nselect last row of foo where alarm = TRUE\n
"},{"location":"adr/2022-10-26-real-time_alarms/#rt-calcular-una-taula-real-time-i-una-historica-mes-lenta","title":"RT - Calcular una taula real-time i una hist\u00f2rica m\u00e9s lenta","text":"

[example | description | pointer to more information | \u2026]

RT:

real-time, per les alarmes que necessiten notificaci\u00f3 immediata, i per cada tipus d'alarma 

with foo as (\nselect\n    count(energy),\n    max(energy)\nfrom registry\nwhere NOW() - device.alarm_offset < time\ngroup by device\n)\nselect\n now() as time,\n count = 0 as no_reading_alarm,\n max = 0 as no_energy_alarm\nfrom foo\n
notify OK/KO en aquella execuci\u00f3 fent un diff de la taula anterior com ara

hist\u00f2ric: com tenim ara, al final del dia o d'una setmana, re-calcular les alarmes hist\u00f2ricament

"},{"location":"adr/2022-10-26-real-time_alarms/#st-trobar-la-manera-de-fer-stream-enlloc-de-batch","title":"ST- Trobar la manera de fer stream enlloc de batch","text":""},{"location":"adr/2022-10-26-real-time_alarms/#links","title":"Links","text":""},{"location":"adr/2023-06-13-obt_plantes/","title":"Pipeline de dades revisited","text":"

Technical Story:

Com ser\u00e0 la obt? horaria calculada un cop cada hora? I quan passem a comptadors 15-minutals, segurem amb dbt run 15-minutal? Quina frescor de les dades cal? --> posem-ho a un adr diferent

"},{"location":"adr/2023-06-13-obt_plantes/#context-and-problem-statement","title":"Context and Problem Statement","text":"

Si les dades cincminutals s\u00f3n horaries, la obt de tots els indicadors tamb\u00e9?

Passarem a comptadors 15minutals en algun moment proper.

"},{"location":"adr/2023-06-13-obt_plantes/#decision-drivers","title":"Decision Drivers","text":""},{"location":"adr/2023-06-13-obt_plantes/#considered-options","title":"Considered Options","text":""},{"location":"adr/2023-06-13-obt_plantes/#decision-outcome","title":"Decision Outcome","text":"

Chosen option: \"[option 1]\", because [justification. e.g., only option, which meets k.o. criterion decision driver | which resolves force force | \u2026 | comes out best (see below)].

"},{"location":"adr/2023-06-13-obt_plantes/#positive-consequences","title":"Positive Consequences","text":""},{"location":"adr/2023-06-13-obt_plantes/#negative-consequences","title":"Negative Consequences","text":""},{"location":"adr/2023-06-13-obt_plantes/#pros-and-cons-of-the-options","title":"Pros and Cons of the Options","text":""},{"location":"adr/2023-06-13-obt_plantes/#option-1","title":"[option 1]","text":"

[example | description | pointer to more information | \u2026]

"},{"location":"adr/2023-06-13-obt_plantes/#option-2","title":"[option 2]","text":"

[example | description | pointer to more information | \u2026]

"},{"location":"adr/2023-06-13-obt_plantes/#option-3","title":"[option 3]","text":"

[example | description | pointer to more information | \u2026]

"},{"location":"adr/2023-06-13-obt_plantes/#links","title":"Links","text":""},{"location":"adr/2023-06-13-pipeline_dades_llargues/","title":"Pipeline de dades revisited","text":"

Technical Story:

https://trello.com/c/rmCTOJrH/205-decidir-com-farem-lobt-i-el-cincminutal-r%C3%A0pid

Passem a una macrotaula de dades llargues clau-valor per a les dades cincminutals?

Com ser\u00e0 la obt? horaria calculada un cop cada hora? I quan passem a comptadors 15-minutals, segurem amb dbt run 15-minutal? Quina frescor de les dades cal? --> posem la discussi\u00f3 de la obt a un adr diferent

"},{"location":"adr/2023-06-13-pipeline_dades_llargues/#context-and-problem-statement","title":"Context and Problem Statement","text":"

Ara que sabem molt m\u00e9s dbt i que incorporarem un prove\u00efdor de dades cincminutals amb dades llargues que deprecar\u00e0 l'esquema antic, ens plantegem si haur\u00edem de basar en una macrotaula de dades llargues per les dades cincminutals.

"},{"location":"adr/2023-06-13-pipeline_dades_llargues/#decision-drivers","title":"Decision Drivers","text":""},{"location":"adr/2023-06-13-pipeline_dades_llargues/#considered-options","title":"Considered Options","text":"
  1. normalitzar dset i fer un multisource normalitzat com est\u00e0 ara
  2. desnormalitzar lo que hi ha ara i fer un multisource com ens envia dset
  3. fer una pseudo-obt llarga a partir de dset i fer le multisource al mart
"},{"location":"adr/2023-06-13-pipeline_dades_llargues/#decision-outcome","title":"Decision Outcome","text":"

Chosen option: opci\u00f3 3

"},{"location":"adr/2023-06-13-pipeline_dades_llargues/#negative-consequences","title":"Negative Consequences","text":""},{"location":"adr/2023-06-13-pipeline_dades_llargues/#pros-and-cons-of-the-options","title":"Pros and Cons of the Options","text":""},{"location":"adr/2023-06-13-pipeline_dades_llargues/#option-1","title":"[option 1]","text":"

[example | description | pointer to more information | \u2026]

"},{"location":"adr/2023-06-13-pipeline_dades_llargues/#option-2","title":"[option 2]","text":"

[example | description | pointer to more information | \u2026]

"},{"location":"adr/2023-06-13-pipeline_dades_llargues/#option-3","title":"[option 3]","text":"

[example | description | pointer to more information | \u2026]

"},{"location":"adr/2023-06-13-pipeline_dades_llargues/#links","title":"Links","text":""},{"location":"adr/2023-09-12-redash_vs_superset/","title":"Pipeline de dades revisited","text":""},{"location":"adr/2023-09-12-redash_vs_superset/#context-and-problem-statement","title":"Context and Problem Statement","text":"

En el seu moment vam triar redash perqu\u00e8 podia tenir apis com a sources i el dev \u00e9s molt m\u00e9s c\u00f2mode perqu\u00e8 no est\u00e0 tant layeritzat (dataset/query - chart - dashboard).

A la pr\u00e0ctica no hem fet servir mai la lectura directa de les apis perqu\u00e8 les aut\u00e8ntificacions no estaven suportades, ens agradava tenir una c\u00f2pia, t'obligava a treballar en queries de l'sqlite de redash.

A m\u00e9s a m\u00e9s, redash delega la gesti\u00f3 d'usuaris a la db, cosa que implica que no pots limitar que un usuari pugui veure uns valors d'una query i no uns altres, per exemple filtrar per plantes la producci\u00f3, els permisos es fan a nivell de db.

La macrofase de plantmonitor i l'evoluci\u00f3 de l'equip de dades cap a superset amb els altres equips ens porta a replantejar-nos-ho.

Iniciem un spike de superset per a fer els nous dashboards.

"},{"location":"adr/2023-09-12-redash_vs_superset/#decision-drivers","title":"Decision Drivers","text":""},{"location":"adr/2023-09-12-redash_vs_superset/#considered-options","title":"Considered Options","text":"
  1. superset
  2. redash
  3. metabase -- en Diego comenta que en la versi\u00f3 free no tens la capa sem\u00e0ntica
"},{"location":"adr/2023-09-12-redash_vs_superset/#decision-outcome","title":"Decision Outcome","text":"

Chosen option: ?

"},{"location":"adr/2023-09-12-redash_vs_superset/#positive-consequences","title":"Positive Consequences","text":""},{"location":"adr/2023-09-12-redash_vs_superset/#negative-consequences","title":"Negative Consequences","text":""},{"location":"adr/2023-09-12-redash_vs_superset/#pros-and-cons-of-the-options","title":"Pros and Cons of the Options","text":""},{"location":"adr/2023-09-12-redash_vs_superset/#redash","title":"redash","text":"

\ud83d\udc4d

mustache

definir parametres a nivell de default de charts, de chart especific o dashboard

\ud83d\udc4e

no apte per heavy duty (cache regular)

"},{"location":"adr/2023-09-12-redash_vs_superset/#superset","title":"superset","text":"

\ud83d\udc4d

cohesi\u00f3 cooperativa -- tota la cooperativa fa servir superset

\ud83d\udc4e

No pots fixar filtres a nivell de chart al dashboard, cal fer-ne una c\u00f2pia. Implica que si volem canviar com s'ensenya un chart de planta, els haurem de canviar tots un a un.

El jinja \u00e9s funky. No \u00e9s clar com fer servir el \"filter_values\" a dins d'una query. Les queries din\u00e0miques s\u00f3n molt menys directes de fer que al redash.

"},{"location":"adr/template/","title":"Template","text":"

^-- [short title of solved problem and solution]

Technical Story: [description | ticket/issue URL]

"},{"location":"adr/template/#context-and-problem-statement","title":"Context and Problem Statement","text":"

[Describe the context and problem statement, e.g., in free form using two to three sentences. You may want to articulate the problem in form of a question.]

"},{"location":"adr/template/#decision-drivers","title":"Decision Drivers","text":""},{"location":"adr/template/#considered-options","title":"Considered Options","text":""},{"location":"adr/template/#decision-outcome","title":"Decision Outcome","text":"

Chosen option: \"[option 1]\", because [justification. e.g., only option, which meets k.o. criterion decision driver | which resolves force force | \u2026 | comes out best (see below)].

"},{"location":"adr/template/#positive-consequences","title":"Positive Consequences","text":""},{"location":"adr/template/#negative-consequences","title":"Negative Consequences","text":""},{"location":"adr/template/#pros-and-cons-of-the-options","title":"Pros and Cons of the Options","text":""},{"location":"adr/template/#option-1","title":"[option 1]","text":"

[example | description | pointer to more information | \u2026]

"},{"location":"adr/template/#option-2","title":"[option 2]","text":"

[example | description | pointer to more information | \u2026]

"},{"location":"adr/template/#option-3","title":"[option 3]","text":"

[example | description | pointer to more information | \u2026]

"},{"location":"adr/template/#links","title":"Links","text":""},{"location":"deprecated/TODOs/","title":"Views de plantmonitor a migrar a dbt","text":"

~ marca les que segurament no cal fer, al gust

view_availability.sql\n~view_q_hourly_debug.sql~\nview_q_monthly.sql\n~view_cos_phi_hourly_debug.sql~\nview_cos_phi_monthly.sql\nview_expected_power.sql\nview_hd_daily.sql\nview_ht_daily.sql\nview_ht_monthly.sql\nview_pr_hourly.sql\nview_target_energy.sql\n
"},{"location":"deprecated/TODOs/#agregacions","title":"Agregacions","text":"

No cal fer-les perqu\u00e8 son agraupacions que segurament farem servir m\u00e8triques de dbt tipus:

view_inverter_energy_daily.sql\nview_inverter_energy_monthly.sql\nview_meter_export_energy_daily.sql\nview_meter_export_energy_monthly.sql\nview_average_yearly_production_plant.sql\nview_satellite_irradiation.sql\n

A m\u00e9s a m\u00e9s les m\u00e8triques de dbt resolen el problema seg\u00fcent:

sum(a/x) -- horaria

sum(a/x) + sum(b/y) ... mensual wrong

sum((a+b+c)/(x+y+z)) -- mensual correct

"},{"location":"deprecated/TODOs/#views-per-limitar-lacces-a-la-info-dels-tecnics","title":"views per limitar l'acc\u00e9s a la info dels t\u00e8cnics","text":"

No cal fer-ho, limitarem l'acc\u00e9s a la wide table o farem servir els permisos del superset o ja veurem

view_plants_energes.sql\nview_plants_energetica.sql\nview_plants_ercam.sql\nview_plants_exiom.sql\nview_meterregistry_energetica.sql\nview_inverter_energy_daily_exiom.sql\nview_inverter_intensity_energes.sql\nview_inverter_intensity_ercam.sql\nview_inverter_intensity_exiom.sql\nview_inverter_power_energes.sql\nview_inverter_power_ercam.sql\nview_inverter_power_exiom.sql\nview_inverterregistry_energes.sql\nview_inverterregistry_ercam.sql\nview_inverterregistry_exiom.sql\nview_inverter_temperature_exiom.sql\nview_sensorirradiationregistry_energes.sql\nview_sensorirradiationregistry_ercam.sql\n
"},{"location":"deprecated/TODOs/#no-shan-de-fer-obsoletes","title":"no s'han de fer (obsoletes)","text":"
zero_inverter_power_at_daylight.sql\nzero_sonda_irradiation_at_daylight.sql\n
"},{"location":"desenvolupadors/2023-02-08-novu_notifications/","title":"Novu helpers and quickstart","text":""},{"location":"desenvolupadors/2023-02-08-novu_notifications/#swagger-api","title":"Swagger API","text":"

Novu api

"},{"location":"desenvolupadors/2023-02-08-novu_notifications/#overview","title":"overview","text":"

Features in novu get to the api waaaay before the get to the UI, so keep it at hand since most of the changes (adding removing subscribers, topics, etc.) will be done via python, react or postman via api rest calls.

"},{"location":"desenvolupadors/2023-02-08-novu_notifications/#helpers","title":"helpers","text":"

We have a handful of scripts to interact with novu via airflow or command-line scripts/notify_alert.py and scripts/novu_interface.py.

The notify_alert is ran by airflow to periodically check the alarm tables and trigger a notification on change.

novu_interface is more of a command-line util to do novu stuff manually, like adding subscribers or topics.

For example, to add a topic of a plant we'd do

$ python -m scripts.novu_interface --help\n\nUsage: python -m scripts.novu_interface [OPTIONS] URL API_KEY TOPIC_KEY\n                                         TOPIC_NAME\n\nUsage: python -m scripts.novu_interface add-topic [OPTIONS] URL API_KEY\n                                                   TOPIC_KEY TOPIC_NAME\n\n\u256d\u2500 Arguments \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502 *    url             TEXT  [default: None] [required]                  \u2502\n\u2502 *    api_key         TEXT  [default: None] [required]                  \u2502\n\u2502 *    topic_key       TEXT  [default: None] [required]                  \u2502\n\u2502 *    topic_name      TEXT  [default: None] [required]                  \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n\u256d\u2500 Options \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502 --help          Show this message and exit.                            \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n

$ python -m scripts.novu_interface add-topic {novu_base_url} {api_key} {topic_key} {topic_name}\n$ python -m scripts.novu_interface add-topic http://moll3.somenergia.lan:3000/v1 {api_key} asomada-topic-1 \"asomada general topic\"\n
and to get the topics

$ python -m scripts.novu_interface get-topic {novu_base_url} {api_key} {topic_key} {topic_name}\n\n2023-01-27 16:39:29,080 [INFO] {\"page\":0,\"totalCount\":1,\"pageSize\":10,\"data\":[\n    {\"_id\":\"63d3ef030d600f1df1a3829e\",\"_environmentId\":\"6349479f3043d373c3d5fc07\",\"_organizationId\":\"6349479f3043d373c3d5fc02\",\"key\":\"asomada-topic-1\",\"name\":\"asomada general topic\",\"subscribers\":[]}\n]}\n

Let's add some subcribers to it:

$ python -m scripts.novu_interface add-subscriber http://moll3.somenergia.lan:3000/v1 {api_key} pol_recipient\n$ python -m scripts.novu_interface add-topic-subscriber http://moll3.somenergia.lan:3000/v1 {api_key} asomada-topic-1 pol_recipient\n
"},{"location":"desenvolupadors/2023-06-05-download_solargis_manually/","title":"Manually download a solargis reading","text":"

To download a specific day you just need to call

python -m scripts.solargis <dbapi> <apikey> 2023-05-13 2023-05-13 1 2 3 4\n 5 6 7 9 13 14 15 16 17 22 40\n

See python -m scripts.solargis --help for info on each parameter (essentially time range and plant id list) and the wiki or the pyxis server for credentials.

Note that to download one day you have to put the same from and to date. The from is start day and to date is end day. The script DOES NOT check for duplicates. You'll have to clean up yourself using the query time if you made a mistake.

"},{"location":"desenvolupadors/2023-06-10-overview-devs/","title":"Overview for devs","text":""},{"location":"desenvolupadors/2023-06-10-overview-devs/#quickstart","title":"quickstart","text":"

To fully deploy the dbt models to the production schema you can do

dbt build -s tag:jardiner,tag:legacy --store-failures --full-refresh --target prod\n

You will need to have the prod profile in your ~/.dbt/profiles.yml file.

"},{"location":"desenvolupadors/2023-06-10-overview-devs/#current-state-of-the-project","title":"Current State of the project","text":""},{"location":"desenvolupadors/2023-06-10-overview-devs/#pieces","title":"Pieces","text":""},{"location":"desenvolupadors/2023-06-10-overview-devs/#repos","title":"repos","text":"

Plantmonitor

Fa d'ORM de les plantes, cont\u00e9 la api i \u00e9s codi de les raspberrypis. A m\u00e9s a m\u00e9s tamb\u00e9 t\u00e9 el codi d'ingesta d'apis de tercers.

Warning

La meitat del plantmonitor acabar\u00e0 deprecated pel prove\u00efdor de dades

Plant Reader

Lectura remota de les plantes. Actualment nom\u00e9s Asomada es llegeix remotament i es guarda a plant_lake.

Acabar\u00e0 deprecated pel prove\u00efdor de dades.

Actualment tamb\u00e9 allotja la lectura de la API del prove\u00efdor de dades.

Jardiner

dbt de les plantes. Actualment tot views.

"},{"location":"desenvolupadors/2023-06-10-overview-devs/#data-sources","title":"Data sources","text":""},{"location":"desenvolupadors/2023-06-10-overview-devs/#visualization","title":"Visualization","text":""},{"location":"desenvolupadors/2023-06-10-overview-devs/#project-tructure","title":"Project tructure","text":""},{"location":"desenvolupadors/2023-06-10-overview-devs/#actual","title":"Actual","text":"
flowchart LR\n\ndw[db/plants]\n\nplants -- pull 5' \\n plant_reader/dags --> lake[db/plant_lake]\nlake -- push 5' \\n airbyte --> dw\n\nplants -- pull 5' \\n plantmonitor/main.py --> rPIs -- push 5'  --> dw\nplants -- pull 2h/12h \\n meter \\n ERP's import_tm_data_click.py --> ERP -- pull 20'\u00ba\\n meter data --> dw\nSAT -- pull daily\u00ba\\n irr/expected energy --> dw\nMETEO <-- pull daily\u00ba\\n meter data +  irr/kWh forecast --> dw\nplants -- pull 15'\\n plant_reader/dags --> PLANT -- pull 15'\\n devices data --> db/plant_lake\n\ndw -- dbt views --> prod --> alarms\nprod --> alerts\nprod --> datasets

\u00ba: plantmonitor does it. Update rate defined at conf/startup_configuration.py and each task is run as a function.

"},{"location":"desenvolupadors/2023-06-10-overview-devs/#planned-scheme","title":"Planned scheme","text":"
flowchart LR\n\ndw[db/plants]\n\nplants -- pull 2h/12h \\n meter \\n import_tm_data_click.py --> ERP -- pull 20' \u00ba\\n meter data --> dw\nSAT -- pull daily\\n irr/expected energy --> dw\nMETEO <-- pull daily\\n meter data +  irr/kWh forecast --> dw\nplants -- pull 15' --> PLANT -- pull 15'\\n devices data --> dw\n\n\ndw -- dbt fast --> fast[dbt_prod/fast_] --> union\ndw -- dbt --> obt --> alarms\nobt --> union\nunion -- notify.py --> alerts\nobt --> datasets

Despr\u00e9s de l'ADR d'estructura de pipeline

See Roadmap

"},{"location":"desenvolupadors/2023-07-10-dbt_cmds_utils/","title":"dbt commands","text":"
  1. Generar i servir la documentaci\u00f3 per a poder veure el directed graph.
$ dbt docs generate --project-dir dbt_jardiner --target pre\n\n$ dbt docs serve --project-dir dbt_jardiner --target pre --port 8010\n
  1. Pujar tots els CSVs de seeds
dbt seed --project-dir dbt_jardiner\n
  1. Podem passar variables per command line (en comptes de definir-les a dbt_project.yml i fer-les accesibles per diferents resources):

dbt run --vars '{\"variable\":\"value\"}'\n
Per fer testing amb un csv determinat:

dbt run --project-dir project_x --target testing --vars '{\"test_sample\":\"nom_del_csv\"}' -m model_x\n
"},{"location":"desenvolupadors/2023-07-13-codi_reciclable/","title":"Reusuable sqls","text":""},{"location":"desenvolupadors/2023-07-13-codi_reciclable/#dbt-sql-for-rolling-window","title":"dbt sql for rolling window","text":"
{%- set plant_alarm_ranges = dbt_utils.get_query_results_as_dict(ref('plant_alarm_ranges')) -%}\n\n{% for plant in plant_alarm_ranges %}\n\n  {% if var('plants_code') == 'SomEnergia_Riudarenes_ZE' %}\n      {%- set time_to_sum = '1 hour' -%}\n  {% elif var('plants_code') == 'SomEnergia_Matallana' %}\n      {%- set time_to_sum = '4 hour' -%}\n  {% else %}\n          {{ ref('accounts') }}   accounts\n  {% endif %}\n\n  SELECT\n      *,\n      SUM(export_energy_wh) OVER (\n          PARTITION BY plant_name\n          ORDER BY \"time\"\n          RANGE BETWEEN interval {{time_to_sum}} PRECEDING AND CURRENT ROW\n      ) as export_energy_wh_sum\n  FROM (\n    select * from {{ref('meter_registry_cleaned')}} where codename = {{plants_code}}\n  ) as meter_registry_cleaned_current_plant\n  UNION ALL\n\n{% endfor %}\n
"},{"location":"desenvolupadors/2023-08-28-alta_planta/","title":"Context","text":"

Tenim un pipe de transformaci\u00f3 de dades que acaba en una serie de dashboards de superset que mostra dades de plantes des de el pipe de transformaci\u00f3 de dades arribant del nostre prove\u00efdor de dades.

"},{"location":"desenvolupadors/2023-08-28-alta_planta/#protocol-dalta-duna-planta-fotovoltaica","title":"Protocol d'alta d'una planta fotovoltaica","text":"

Per afegir una nova planta a la visualitzaci\u00f3, cal seguir un protocol documentat al Formulari d\u2019inscripci\u00f3 de noves plantes a Superset

El equip de gestio d'actius ha d'omplir aquest formulari amb les dades de la planta i ens ho han de comunicar per a que puguem seguir amb el proc\u00e9s nosaltres.

Obres en curs amb jardiner-admin

Aquesta tasca est\u00e0 en proc\u00e9s de ser portada al jardiner-admin. Per tant, aquest protocol pot canviar en el futur.

Necessites permisos d'administrador

Aquesta tasca \u00e9s per a desenvolupadors que treballen amb el pipe de transformaci\u00f3 de dades. Si no ets un desenvolupador, no cal que facis aquesta tasca.

Aqui sota ens dediquem a documentar t\u00e9cnicament els passos a seguir des de la perspectiva de desenvolupament.

"},{"location":"desenvolupadors/2023-08-28-alta_planta/#checklist-general-per-it","title":"Checklist general per IT","text":""},{"location":"desenvolupadors/2023-08-28-alta_planta/#descripcio-de-dades","title":"Descripci\u00f3 de Dades","text":""},{"location":"desenvolupadors/2023-08-28-alta_planta/#fitxer-mapeig-de-senyals","title":"Mapeig de senyals de Planta","text":"

Per cada planta, GdA genera fitxers amb informaci\u00f3 de la planta per la seva gesti\u00f3. Aquests fitxers contenen informaci\u00f3 de senyals, aparells, plantes, etc. per\u00f2 sense el context del pipe de Jardiner. Com IT, portem aquesta informaci\u00f3 al fitxer de mapeig de senyals per a que el pipe de Jardiner pugui fer integrar les dades de la planta.

Un exemple d'aquest fitxer pot ser aquest.

"},{"location":"desenvolupadors/2023-08-28-alta_planta/#fitxer-dades-fixes-de-planta","title":"Dades fixes de planta","text":"

Aquest fitxer cont\u00e9 dades fixes de la planta, com ara la pot\u00e8ncia de la planta, la seva ubicaci\u00f3, etc. Es trova a google drive i es transforma amb airbyte aqui

"},{"location":"desenvolupadors/2023-08-28-alta_planta/#fitxer-info-api-solargis","title":"Info API SolarGIS","text":"

GdA ha de crear una entrada per la planta nova al fitxer excel per aix\u00f2

"},{"location":"desenvolupadors/2023-08-28-alta_planta/#fitxer-rendiment-de-planta","title":"Rendiment de planta","text":"

T\u00e9 un nom tipus \"C\u00e0lcul Rendiment de Planta <nom planta>\". El podeu buscar al cercador de drive.

"},{"location":"desenvolupadors/2023-08-28-alta_planta/#fitxer-produccio-actual-de-planta","title":"Objectius de producci\u00f3 de planta","text":"

Es troba al fitxer de google sheets aqui i es transforma amb airbyte aqui.

"},{"location":"desenvolupadors/2023-08-28-alta_planta/#fitxer-unificacio-noms-projectes-generacio","title":"Unificaci\u00f3 Noms Projectes Generaci\u00f3","text":"

Cont\u00e9 els llistat de noms unificats en un sol fitxer. Ves a Unificaci\u00f3 Noms Projectes Generaci\u00f3.

"},{"location":"desenvolupadors/2023-08-28-alta_planta/#fitxer-modelos-contadores-inversores-scadas","title":"Modelos contadores_inversores_SCADAS","text":"

Modelos contadores_inversores_SCADAS

"},{"location":"desenvolupadors/2023-08-28-alta_planta/#descripcio-de-tasques","title":"Descripci\u00f3 de tasques","text":""},{"location":"desenvolupadors/2023-08-28-alta_planta/#afegir-mapeig-airbyte","title":"Afegir dades al mapeig d'airbyte","text":"

Tenim un document on consolidem relacions entre UUIDs de senyals, aparells i plantes que inserim amb airbyte

Aquest mapeig es una traducci\u00f3 del fitxer de mapeigs.

Un cop heu acabat de fer el mapeig, cal engegar la tasca de sync de l'airbyte d'uuids per que les dades arribin a la base de dades. Heu de tenir especial cura d'omplir les columnes updated_at i aquelles que posen UUIDs, ja que amb elles se creuen les dades dins del pipe.

update_at s'actualitza automagicament

Al fitxer de google sheets hi ha un script que actualitza la columna updated_at quan hi ha hagut un canvi a nivell de fila. Es molt limitat i heu de supervisar que s'ha actualitzat correctament, ja que de vegades google sheets no ho fa. Podeu mirar a Extensions > Apps Script per veure el codi.

UUID s\u00f3n versi\u00f3 4

Els senyals porten UUID4 (han de ser versio 4) \u00fanics al mapeig de la planta indicada. Es poden generar UUID4 amb https://www.uuidgenerator.net/ O ben v\u00e9 amb un script de python que podeu afegir a ~/.bash_aliases:

#\u00b7function\u00b7to\u00b7genenerate\u00b7random\u00b7uuid4\u00b7using\u00b7python$\nfunction pyuuid4() {\npython -c \"import uuid$\nfor _ in range(${N:-1}):\n    print(uuid.uuid4())\" ; }\n

i el podeu executar amb N=<n> pyuuid4 per a generar n UUID4.

$ N=10 pyuuid4\nbf98707e-688c-471c-8700-9ffeaf436531\n8f6bfcc0-48bf-4b7c-895f-0b2efa37dd63\nee5dcfd1-80ba-48fb-800d-5bc8c04d217f\nd32b0a23-c479-4ae9-bb54-43390fedf8e4\neced945e-7a91-4451-bbd5-974a82c9c0d4\nfbacbf3d-23f6-4361-b5ec-b48b390d9c09\nb9c4ea66-672e-4dca-ad08-fddacdfb3627\nae3e242e-9a57-4359-a72b-313a72b48348\n9e11a595-c32c-42e1-8549-a5161c0ea624\n172061ed-96b4-4aa3-b5bd-0c66e850383b\n
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#afegir-meteologica-plantmonitor","title":"Afegir planta per meteologica amb plantmonitor","text":"

Qu\u00e8 \u00e9s meteologica?

meteologica \u00e9s un servei que ens permet tenir dades meteorol\u00f2giques de les plantes fotovoltaiques. Aquest servei es comunica amb plantmonitor per a obtenir dades de les plantes.

Qu\u00e8 \u00e9s plantmonitor?

Tots els scripts en aquest apartat fan refer\u00e8ncia al repositori de github plantmonitor. \u00c9s un projecte previ a jardiner que ens permet monitoritzar plantes fotovoltaiques quan l'obtenci\u00f3 de dades la gestionem nosaltres.

A plantmonitor tenim un script que ens permet afegir plantes noves. Aquest script s'ha de cridar amb un fitxer yaml que contingui les dades de la planta.

S'ha de crear la planta a la base de dades tant a la raspberrypi com a plantmonitor (no calen els dispositius, que els crea autom\u00e0ticament). Haureu d'afegir plantlocation, plantparameters i moduleparameters. Trobareu un exemple a plantmonitor

Un cop teniu el fitxer yaml, per exemple data/plant-nova.yaml, podeu cridar l'script addPlant.py amb el seg\u00fcent comandament:

PLANTMONITOR_MODULE_SETTINGS='conf.settings.prod' python addPlant.py data/plant-asomada.yaml\n

Haureu de tenir un fitxer .env.prod amb les variables d'entorn necess\u00e0ries per a que l'script funcioni correctament. Mireu a la documentaci\u00f3 de plantmonitor per a m\u00e9s informaci\u00f3.

De moment manualment cal afegir els registres de plantestimatedmonthlyenergy. L'any \u00e9s irrellevant. Si tenim hist\u00f2rics d'energia objectiu, els podeu afegir.

INSERT INTO\n public.plantestimatedmonthlyenergy(plantparameters, \"time\", monthly_target_energy_kwh)\nVALUES\n (33, '2022-01-01 00:00:00+01', 473630),\n (33, '2022-02-01 00:00:00+01', 453370),\n (33, '2022-03-01 00:00:00+01', 592040),\n (33, '2022-04-01 00:00:00+02', 617000),\n (33, '2022-05-01 00:00:00+02', 667210),\n (33, '2022-06-01 00:00:00+02', 648940),\n (33, '2022-07-01 00:00:00+02', 703030),\n (33, '2022-08-01 00:00:00+02', 644500),\n (33, '2022-09-01 00:00:00+02', 574580),\n (33, '2022-10-01 00:00:00+02', 524110),\n (33, '2022-11-01 00:00:00+01', 446490),\n (33, '2022-12-01 00:00:00+01', 442060);\n
"},{"location":"desenvolupadors/2023-08-28-alta_planta/#afegir-solargis","title":"Afegir planta al script de SolarGIS","text":"

S'ha de modificar el codi de plantmonitor per incloure la nova planta. Podeu trobar un exemple de modificacions github

Amb la MR aprovada i fusionada, s'ha de fer git pull des de el servidor de producci\u00f3 i reiniciar el servei de plantmonitor.

ssh someone@somewhere.lan\nsource /home/<someone>/Envs/plantmonitor/bin/activate\ncd /home/<someone>/somenergia/plantmonitor\ngit pull\npython -m scripts.solargis --help\npython -m scripts.solargis [OPTIONS] DBAPI SOLARGIS_API_KEY FROM_DATE TO_DATE [PLANT_IDS]\nsudo supervisorctl restart plantmonitor\n

Aqui, someone \u00e9s el nom de l'usuari que t\u00e9 acc\u00e9s a somewhere.lan, i PLANT_IDS s\u00f3n els ids de les plantes que vau afegir a l'API de SolarGIS amb la PR anterior. Demaneu acc\u00e9s a somewhere.lan a l'equip de dades.

La API de SolarGIS requereix un token d'acc\u00e9s i es crida remotament amb supervisord, amb codi al repositori de plantmonitor.

L'API de SolarGIS \u00e9s SOAP

L'API es del tipus SOAP i no REST. \u00c9s a dir, demana un XML amb les dades de la planta. Podeu trobar informaci\u00f3 sobre com cridar-la a la seva documentaci\u00f3 d'usuari.

"},{"location":"desenvolupadors/2023-08-28-alta_planta/#dades-requerides-per-aparells","title":"Dades requerides per aparells","text":""},{"location":"desenvolupadors/2023-08-28-alta_planta/#afegir-comptadors-a-lerp","title":"Afegir Comptadors a l'ERP","text":"

Les lectures a l'ERP s\u00f3n necessaries mentre metelogica faci servir les lectures de comptadors que plantmonitor copia de l'ERP. Tamb\u00e9 s\u00f3n necess\u00e0ries per al generationkwh. Per aix\u00f2 cal agafar les dades de Modelos contadores_inversores_SCADAS i afegir-les a l'erp Infrastructura -> Registradors -> tots els registradors.

"},{"location":"desenvolupadors/2023-08-28-alta_planta/#serveis-de-somenergia-addicionals","title":"Serveis de SomEnergia addicionals","text":""},{"location":"desenvolupadors/2023-08-28-alta_planta/#nagios","title":"nagios","text":"

Preguntar a Gesti\u00f3 d'Actius si encara fan servir el nagios. Si si, afegir-la al nagios o modificar-la en cas de canvi d\u2019ip p\u00fablica

"},{"location":"desenvolupadors/2023-08-28-alta_planta/#opendata","title":"opendata","text":"

Cal afegir la info de la nova planta a taula literal que es fa servir a somenergia-opendata/som_opendata/queries/plantpower.sql. Un cop aquesta info estigui a plantmonitor, l\u2019opendata podr\u00e0 agafar-la directament per sql i no caldr\u00e0 fer aquest pas.

"},{"location":"desenvolupadors/2023-08-28-alta_planta/#generationkwh","title":"generationkwh","text":"

Si la planta est\u00e0 inclosa al generation, \u00e9s important afegir-la a l\u2019erp abans de la mitja nit del dia de posada a producci\u00f3. Si no, cal descartar c\u00e0lculs de drets del generation per a que incloguin els resultats. Si, a m\u00e9s, els drets s\u2019han comen\u00e7at a gastar (15 dies de marge de facturaci\u00f3), cal reperfilar els nous drets amb la producci\u00f3 afegida perqu\u00e8 no superi els drets ja atorgats.

La nova planta es crea fent servir l\u2019script a somenergia-generationkwh/scripts/genkwh_plants.py

Un exemple complert de com afegir una planta amb aquest script el podeu trobar a https://github.com/Som-Energia/somenergia-generationkwh/blob/master/scripts/genkwh_migrate_plantmeter_1_7_0_newplant.sh

Es recomana provar primer a un ERP de testing o local, i, despr\u00e9s aplicar-ho a producci\u00f3.

"},{"location":"desenvolupadors/2023-08-28-alta_planta/#reperfilacio-de-dades","title":"Reperfilaci\u00f3 de dades","text":"

Un exemple d\u2019script de migraci\u00f3 en la que vam haver de reperfilar: https://github.com/Som-Energia/somenergia-generationkwh/blob/master/scripts/genkwh_migrate_plantmeter_1_7_2_fontivsolarfix.sh

"},{"location":"desenvolupadors/2024-01-11-incremental-dedup/","title":"Incremental Dedup","text":"

Exemple de com fer un incremental amb deduplicaci\u00f3 en dbt.

{{\n  config(\n    materialized = 'incremental',\n    unique_key = ['signal_uuid','ts'],\n    incremental_strategy = 'merge',\n    incremental_predicates = [\n      \"DBT_INTERNAL_DEST.queried_at > dateadd(day, -1, current_date)\"\n    ]\n  )\n}}\n\nwhere\n    ts < now() - interval '1 hour'  {#- select only freshly ingested rows #}\n\n    {% if is_incremental() -%}\n        and queried_at > coalesce((select max(queried_at) from {{ this }}), '1900-01-01') and queried_at > now() - interval '2 hour'\n    {%- endif %}\n
source senyal ts valor queried_at energia_comptador 03:00 80 +1d 04:00 energia_comptador 03:00 null 04:00 potencia_inversor 03:00 50 +1d 04:00 potencia_inversor 03:00 50 04:00

Son ara les 15:04. en la materialitzada tindrem

source senyal ts valor potencia_inversor 14:00 67 energia_comptador 14:00 null

o bien, la \u00faltima fila no hi ser\u00e0:

source senyal ts valor potencia_inversor 14:00 67"},{"location":"desenvolupadors/2024-01-11-incremental-dedup/#explicacio","title":"Explicaci\u00f3","text":"

where max(ts) = 14:00:

source senyal ts valor potencia_inversor 14:00 67 energia_comptador 14:00 null

where max(ts) - 24h:

source senyal ts valor queried_at energia_comptador 03:00 80 +1d 04:00 potencia_inversor 03:00 50 +1d 04:00 potencia_inversor 03:00 50 04:00 potencia_inversor 14:00 67 energia_comptador 14:00 null

unique (signal_uuid, ts):

source senyal ts valor queried_at energia_comptador 03:00 80 +1d 04:00"},{"location":"desenvolupadors/2024-01-11-incremental-dedup/#prova-practica","title":"prova pr\u00e0ctica","text":"

Abans d'executar l'incremental tenim aquest tres \u00faltims registres a materialized one hour late

group_name queried_at ts signal_code signal_device_type signal_device_uuid signal_frequency signal_id signal_is_virtual signal_last_ts signal_last_value signal_type signal_tz signal_unit signal_uuid signal_uuid_raw signal_value materialized_at SE_vallehermoso 2024-01-09 13:24:42.488123+01 2024-01-05 23:15:00+01 ce_eactexp meter 08bc8d88-4ea2-4ee9-b817-00e1f07debba 15 minutes 983894 false 2024-01-08 23:45:00+01 0 absolute Europe/Madrid kWh 646d1ec2-0ca2-4c22-9739-8161aafb224e 646d1ec2-0ca2-4c22-9739-8161aafb224e 0.0 2024-01-09 13:36:52.925381+01 SE_asomada 2024-01-09 13:24:42.488123+01 2024-01-05 23:00:00+01 s8102 meter c5ed9fab-e73c-40a5-acb3-113e22052fd6 5 minutes 1031060 false 2024-01-09 11:47:00+01 60.2 absolute Europe/Madrid V dd065f6a-4ded-41a8-98d3-ba239f9a2a47 dd065f6a-4ded-41a8-98d3-ba239f9a2a47 60.2 2024-01-09 13:36:52.925381+01 SE_tahal 2024-01-09 13:24:42.488123+01 2024-01-05 23:45:00+01 ce_ercapexp_er3 meter 5deb3780-02e2-4dcf-a6a7-de646991762c 15 minutes 983890 false 2024-01-08 23:45:00+01 0 absolute Europe/Madrid kVArh 31ae7d09-95db-447d-b5bf-358b056ef5bf 31ae7d09-95db-447d-b5bf-358b056ef5bf 0.0 2024-01-09 13:36:52.925381+01"},{"location":"desenvolupadors/2024-02-16-row-level-security/","title":"Row Level Security","text":""},{"location":"desenvolupadors/2024-02-16-row-level-security/#context","title":"Context","text":"

Tenim usuaris externs que necessiten mirar dades de plantes, per\u00f2 nom\u00e9s de les plantes que gestionen. Aix\u00f2 ho podem fer amb Row Level Security a Superset.

El proc\u00e9s \u00e9s el seg\u00fcent:

  1. Crear un rol Manteniment_<nom empresa> si no existeix, pel cas de noves plantes.
  2. Afegir una regla Row Level Security Pareu atenci\u00f3 al camp \"clause\", s'afegir\u00e0 com a condici\u00f3 where exclusiva (and) als datasets que seleccionem.
  3. Donar acc\u00e9s als charts d'un dashboard
  4. Afegir la o les usu\u00e0ries i assignar-los els rols [Gamma] i [Manteniment_<nom empresa>]

I ja ho tens. Ho he fet amb energes amb la clause nom_planta in (noms plantes)

"},{"location":"desenvolupadors/2024-02-16-row-level-security/#decisions","title":"Decisions","text":"

Hem decidit a 02/2024 que no val la pena propagar l'empresa de manteniment en el pipe per a cobrir aquest use case, ja que no cal afegir 5m aquestes metadadades per a un tema de visualitzaci\u00f3 i permisos. Ho gestionem en la capa de visualitzaci\u00f3.

Canvis d'empresa de manteniment

Canvis d'empresa de manteniment s'hauran de fer al Superset (i a les metadades de la planta, per coher\u00e8ncia).

"},{"location":"desenvolupadors/2024-02-16-row-level-security/#referencies","title":"Refer\u00e8ncies","text":""},{"location":"projecte/2023-06-03-macrofase%20roadmap/","title":"Macrofase Jardiner 2023","text":"

Document de Roadmap

"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#roadmap-overview","title":"Roadmap overview","text":""},{"location":"projecte/2023-06-03-macrofase%20roadmap/#flux-de-dades","title":"Flux de dades","text":"
flowchart LR\n\nrPIs -- push 5'  --> dw[db/plants]\nERP -- pull 20' \u00ba\\n meter data --> dw\nSAT -- pull daily\\n irr/expected energy --> dw\nMETEO <-- pull daily\\n meter data +  irr/kWh forecast --> dw\nPLANT -- pull 15'\\n devices data --> dw\n\ndw -- dbt views --> prod --> alarms\nprod --> alerts\nprod --> datasets
"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#fita-1","title":"Fita 1","text":"

Prioritat 1

Aprox 1-2 fases

Definition of Done (DoD): veure per tema

Cost en mitjos dies de l'equip sencer entre par\u00e8ntesis (N)

"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#dashboard-tecnics-de-plantes-critiques","title":"Dashboard t\u00e8cnics de plantes cr\u00edtiques","text":"

Pulir el dashboard de t\u00e8cnics per a Asomada i Llanillos.

DoD: TODO

"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#obtencio-de-dades-de-tots-els-equips-i-totes-les-plantes","title":"Obtenci\u00f3 de dades de tots els equips i totes les plantes","text":"

DoD:

"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#introduccio-al-jardiner","title":"Introducci\u00f3 al jardiner","text":"

Estat del codi i rep\u00e0s de l'estructura. Documentaci\u00f3 per devs.

"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#dt-obt-i-fast-lane","title":"[DT] OBT i fast-lane","text":"

Crear la obt cincminutal incremental i/o el fast-lane cincminutal i la obt di\u00e0ria.

Crear el live per late arriving facts de comptadors i apis.

En funcionament normal s\u00f3n delays de m\u00e0xim un dia. Amb incid\u00e8ncia poden ser 1-2 mesos m\u00e0xim. I en casos excepcionals, molt m\u00e9s (candidat a full-refresh manual o alguna altra cosa)

"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#aterracio-de-tot-plegat","title":"Aterraci\u00f3 de tot plegat","text":""},{"location":"projecte/2023-06-03-macrofase%20roadmap/#fita-2-dashboard-overview-plantes","title":"Fita 2: Dashboard Overview plantes","text":"

Prioritat 4

Disseny d'un Dashboard amb un llistat de totes les plantes amb els principals indicadors (mesures en temps real i acumulades i resum alarmes). Pendent fer una proposta per part de GA. Des d'aquest dashboard s'ha de poder accedir al Dashboard de cada planta per obtenir informaci\u00f3 detallada. Esborrany inicial falta versi\u00f3 actualitzada. (6)

Implementaci\u00f3 (9)

"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#fita-3-dashboard-planta","title":"Fita 3: Dashboard planta","text":"

Prioritat 5

Definir versi\u00f3 revisada del Dashboard de planta per a totes les plantes: Unifilar din\u00e0mic, dades fixes i reempla\u00e7ar alarmes, millorar la velociatat i treure el dataset de sota, afegir solargis + coses que puguin sortir despr\u00e9s (3)

Implementar versi\u00f3 revisada del Dashboard de planta per a totes les plantes: Unifilar din\u00e0mic, dades fixes i reempla\u00e7ar alarmes, millorar la velocitat i treure el dataset de sota, afegir solargis + coses que puguin sortir despr\u00e9s (3)

Dashboard Ringsted per plantes -- particularitats (4)

Dashboard Gasdtrup -- particularitats (4)

Dashboard de planta t\u00edpica (Alcolea, etc) (0)

"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#fita-4-dashboard-tecnics","title":"Fita 4: Dashboard t\u00e8cnics","text":"

Prioritat 6

Definir la versi\u00f3 final amb grafiques de Solar Gis, Alarmes i pot ser alarmes. Pendent fer una revisi\u00f3 de les funcionalitats per part de GA. Incloure un dashboard per cadascuna de les empreses de manteniment. Al dashboard unificat d'alarmes estat actual afegir des de quan est\u00e0 activada l'alarma

Implementar la versi\u00f3 final amb grafiques de Solar Gis, Alarmes i pot ser alarmes. Pendent fer una revisi\u00f3 de les funcionalitats per part de GA. Incloure un dashboard per cadascuna de les empreses de manteniment

Fer el Dashboard per a 8 empreses instal\u00b7ladores. Haur\u00edem de passar a Superset que t\u00e9 una millor gesti\u00f3 d'usuaris?

"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#fita-5-dashboard-ineco","title":"Fita 5: Dashboard INECO","text":"

Prioritat 7

Per definir

"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#fita-6-noves-queries","title":"Fita 6: Noves queries","text":"

Prioritat 8

"},{"location":"projecte/2023-06-03-macrofase%20roadmap/#fita-7-dashboard-gurb","title":"Fita 7: Dashboard GURB","text":"

Prioritat 9

"},{"location":"projecte/2023-06-03-overview/","title":"Overview del projecte per a usuaris","text":"

Descripci\u00f3 del projecte des del punt de vista de l'usuari

"},{"location":"projecte/2023-06-03-overview/#context-de-naixement","title":"Context de naixement","text":""},{"location":"projecte/2023-06-03-overview/#peces","title":"Peces","text":""}]} \ No newline at end of file