catch and display disk usage issue in wis2box-api

[maaikelimper]

While debugging and supporting users I have noted that a lack of disk space results in the error:
elasticsearch.exceptions.TransportError: TransportError(429, 'cluster_block_exception', 'index [stations] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];')
which is raised inside the wis2box-api container.

This issue often goes unnoticed as the status of the wis2box-containers does not change and the error is not displayed in the Grafana Dashboard.

We should display this issue in the main Grafana dashboard and document it in the FAQ

[tomkralidis]

2024-07-24:

• @maaikelimper to check w/ @alimand on status/updates

[maaikelimper]

wis2box silently failing due the disk usage issue resulted in 5 days of data loss in the CMO-server, so I'm raising the priority of this issue.
I think instead of just displaying the disk usage issue we should consider disabling the wis2box as a whole when we detect that Elasticsearch stops functioning.

[tomkralidis]

2024-08-14:

• PR in Issue 674 #721 (needs to resolve conflicts)
• needs deploy/test/validate

[alimand]

Thanks for checking and now there is no conflicts with main branch

[maaikelimper]

I've tested and reviewed the fix for and merged this in #729

This fix adds a new panel to Grafana to display alerts and includes an alert triggered by the elasticsearch-exporter: