Issue 674 #721

alimand · 2024-08-05T14:08:47Z

Fix issue 674 and issue 683:
Adding new dashboard in Grafana to monitor the status of elasticsearch health by using metrics:

elasticsearch_indices_settings_stats_read_only_indices(add alert)
elasticsearch_filesystem_data_available_bytes
elasticsearch_filesystem_data_free_bytes
Adding disk space usage status moniring and alert in new dashboard in Grafana
Optimize container relationship in wis2box to make sure when machine restart wis2box could bahave well in 1-2 minutes.

… status of elasticsearch and disk usage fix issue 674 and 683

maaikelimper · 2024-08-05T14:12:57Z

docker-compose.monitoring.yml

@@ -108,6 +125,14 @@ services:
    <<: *logging
  elasticsearch:
    <<: *logging
+    container_name: elasticsearch


why did you add another "elasticsearch"-services into the docker-compose.monitoring.yml ? I think this is a mistake and should be removed

Yes, thats an error! I removed it

maaikelimper

Hi Peiwen, I see several changes in the code that I think are related to local development and should be removed. Could you please add a screenshot of the new dashboard that you made ?

maaikelimper · 2024-08-05T14:13:54Z

docker-compose.yml

@@ -86,6 +87,8 @@ services:
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - cluster.name=es-wis2box
      - xpack.security.enabled=false
+    ports:
+      - "9200:9200"


This additional port-mapping should not be required, please remove it

maaikelimper · 2024-08-05T14:14:41Z

docker-compose.yml

@@ -147,6 +150,8 @@ services:

  wis2downloader:
    container_name: wis2downloader
+    image: wis2box_project-wis2downloader


why did you define the image name to the downloader image ? should be removed

maaikelimper · 2024-08-05T14:15:30Z

prometheus/prometheus.yml

+
+- job_name: 'prometheus'
+  static_configs:
+  - targets: ['localhost:9090']


what does this job do ?

I removed this part it's useless

maaikelimper · 2024-08-05T14:15:41Z

prometheus/prometheus.yml

+  static_configs:
+  - targets: ['localhost:9090']
+
+- job_name: 'node_exporter'


remove the node_exporter for now

maaikelimper · 2024-08-05T14:16:09Z

docker-compose.monitoring.yml

-        ipv4_address: 10.5.0.2
-      default:
+
+  node_exporter:


remove the node-exporter for now

alimand · 2024-08-05T14:24:41Z

Hi Peiwen, I see several changes in the code that I think are related to local development and should be removed. Could you please add a screenshot of the new dashboard that you made ?

Yes, sure Maiike. Thanks for your correctness, let me change this part!

alimand · 2024-08-05T14:31:33Z

Hi Peiwen, I see several changes in the code that I think are related to local development and should be removed. Could you please add a screenshot of the new dashboard that you made ?

Of course, here is the screenshot of the new dashboard in Grafana

maaikelimper · 2024-08-05T14:48:52Z

Hi Peiwen, thank your for the screenshot.

I think you can improve this dashboard for the wis2box-users:

"alarmtracking dashboard" -> this is the title of the dashboard but you named the json file that defines it "metricsmonitoring.json" please be consistent and think of a proper title and filename , you should mention "Elasticsearch" if the metrics are based on Elasticsearch

"disk space usage" -> what is the unit ? the Y-axis shows 3 times 27 which is confusing , can you please improve this

"count of indices that have read_only_allow_delete=true" -> it is sufficient to display the current value for this metric, anything above 0 should be displayed in red

What is the difference between "available space" and "free space" ? Are they both required for the user ? Please display bytes using a human-friendly unit (Grafana has built-in option to auto-convert bytes into kB/MB)

the screenshots shows a range of "last 5 minutes" please define a sensible default for the default time-range

finally think about the usage of colors in your dashboard, for example you can set thresholds in Grafana to make a bad value display as red, while good values can be displayed in green

alimand · 2024-08-05T14:53:25Z

Hi Peiwen, thank your for the screenshot.

I think you can improve this dashboard for the wis2box-users:

"alarmtracking dashboard" -> this is the title of the dashboard but you named the json file that defines it "metricsmonitoring.json" please be consistent and think of a proper title and filename , you should mention "Elasticsearch" if the metrics are based on Elasticsearch

"disk space usage" -> what is the unit ? the Y-axis shows 3 times 27 which is confusing , can you please improve this

"count of indices that have read_only_allow_delete=true" -> it is sufficient to display the current value for this metric, anything above 0 should be displayed in red

What is the difference between "available space" and "free space" ? Are they both required for the user ? Please display bytes using a human-friendly unit (Grafana has built-in option to auto-convert bytes into kB/MB)

the screenshots shows a range of "last 5 minutes" please define a sensible default for the default time-range

finally think about the usage of colors in your dashboard, for example you can set thresholds in Grafana to make a bad value display as red, while good values can be displayed in green

Sure, thanks for your review Maaike, let me impove it based on the comments. Will provide new version soon.

…y to user's personal emails/friendly dashboard to users

alimand · 2024-08-11T21:04:01Z

Some updates based on suggestions and optimize:
Modify all the sugguestions part;
At each top of the dashboard, the alert list will be shown (only the alerting one)
When alert is triggered, notifications could be sent to personal emails
Test Showcase:

alimand · 2024-08-11T21:12:18Z

Current Alerts Overview:
Purpose: Displays a list of currently active alerts.

Elasticsearch Disk Space Utilization:
Purpose: This panel shows the percentage of disk space used on the Elasticsearch node. It is an important metric to monitor the available disk space and ensure that the Elasticsearch cluster has sufficient capacity.
Alert Condition: Triggers if disk usage exceeds 80% for 5 minutes.

Available Disk Space (GB):
Purpose: This panel shows the amount of free disk space available on the block device where Elasticsearch stores its data. Monitoring this helps prevent the disk from running out of space.
Alert Condition: Triggers if free space falls below 10 GB for 5 minutes.

Elasticsearch Disk Usage Thresholds:
Purpose: This panel shows the high and low disk watermarks configured for Elasticsearch in kilobytes (KB). These watermarks determine when Elasticsearch should start taking action to prevent nodes from running out of disk space, such as stopping new data allocations.
Disk Read and Write Activity:
Purpose: This panel displays the rate of filesystem read and write operations per second. It helps to monitor how frequently data is being read from or written to the disk, which can be indicative of disk performance or potential bottlenecks.

Read-Only Indices Count:
Purpose: This setting is often used to prevent further writes to indices that are running out of disk space.
Alert Condition: Triggers if any read-only indices are detected (count > 0).

make sure there is no conflict with main branch

remove the whitespace

alimand · 2024-08-16T09:48:13Z

updates shown in issue-674-new branch

pjjin-design added 4 commits August 5, 2024 15:23

add metrics-monitoring dashboard and alarm to help monitor the health…

f0d4145

… status of elasticsearch and disk usage fix issue 674 and 683

remove node-exporter in docker-comnpose and add it in monitoring yml

2c250cd

move disk space monitoring to new dashboard

c382ee5

try to fix integrate testing bugs

7aa3326

alimand requested a review from maaikelimper August 5, 2024 14:08

maaikelimper reviewed Aug 5, 2024

View reviewed changes

maaikelimper requested changes Aug 5, 2024

View reviewed changes

add alerting list and when alert is triggered could configure manuall…

9163044

…y to user's personal emails/friendly dashboard to users

alimand requested a review from maaikelimper August 11, 2024 21:04

tomkralidis mentioned this pull request Aug 14, 2024

catch and display disk usage issue in wis2box-api #674

Closed

alimand added 2 commits August 14, 2024 13:58

Update docker-compose.yml

83dd4c3

make sure there is no conflict with main branch

Update docker-compose.yml

a171273

remove the whitespace

alimand closed this Aug 16, 2024

tomkralidis deleted the issue-674 branch August 16, 2024 11:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue 674 #721

Issue 674 #721

alimand commented Aug 5, 2024

maaikelimper Aug 5, 2024

alimand Aug 5, 2024

maaikelimper left a comment

maaikelimper Aug 5, 2024

maaikelimper Aug 5, 2024

maaikelimper Aug 5, 2024

alimand Aug 5, 2024

maaikelimper Aug 5, 2024

maaikelimper Aug 5, 2024

alimand commented Aug 5, 2024

alimand commented Aug 5, 2024

maaikelimper commented Aug 5, 2024

alimand commented Aug 5, 2024

alimand commented Aug 11, 2024

alimand commented Aug 11, 2024

alimand commented Aug 16, 2024

Issue 674 #721

Issue 674 #721

Conversation

alimand commented Aug 5, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maaikelimper left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alimand commented Aug 5, 2024

alimand commented Aug 5, 2024

maaikelimper commented Aug 5, 2024

alimand commented Aug 5, 2024

alimand commented Aug 11, 2024

alimand commented Aug 11, 2024

alimand commented Aug 16, 2024