Redis DB - please help to clear my understanding ! #291

M0hanrajp · 2024-07-16T10:15:38Z

M0hanrajp
Jul 16, 2024

Why is redis not storing anything for the data sent by device-virtual ?

Description

Message bus is MQTT, database is Redis, device virtual is auto configured to send data.
During such event in core-data:
- it was connected to data base (which must be redis per my understanding).
- On core data side log is printed as app=core-data source=event.go:64 msg="Event created on DB successfully. Event-id: e4a0319e-7324-4395-ba39-6a4f4dff2569, Correlation-id: "
I do not observe any data being stored in redis db ? so where did this data go ?
How and where was this data stored ? I have enabled logging level to trace but not able to grasp
on when and where was this data processed.

Core-data logs :

// Database connected
edgex-core-data  | level=INFO ts=2024-07-16T07:40:01.815540982Z app=core-data source=database.go:135 msg="Database connected"
// Connected to mqtt Message Bus @ tcp://edgex-mqtt-broker:1883
edgex-core-data  | level=INFO ts=2024-07-16T07:40:01.816346077Z app=core-data source=messaging.go:104 msg="Connected to mqtt Message Bus @ tcp://edgex-mqtt-broker:1883 with AuthMode='none"
...
// Random-Boolean-Device/Bool, Correlation-id: 93aa2620-3881-44bb-bf98-4409baca1cda
edgex-core-data  | level=DEBUG ts=2024-07-16T07:40:11.532770996Z app=core-data source=subscriber.go:66 msg="Event received from MessageBus. Topic: edgex/events/device/device-virtual/Random-Boolean-Device/Random-Boolean-Device/Bool, Correlation-id: 93aa2620-3881-44bb-bf98-4409baca1cda"
// Event-id: e4a0319e-7324-4395-ba39-6a4f4dff2569 , was created on DB.
edgex-core-data  | level=DEBUG ts=2024-07-16T07:40:11.535546161Z app=core-data source=event.go:64 msg="Event created on DB successfully. Event-id: e4a0319e-7324-4395-ba39-6a4f4dff2569, Correlation-id:  "

Redis DB logs:

mpunix@LIN-MP22QN1X:~/edgex-foundry/edgex-compose-3.1.1-mqtt$ docker compose logs -f database
edgex-redis  | 1:C 16 Jul 2024 07:39:48.885 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
edgex-redis  | 1:C 16 Jul 2024 07:39:48.885 # Redis version=7.0.15, bits=64, commit=00000000, modified=0, pid=1, just started
edgex-redis  | 1:C 16 Jul 2024 07:39:48.885 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
edgex-redis  | 1:M 16 Jul 2024 07:39:48.886 * monotonic clock: POSIX clock_gettime
edgex-redis  | 1:M 16 Jul 2024 07:39:48.890 * Running mode=standalone, port=6379.
edgex-redis  | 1:M 16 Jul 2024 07:39:48.890 # Server initialized
edgex-redis  | 1:M 16 Jul 2024 07:39:48.890 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
edgex-redis  | 1:M 16 Jul 2024 07:39:48.892 * Ready to accept connections
// I don't see any device data being stored here or having a log information regarding the device data.
^Ccanceled

Thanks for the help.

Answered by cloudxxx8

Jul 30, 2024

Yes, they are all read from the database

The APIs of count are:
/event/count/device/name/{name}
/reading/count/device/name/{name}

In your example, GET /api/v3/reading/count/device/name/Random-Integer-Device/Int32, which means the deviceName is Random-Integer-Device/Int32, but there is no device name called Random-Integer-Device/Int32
Please try GET /api/v3/reading/count/device/name/Random-Integer-Device

https://docs.edgexfoundry.org/3.1/microservices/core/data/ApiReference/

We don't have an API to count both device name plus source name as the condition

View full answer

M0hanrajp · 2024-07-22T17:51:28Z

M0hanrajp
Jul 22, 2024
Author

Please let me know if any further information is needed. Thanks

0 replies

cloudxxx8 · 2024-07-23T06:44:55Z

cloudxxx8
Jul 23, 2024
Maintainer

the data is stored in Redis. Please try the Get API from core-data or access the RedisDB with other Redis Client to confirm.
Redis log won't show anything.

1 reply

M0hanrajp Jul 23, 2024
Author

Thanks for your response I'll look into it. Thanks

M0hanrajp · 2024-07-25T19:56:00Z

M0hanrajp
Jul 25, 2024
Author

@cloudxxx8 , I accessed Redis cli from docker container & found the following:

~/edgex-foundry/edgex-compose-3.1.1-mqtt$ docker exec -it edgex-redis redis-cli
127.0.0.1:6379>keys *
  1) "cd|evt:readings:1e96f9d8-d37f-44b5-a7df-be740611b209"
...
...

127.0.0.1:6379> get "cd|evt:readings:1e96f9d8-d37f-44b5-a7df-be740611b209"
(error) WRONGTYPE Operation against a key holding the wrong kind of value
127.0.0.1:6379> type "cd|evt:readings:1e96f9d8-d37f-44b5-a7df-be740611b209"
zset
127.0.0.1:6379> ZRANGE "cd|evt:readings:1e96f9d8-d37f-44b5-a7df-be740611b209" 0 -1 WITHSCORES
1) "cd|rd:a3d13697-3700-41ba-9d4e-b698d6a7e7c4"
2) "0"
127.0.0.1:6379> get "cd|rd:a3d13697-3700-41ba-9d4e-b698d6a7e7c4"
"{\"Id\":\"a3d13697-3700-41ba-9d4e-b698d6a7e7c4\",\"Origin\":1721936563885058777,\"DeviceName\":\"Random-Integer-Device\",\"ResourceName\":\"Int8\",\"ProfileName\":\"Random-Integer-Device\",\"ValueType\":\"Int8\",\"Units\":\"\",\"Tags\":null,\"Value\":\"-84\"}"

Can you please validate if this is the way I can check if a small snapshot of a data can be found using keys *?

also I do not have much idea on checking with API from core-data (if possible then please do help)

Thanks

1 reply

cloudxxx8 Jul 26, 2024
Maintainer

I am not familiar with the Redis CLI
Why don't you just verify with the Get API of Core Data? This API read data from RedisDB.

M0hanrajp · 2024-07-26T15:15:12Z

M0hanrajp
Jul 26, 2024
Author

Hello @cloudxxx8 thanks for your response sorry I missed reading the API reference mentioned for core-data
in edgex foundry & I used the GET API and I think I'm able to find some data being returned.

Question a: so to confirm again this data is being read from the database right ?

Question b: But I found some irregularities in the readings is this expected ?

Event Count Discrepancy:

Why does the GET /api/v3/event/count/device/name/Random-Integer-Device/Int32 command return a count of 0 for the Random-Integer-Device/Int32 when the GET /api/v3/event/all command shows events recorded for this device?

Reading Count Discrepancy:

Why does the GET /api/v3/reading/count/device/name/Random-Integer-Device/Int32 command return a count of 0 for the Random-Integer-Device/Int32 when the GET /api/v3/reading/all command shows readings recorded for this device?

Supporting logs:

# Random-Integer-Device GET command returns 613 event counts
~/edgex-foundry/edgex-compose-3.1.1-mqtt$ curl -X GET http://localhost:59880/api/v3/event/all | json_pp
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  8468    0  8468    0     0   520k      0 --:--:-- --:--:-- --:--:--  551k
...
      {
         "apiVersion" : "v3",
         "deviceName" : "Random-Integer-Device",
         "id" : "35649f92-270d-4f25-955b-0dbd53aa611f",
         "origin" : 1721984337644178435,
         "profileName" : "Random-Integer-Device",
         "readings" : [
            {
               "deviceName" : "Random-Integer-Device",
               "id" : "b84915db-48b1-4b00-9976-dfaa4ba31f81",
               "origin" : 1721984337644178435,
               "profileName" : "Random-Integer-Device",
               "resourceName" : "Int32",
               "value" : "-653362182",
               "valueType" : "Int32"
            }
   "statusCode" : 200,
   "totalCount" : 613
}
# Total reading count of all devices was 331
~/edgex-foundry/edgex-compose-3.1.1-mqtt$ curl -X GET http://localhost:59880/api/v3/reading/all | json_pp
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4440    0  4440    0     0   476k      0 --:--:-- --:--:-- --:--:--  541k
...
      {
         "deviceName" : "Random-Integer-Device",
         "id" : "ad5172a3-9998-4e42-94b5-9f60ee968d97",
         "origin" : 1721983887618537952,
         "profileName" : "Random-Integer-Device",
         "resourceName" : "Int32",
         "value" : "866996316",
         "valueType" : "Int32"
      },
...
...
   "statusCode" : 200,
   "totalCount" : 331
}
# Why event count is 0 for Random-Integer-Device/Int32 ? when we already see the event recorded by GET command for event/all performed above. 
~/edgex-foundry/edgex-compose-3.1.1-mqtt$ curl -X GET http://localhost:59880/api/v3/event/count/device/name/Random-Integer-Device/Int32 | json_pp
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    47  100    47    0     0  17550      0 --:--:-- --:--:-- --:--:-- 23500
{
   "Count" : 0,
   "apiVersion" : "v3",
   "statusCode" : 200
}
# Why event count is 0 for Random-Integer-Device/Int32 ? when we already see the event recorded by GET command for reading/all performed above. 
~/edgex-foundry/edgex-compose-3.1.1-mqtt$ curl -X GET http://localhost:59880/api/v3/reading/count/device/name/Random-Integer-Device/Int32 | json_pp
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    47  100    47    0     0  14063      0 --:--:-- --:--:-- --:--:-- 23500
{
   "Count" : 0,
   "apiVersion" : "v3",
   "statusCode" : 200
}

Thanks

1 reply

cloudxxx8 Jul 30, 2024
Maintainer

Yes, they are all read from the database

The APIs of count are:
/event/count/device/name/{name}
/reading/count/device/name/{name}

In your example, GET /api/v3/reading/count/device/name/Random-Integer-Device/Int32, which means the deviceName is Random-Integer-Device/Int32, but there is no device name called Random-Integer-Device/Int32
Please try GET /api/v3/reading/count/device/name/Random-Integer-Device

https://docs.edgexfoundry.org/3.1/microservices/core/data/ApiReference/

We don't have an API to count both device name plus source name as the condition

Answer selected by M0hanrajp

M0hanrajp · 2024-07-30T07:44:26Z

M0hanrajp
Jul 30, 2024
Author

@cloudxxx8 Thanks for your response.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EdgeX Foundry Project

Redis DB - please help to clear my understanding ! #291

{{title}}

Replies: 5 comments 3 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

EdgeX Foundry Project

Redis DB - please help to clear my understanding ! #291

M0hanrajp Jul 16, 2024

Why is redis not storing anything for the data sent by device-virtual ?

Replies: 5 comments · 3 replies

M0hanrajp Jul 22, 2024 Author

cloudxxx8 Jul 23, 2024 Maintainer

M0hanrajp Jul 23, 2024 Author

M0hanrajp Jul 25, 2024 Author

cloudxxx8 Jul 26, 2024 Maintainer

M0hanrajp Jul 26, 2024 Author

cloudxxx8 Jul 30, 2024 Maintainer

M0hanrajp Jul 30, 2024 Author

M0hanrajp
Jul 16, 2024

Replies: 5 comments 3 replies

M0hanrajp
Jul 22, 2024
Author

cloudxxx8
Jul 23, 2024
Maintainer

M0hanrajp Jul 23, 2024
Author

M0hanrajp
Jul 25, 2024
Author

cloudxxx8 Jul 26, 2024
Maintainer

M0hanrajp
Jul 26, 2024
Author

cloudxxx8 Jul 30, 2024
Maintainer

M0hanrajp
Jul 30, 2024
Author