Redis ZCARD member limit problem

[Yangsta911]

Hello,

I have currently deployed a native copy of Edgex V3.1 onto an arm64 embedded board running Linux 4.19. I am doing some load testing on the device by adding as many of the official virtual devices simulators as possible to see how much the board and Edgex can handle. The problem I am facing is that at around 200-250 device services with 5 device endpoints each added into the edgex UI server I get error message: code: 500 , message: failed to get member number with command ZCARD from ss|iv and the entire UI server crashes. I am wondering if this is a hard limit set by the redis client or is there a way to resolve this issue through configuration files?

Looking at the hardware stats such as cpu, memory and storage usage of the embedded board it does not seem like that is the bottle neck for performance causing this error. I have attached the error messages received as well as the hardware statistics below.

[cloudxxx8]

you may use docker stats to check the container status

[Yangsta911]

Sorry, for not making it clear in the initial post, but I am running edgex natively without docker is there a similar way of checking the status?

[qs-wang]

I guess the redis is down or time out.
Look at this issue edgexfoundry/edgex-go#4463 and the code at https://github.com/edgexfoundry/edgex-go/blob/17930b14352cbfb7930b1ac1271c629a4bf190e4/internal/pkg/infrastructure/redis/queries.go#L199.

I guess if the redis is not response some how by consider you're running loading testing, the function call failed with the ZCARD error. I would suggest you to trace the code around the function getMemberNumber at https://github.com/edgexfoundry/edgex-go/blob/17930b14352cbfb7930b1ac1271c629a4bf190e4/internal/pkg/infrastructure/redis/queries.go#L215 to see if there's bottle neck.

[Yangsta911]

By reading the redis logs I was able to find that the problem was that redis had stored too many devices and used up all the memory that it had been allocated. This can be solved by setting the retention flags on edgex to periodically clear the stored data points.