Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] When Server's io-threads > 1 and large data is transferred over RDMA, the server gets stuck in an endless loop #1385

Open
fengquyoumo opened this issue Dec 3, 2024 · 0 comments · May be fixed by #1386

Comments

@fengquyoumo
Copy link

Describe the bug

When Server's io-threads > 1 and large data is transferred over RDMA, the server gets stuck in an endless loop

To reproduce

Run Server

./src/valkey-server \
    --port 6379 \
    --protected-mode no \
    --appendonly no \
    --loglevel notice \
    --loadmodule src/valkey-rdma.so bind=<your host> port=6380 \
    --server_cpulist 0-31:2 \
    --bgsave_cpulist 96-127:2 \
    --io-threads 8

Run redis-benchmark with rdma (provided by @pizhenwei)
You may need to modify the code before running it

diff --git a/src/redis-benchmark.c b/src/redis-benchmark.c
index a77565526..390d33b6b 100644
--- a/src/redis-benchmark.c
+++ b/src/redis-benchmark.c
@@ -636,6 +636,9 @@ static void writeHandler(aeEventLoop *el, int fd, void *privdata, int mask) {
                         fprintf(stderr, "Error writing to the server: %s\n", strerror(errno));
                     freeClient(c);
                     return;
+                } else if (nwritten > 0) {
+                    c->written += nwritten;
+                    return;
                 }
             } else {
                 aeDeleteFileEvent(el,c->context->fd,AE_WRITABLE);

and then run

./src/redis-benchmark \
    -h <your host> \
    -n 10000000 \
    -r 100 \
    -t set \
    -p 6380 \
    -d 10485760  \
    -c 50 \
    --threads 16 \
    --rdma

Expected behavior

Get a benchmark report as usual

Actual behavior

Main thread of Server run 100% cpu, and redis-benchmark waits

Additional information

I am a newcomer, please bear with me for my English level and description ability

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant