Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase rioConnsetWrite max chunk size to 16K #817

Merged
merged 6 commits into from
Jul 24, 2024

Conversation

ranshid
Copy link
Member

@ranshid ranshid commented Jul 23, 2024

Fixes #796

Currently rioConnWite uses 1024 bytes chunk when feeding the replication sockets on RDB write. This value seems too small and we will end up with high syscall overhead.
This PR sets the max chunk size to 16K.

Using a simple test program we did not observe any significant improvement in read/write times going with chunks bigger than 4K but that might be la bottleneck on network throughput. We did observe sweet point of CPU utilization at 16K when using TLS.

Test
test-program.zip
Include client/server implementations for client sending 100MB data to a server. the tests are both in TLS and TCP.
I run the tests on separate m5.xlarge EC2 instances on AWS.
the executables were compiled and run on ubuntu:

lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 24.04 LTS
Release:	24.04
Codename:	noble
uname -a
Linux ip-172-31-22-140 6.8.0-1009-aws #9-Ubuntu SMP Fri May 17 14:39:23 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

All files were compiled with O3 optimization level.

gcc --version
gcc (Ubuntu 13.2.0-23ubuntu4) 13.2.0

Results:

<style> </style>
Chunk Size write-time sec writes total write cpu-time (usr+sys) read-time sec read total syscalls read cpu-time (usr+sys)
1K 0.162946 102400 0.185916 0.168479 2447 0.026945
4K 0.163036 25600 0.122629 0.168627 715 0.023382
8K 0.163942 12800 0.121131 0.168887 704 0.039388
16K 0.163614 6400 0.104742 0.168202 2483 0.025574
64K 0.16279 1600 0.098792 0.168854 1068 0.046929
1K - TLS 0.32648 102400 0.366961 0.330785 102400 0.337377
4K - TLS 0.164296 25600 0.183326 0.169032 25600 0.129952
8K - TLS 0.163977 12800 0.163118 0.169484 12800 0.098432
16K - TLS 0.164861 6400 0.150666 0.169878 6383 0.094794
64K - TLS 0.163704 6400 0.156125 0.169323 6388 0.089971

this is in order to fix valkey-io#796

Signed-off-by: Ran Shidlansik <[email protected]>
Copy link

codecov bot commented Jul 23, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 70.37%. Comparing base (6eb19cf) to head (b898413).
Report is 7 commits behind head on unstable.

Additional details and impacted files
@@             Coverage Diff              @@
##           unstable     #817      +/-   ##
============================================
+ Coverage     70.32%   70.37%   +0.04%     
============================================
  Files           112      112              
  Lines         61269    61310      +41     
============================================
+ Hits          43089    43144      +55     
+ Misses        18180    18166      -14     
Files Coverage Δ
src/rio.c 84.18% <100.00%> (ø)
src/server.h 100.00% <ø> (ø)

... and 17 files with indirect coverage changes

@PingXie PingXie changed the title Increase rioConnsetWrite max chunk size to 8K Increase rioConnsetWrite max chunk size to 16K Jul 23, 2024
Copy link
Member

@madolson madolson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You reference fdset, which is the old version before we added the connection abstraction. I think this change makes sense though.

src/server.h Outdated Show resolved Hide resolved
src/server.h Outdated Show resolved Hide resolved
src/server.h Outdated Show resolved Hide resolved
ranshid and others added 3 commits July 24, 2024 07:49
Co-authored-by: Madelyn Olson <[email protected]>
Signed-off-by: ranshid <[email protected]>
Co-authored-by: Madelyn Olson <[email protected]>
Signed-off-by: ranshid <[email protected]>
Co-authored-by: Madelyn Olson <[email protected]>
Signed-off-by: ranshid <[email protected]>
src/rio.c Outdated Show resolved Hide resolved
Signed-off-by: Binbin <[email protected]>
@madolson madolson merged commit 5c073a5 into valkey-io:unstable Jul 24, 2024
47 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Update rioconnwrite size from https://github.com/valkey-io/valkey/pull/60
4 participants