Feature COMMANDLOG to record slow execution and large request/reply

[soloestoy]

As discussed in PR #336.

We have different types of resources like CPU, memory, network, etc. The slowlog can only record commands eat lots of CPU during the processing phase (doesn't include read/write network time), but can not record commands eat too many memory and network. For example:

1. run "SET key value(10 megabytes)" command would not be recored in slowlog, since when processing it the SET command only insert the value's pointer into db dict. But that command eats huge memory in query buffer and bandwidth from network. In this case, just 1000 tps can cause 10GB/s network flow.
2. run "GET key" command and the key's value length is 10 megabytes. The get command can eat huge memory in output buffer and bandwidth to network.

This PR introduces a new command COMMANDLOG, to log commands that consume significant network bandwidth, including both input and output. Users can retrieve the results using COMMANDLOG get <count> large-request and COMMANDLOG get <count> large-reply, all subcommands for COMMANDLOG are:

• COMMANDLOG HELP
• COMMANDLOG GET <count> <slow|large-request|large-reply>
• COMMANDLOG LEN <slow|large-request|large-reply>
• COMMANDLOG RESET <slow|large-request|large-reply>

And the slowlog is also incorporated into the commandlog.

For each of these three types, additional configs have been added for control:

• commandlog-request-larger-than and commandlog-large-request-max-len represent the threshold for large requests(the unit is Bytes) and the maximum number of commands that can be recorded.
• commandlog-reply-larger-than and commandlog-large-reply-max-len represent the threshold for large replies(the unit is Bytes) and the maximum number of commands that can be recorded.
• commandlog-execution-slower-than and commandlog-slow-execution-max-len represent the threshold for slow executions(the unit is microseconds) and the maximum number of commands that can be recorded.
  • Additionally, slowlog-log-slower-than and slowlog-max-len are now set as aliases for these two new configs.

[codecov]

Codecov Report

Attention: Patch coverage is 93.75000% with 8 lines in your changes missing coverage. Please review.

Project coverage is 70.93%. Comparing base (3032ccd) to head (63d1d88).
Report is 5 commits behind head on unstable.

Files with missing lines	Patch %	Lines
src/commandlog.c	94.87%	6 Missing ⚠️
src/module.c	0.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable    #1294      +/-   ##
============================================
+ Coverage     70.81%   70.93%   +0.12%     
============================================
  Files           121      121              
  Lines         65132    65162      +30     
============================================
+ Hits          46121    46225     +104     
+ Misses        19011    18937      -74     

Files with missing lines	Coverage Δ
src/blocked.c	91.38% <100.00%> (ø)
src/commands.def	100.00% <ø> (ø)
src/config.c	78.39% <ø> (ø)
src/latency.c	80.92% <100.00%> (ø)
src/multi.c	97.30% <ø> (ø)
src/networking.c	88.83% <ø> (+0.20%)	⬆️
src/server.c	87.60% <100.00%> (-0.03%)	⬇️
src/server.h	100.00% <ø> (ø)
src/module.c	9.59% <0.00%> (-0.01%)	⬇️
src/commandlog.c	94.87% <94.87%> (ø)

... and 9 files with indirect coverage changes

[hwware]

Just for Note: Let's first refresh the memory in pr #336, the last comment conclusion is:

After a core team meeting, we decided adding a new command COMMANDLOG with subcommands HEAVYTRAFFIC and SLOW, and then slowlog.c can be renamed to a common commandlog.c. #336 (comment)

[hwware]

From my understanding, for the command: commandlog get should be:
COMMANDLOG get count slow | heavytraffic-input | heavytraffic-output.

commandlog len should be:
COMMANDLOG len slow | heavytraffic-input | heavytraffic-output.

commandlog reset should be:
COMMANDLOG reset slow | heavytraffic-input | heavytraffic-output.

Can you describe the terms heavytraffic-input and heavytraffic-output in the json file (argument part) because I can only know
them from the source codes and valkey.conf so far?

And I think the existing slowlog commands should be deprecated? If yes, I think you should update the related json files as well.

[hwware]

LGTM, approved

[zuiderkwast]

@valkey-io/core-team Let's make a TSC decision on the command syntax and configs as described in the top comment.

Approve or vote 👍 or 👎.

[madolson]

Mostly LGTM, still agree with high level direction.

[enjoy-binbin]

overall LGTM, didn't review the code carefully in depth though.

[PingXie]

Thanks @soloestoy!

[soloestoy]

Merged! Thanks every one, the next work is adding "not-recommended" for command's spec.