From 25dd943087d0f2bdbea76edd059cf5805d4db7e2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Viktor=20S=C3=B6derqvist?= Date: Wed, 28 Aug 2024 21:17:04 +0200 Subject: [PATCH] Delete TLS.md and update README.md about tests (#960) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Most of the content of TLS.md has already been copied to README.md in #927. The description of how to run tests with TLS is moved to tests/README.md. Descriptions of the additional scripts runtest-cluster, runtest-sentinel and runtest-module are added in tests/README.md. Links to tests/README.md and src/unit/README.md are added in the top-level README.md along with a brief overview of the `make test-*` commands. Signed-off-by: Viktor Söderqvist --- README.md | 12 ++++-- TLS.md | 106 ------------------------------------------------ tests/README.md | 26 ++++++++++-- 3 files changed, 31 insertions(+), 113 deletions(-) delete mode 100644 TLS.md diff --git a/README.md b/README.md index 4c7c9172b1..bce092fceb 100644 --- a/README.md +++ b/README.md @@ -60,12 +60,16 @@ After building Valkey, it is a good idea to test it using: % make test -If TLS is built, running the tests with TLS enabled (you will need `tcl-tls` -installed): +The above runs the main integration tests. Additional tests are started using: - % ./utils/gen-test-certs.sh - % ./runtest --tls + % make test-unit # Unit tests + % make test-modules # Tests of the module API + % make test-sentinel # Valkey Sentinel integration tests + % make test-cluster # Valkey Cluster integration tests +More about running the integration tests can be found in +[tests/README.md](tests/README.md) and for unit tests, see +[src/unit/README.md](src/unit/README.md). Fixing build problems with dependencies or cached build options --------- diff --git a/TLS.md b/TLS.md deleted file mode 100644 index 4ac61f406b..0000000000 --- a/TLS.md +++ /dev/null @@ -1,106 +0,0 @@ -TLS Support -=========== - -Getting Started ---------------- - -### Building - -To build with TLS support you'll need OpenSSL development libraries (e.g. -libssl-dev on Debian/Ubuntu). - -To build TLS support as Valkey built-in: -Run `make BUILD_TLS=yes`. - -Or to build TLS as Valkey module: -Run `make BUILD_TLS=module`. - -Note that sentinel mode does not support TLS module. - -### Tests - -To run Valkey test suite with TLS, you'll need TLS support for TCL (i.e. -`tcl-tls` package on Debian/Ubuntu). - -1. Run `./utils/gen-test-certs.sh` to generate a root CA and a server - certificate. - -2. Run `./runtest --tls` or `./runtest-cluster --tls` to run Valkey and Valkey - Cluster tests in TLS mode. - -3. Run `./runtest --tls-module` or `./runtest-cluster --tls-module` to - run Valkey and Valkey cluster tests in TLS mode with Valkey module. - -### Running manually - -To manually run a Valkey server with TLS mode (assuming `gen-test-certs.sh` was -invoked so sample certificates/keys are available): - -For TLS built-in mode: - - ./src/valkey-server --tls-port 6379 --port 0 \ - --tls-cert-file ./tests/tls/valkey.crt \ - --tls-key-file ./tests/tls/valkey.key \ - --tls-ca-cert-file ./tests/tls/ca.crt - -For TLS module mode: - - ./src/valkey-server --tls-port 6379 --port 0 \ - --tls-cert-file ./tests/tls/valkey.crt \ - --tls-key-file ./tests/tls/valkey.key \ - --tls-ca-cert-file ./tests/tls/ca.crt \ - --loadmodule src/valkey-tls.so - -To connect to this Valkey server with `valkey-cli`: - - ./src/valkey-cli --tls \ - --cert ./tests/tls/valkey.crt \ - --key ./tests/tls/valkey.key \ - --cacert ./tests/tls/ca.crt - -Specifying `port 0` will disable TCP. It's also possible to have -both TCP and TLS available, but you'll need to assign different ports. - -To make a Replica connect to the master using TLS, use `--tls-replication yes`, -and to make Valkey Cluster use TLS across nodes use `--tls-cluster yes`. - -Connections ------------ - -All socket operations now go through a connection abstraction layer that hides -I/O and read/write event handling from the caller. - -**Multi-threading I/O is not currently supported for TLS**, as a TLS connection -needs to do its own manipulation of AE events which is not thread safe. The -solution is probably to manage independent AE loops for I/O threads and longer -term association of connections with threads. This may potentially improve -overall performance as well. - -Sync IO for TLS is currently implemented in a hackish way, i.e. making the -socket blocking and configuring socket-level timeout. This means the timeout -value may not be so accurate, and there would be a lot of syscall overhead. -However I believe that getting rid of syncio completely in favor of pure async -work is probably a better move than trying to fix that. For replication it would -probably not be so hard. For cluster keys migration it might be more difficult, -but there are probably other good reasons to improve that part anyway. - -To-Do List ----------- - -- [ ] valkey-benchmark support. The current implementation is a mix of using - hiredis for parsing and basic networking (establishing connections), but - directly manipulating sockets for most actions. This will need to be cleaned - up for proper TLS support. The best approach is probably to migrate to hiredis - async mode. -- [ ] valkey-cli `--slave` and `--rdb` support. - -Multi-port ----------- - -Consider the implications of allowing TLS to be configured on a separate port, -making Valkey listening on multiple ports: - -1. Startup banner port notification -2. Proctitle -3. How slaves announce themselves -4. Cluster bus port calculation diff --git a/tests/README.md b/tests/README.md index efe936aa5b..06756f394b 100644 --- a/tests/README.md +++ b/tests/README.md @@ -21,7 +21,8 @@ enabled using the `--host` and `--port` parameters. When executing against an external server, tests tagged `external:skip` are skipped. There are additional runtime options that can further adjust the test suite to -match different external server configurations: +match different external server configurations. All options are listed by +`./runtest --help`. The following table is just a subset of the options: | Option | Impact | | -------------------- | -------------------------------------------------------- | @@ -29,7 +30,26 @@ match different external server configurations: | `--ignore-encoding` | Skip all checks for specific encoding. | | `--ignore-digest` | Skip key value digest validations. | | `--cluster-mode` | Run in strict Valkey Cluster compatibility mode. | -| `--large-memory` | Enables tests that consume more than 100mb | +| `--large-memory` | Enables tests that consume more than 100MB | +| `--tls` | Run tests with TLS. See below. | +| `--tls-module` | Run tests with TLS, when TLS support is built as a module. | +| `--help` | Displays the full set of options. | + +Running with TLS requires the following preparations: + +* Build Valkey is TLS support, e.g. using `make BUILD_TLS=yes`, or `make BUILD_TLS=module`. +* Run `./utils/gen-test-certs.sh` to generate a root CA and a server certificate. +* Install TLS support for TCL, e.g. the `tcl-tls` package on Debian/Ubuntu. + +Additional tests +---------------- + +Not all tests are included in the `./runtest` scripts. Some additional entry points are provided by the following scripts, which support a subset of the options listed above: + +* `./runtest-cluster` runs more extensive tests for Valkey Cluster. + Some cluster tests are included in `./runtest`, but not all. +* `./runtest-sentinel` runs tests of Valkey Sentinel. +* `./runtests-module` runs tests of the module API. Debugging --------- @@ -69,7 +89,7 @@ The following compatibility and capability tags are currently used: | --------------------- | --------- | | `external:skip` | Not compatible with external servers. | | `cluster:skip` | Not compatible with `--cluster-mode`. | -| `large-memory` | Test that requires more than 100mb | +| `large-memory` | Test that requires more than 100MB | | `tls:skip` | Not compatible with `--tls`. | | `needs:repl` | Uses replication and needs to be able to `SYNC` from server. | | `needs:debug` | Uses the `DEBUG` command or other debugging focused commands (like `OBJECT REFCOUNT`). |