From 77aef929363e5e0d1203a1db41d9ea9817e3133e Mon Sep 17 00:00:00 2001 From: mbshields Date: Mon, 13 May 2024 10:32:31 -0700 Subject: [PATCH] docs: add info to scaleout article Signed-off-by: mbshields --- docs/articles/clustering.md | 57 +++---------- docs/articles/scaleout.md | 154 ++++++++++++++++++++++++++++++++++++ 2 files changed, 163 insertions(+), 48 deletions(-) create mode 100644 docs/articles/scaleout.md diff --git a/docs/articles/clustering.md b/docs/articles/clustering.md index 60d3351..98631df 100644 --- a/docs/articles/clustering.md +++ b/docs/articles/clustering.md @@ -14,7 +14,9 @@ service to remain available even if a few replicas fail or become unavailable. Load balancing across many zot replicas can also increase aggregate network throughput. -![504569](../assets/images/504569.jpg){width="500"}
Figure 1: a zot cluster with load balancing
+![504569](../assets/images/504569.jpg){width="500"} + +> :pencil2: Beginning with zot release v2.1.0, you can design a highly scalable cluster that does not require configuring the load balancer to direct repository queries to specific zot instances within the cluster. See [Easy scaling of a zot cluster](scaleout.md). This is the preferred method if you are running v2.1.0 or later. Clustering is supported in both bare-metal and Kubernetes environments. > :pencil2: @@ -25,11 +27,11 @@ Clustering is supported in both bare-metal and Kubernetes environments. ### Prerequisites -- A highly-available load balancer such as HAProxy configured to direct traffic to zot replicas. +- A highly-available load balancer such as HAProxy configured to direct traffic to zot replicas -- Multiple zot replicas as `systemd` services hosted on multiple hosts or VMs. +- Multiple zot replicas as `systemd` services hosted on multiple hosts or VMs -- AWS S3 API-compatible remote backend storage. +- AWS S3 API-compatible remote backend storage ## Kubernetes deployment @@ -37,16 +39,16 @@ Clustering is supported in both bare-metal and Kubernetes environments. - A zot Kubernetes [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) - with required number of replicas. + with required number of replicas - AWS S3 API-compatible remote backend storage. - A zot Kubernetes - [Service](https://kubernetes.io/docs/concepts/services-networking/service/). + [Service](https://kubernetes.io/docs/concepts/services-networking/service/) - A zot Kubernetes [Ingress Gateway](https://kubernetes.io/docs/concepts/services-networking/ingress/) - if the service needs to be exposed outside. + if the service needs to be exposed outside ## Implementing stateless zot @@ -170,44 +172,3 @@ backend zot-cluster ``` - - -## Easy scaling of the cluster - -You can design a cluster (see [Figure 1](#figure1)) in which the number of replicas can easily be expanded (or reduced) with no programming of the load balancer other than adding the IP addresses of the new replicas. The shared storage can also be easily increased or decreased. - -### Prerequisites - -For easy scaling of replicas, the following conditions must be met: - -- All zot replicas must be running zot release v2.0.4 (or later) with identical configurations. -- All zot replicas in the cluster use remote storage at a single shared S3 backend. There is no local caching in the zot replicas. -- Each repo is served by one zot replica, and that replica is solely responsible for serving all images of that repo. -- A repo in storage can be written to only by the zot replica associated with that repo. -- Each zot replica in the cluster has its own IP address, but all replicas use the port number. -- The URI format sent to the load balancer must be /v2//: - -### How it works - -A highly available and scalable cluster can be architected by sharding on the repository name. In the cluster, each replica is the owner of a small subset of the repository. The load balancer does not need to know which replica owns which repo. The replicas themselves can determine this. - -When the load balancer receives an image request, it sends the request to any replica in the cluster. The receiving replica hashes the repo path and consults a hash table in shared storage to determine which replica is responsible for the repo. The receiving replica forwards the request to the responsible replica and then acts as a proxy, returning the requested image to the requestor. - -### Cluster member configuration - -Each replica must have a list of its peers and must have a hash key for hashing the repo path of the image request. The following is an example of the cluster configuration in each replica: - -```json - "cluster": { - "members": [ - "127.0.0.1:9000", - "127.0.0.2:9000", - "127.0.0.3:9000" - ], - "hashKey": "loremipsumdolors" - } - -``` -## CVE repository in a zot cluster environment - -In the clustering scheme described in this article, CVE scanning is disabled. In this case, we recommend implementing a CVE repository with a zot instance outside of the cluster using a local disk for storage and Trivy as the detection engine. \ No newline at end of file diff --git a/docs/articles/scaleout.md b/docs/articles/scaleout.md new file mode 100644 index 0000000..9102ef3 --- /dev/null +++ b/docs/articles/scaleout.md @@ -0,0 +1,154 @@ +# Easy scaling of a zot cluster + +> :point_right: A cluster of zot replicas can easily be scaled with no repo-specific programming of the load balancer using: +> +> - Stateless zot instances to simplify scale out +> - Shared remote storage +> - zot release v2.1.0 or later + +Beginning with zot release v2.1.0, a new "scale-out" architecture greatly reduces the configuration required when deploying large numbers of zot instances. As before, multiple identical zot replicas run simultaneously using the same shared reliable storage, but with improved scale and performance in large deployments. + +![504569](../assets/images/504569.jpg){width="500"} + +The number of replicas can easily be expanded by simply adding the IP addresses of the new replicas in the load balancer configuration. No repo-specific programming of the load balancer is needed. The shared storage can also be easily increased or decreased. + +> :pencil2: For high availability clustering with earlier zot releases, see [zot Clustering](clustering.md). + +## Prerequisites + +For easy scaling of replicas, the following conditions must be met: + +- All zot replicas must be running zot release v2.1.0 (or later) with identical configurations. +- All zot replicas in the cluster use remote storage at a single shared S3 backend. There is no local caching in the zot replicas. +- Each repo is served by one zot replica, and that replica is solely responsible for serving all images of that repo. +- A repo in storage can be written to only by the zot replica associated with that repo. +- Each zot replica in the cluster has its own IP address, but all replicas use the port number. +- The URI format sent to the load balancer must be /v2//: + +Beginning with zot release v2.1.0, garbage collection is allowed in the shared cluster storage. + +## How it works + +A highly scalable cluster can be architected by sharding on the repository name. In the cluster, each replica is the owner of a small subset of the repository. The load balancer does not need to know which replica owns which repo. The replicas themselves can determine this. + +When the load balancer receives an image push or pull request, it forwards the request to any replica in the cluster. The receiving replica hashes the repo path and consults a hash table in shared storage to determine which replica is responsible for the repo. The receiving replica forwards the request to the responsible replica and then acts as a proxy, returning the requested image to the requestor. + +When an image push request is received but no responsible replica is found for the requested repo, the receiving replica becomes the responsible replica and updates the hash table. + +## Configuration examples + +Clustering is supported by using multiple stateless zot replicas with shared S3 storage and an HAProxy (with sticky session) load balancing traffic to the replicas. + +### Cluster member configuration + +In the replica configuration, each replica must have a list of its peers. The replica must also have a hash key for hashing the repo path of the image request and a TLS certificate for authenticating with its peers. + +
+ Click here to view a sample cluster configuration for each replica. See the "cluster" section in the JSON structure. + +```json +{ + "distSpecVersion": "1.1.0", + "storage": { + "rootDirectory": "/tmp/zot", + "dedupe": false, + "remoteCache": true, + "storageDriver": { + "name": "s3", + "rootdirectory": "/zot", + "region": "us-east-1", + "regionendpoint": "localhost:4566", + "bucket": "zot-storage", + "secure": false, + "skipverify": false + }, + "cacheDriver": { + "name": "dynamodb", + "endpoint": "http://localhost:4566", + "region": "us-east-1", + "cacheTablename": "ZotBlobTable", + "repoMetaTablename": "ZotRepoMetadataTable", + "imageMetaTablename": "ZotImageMetaTable", + "repoBlobsInfoTablename": "ZotRepoBlobsInfoTable", + "userDataTablename": "ZotUserDataTable", + "versionTablename": "ZotVersion", + "apiKeyTablename": "ZotApiKeyTable" + } + }, + "http": { + "address": "127.0.0.1", + "port": "9001", + "tls": { + "cert": "test/data/server.cert", + "key": "test/data/server.key" + } + }, + "log": { + "level": "debug" + }, + "cluster": { + "members": [ + "127.0.0.1:9000", + "127.0.0.2:9000", + "127.0.0.3:9000" + ], + "hashKey": "loremipsumdolors", + "tls": { + "cacert": "test/data/ca.crt" + } + } +} +``` + +
+ +### HAProxy YAML configuration + +The HAProxy load balancer uses a simple round-robin balancing scheme and delivers a cookie to the requestor to maintain a sticky session connection to the assigned replica. + +
+ Click here to view a sample HAProxy configuration. + +```yaml + +global + log /dev/log local0 + log /dev/log local1 notice + chroot /var/lib/haproxy + maxconn 2000 + stats timeout 30s + +defaults + log global + mode tcp + option tcplog + option dontlognull + timeout connect 5000 + timeout client 50000 + timeout server 50000 + +frontend zot + bind *:8080 + default_backend zot-cluster + +backend zot-cluster + mode http + balance roundrobin + cookie SERVER insert indirect nocache + server zot0 127.0.0.1:9000 check cookie zot0 + server zot1 127.0.0.2:9000 check cookie zot1 + server zot2 127.0.0.3:9000 check cookie zot2 + +``` + +
+ +## When a replica fails + +Unlike the earlier [simple clustering scheme](clustering.md), the scale-out scheme described in this article is not self-healing when a replica fails. In case of a replica failure, you must bring down the cluster, repair the failed replica, and reestablish the cluster. + +With an HAProxy load balancer, we recommend implementing an [HAProxy circuit breaker](https://www.haproxy.com/blog/circuit-breaking-haproxy) to monitor and protect the cluster. + +## CVE repository in a zot cluster environment + +In the scale-out clustering scheme described in this article, CVE scanning is disabled. In this case, we recommend implementing a CVE repository with a zot instance outside of the cluster using a local disk for storage and [Trivy](https://trivy.dev/) as the detection engine.