Skip to content

Commit

Permalink
update readme
Browse files Browse the repository at this point in the history
  • Loading branch information
@polefishu
polefishu committed Jan 20, 2020
1 parent 2659b73 commit 4f4ef2e
Show file tree
Hide file tree
Showing 2 changed files with 92 additions and 30 deletions.
122 changes: 92 additions & 30 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,37 @@ The operator itself is built with the [Operator framework](https://github.com/op

It inspired by [spotahome/redis-operator](https://github.com/spotahome/redis-operator).

![Redis Cluster atop Kubernetes](/static/redis-sentinel-readme.png)

* Create a statefulset to mange Redis instances (masters and replicas), each redis instance has default PreStop script that can do failover if master is down.
* Create a statefulset to mange Sentinel instances that will control the Redis nodes, each Sentinel instance has default ReadinessProbe script to detect whether the current sentinel's status is ok. When a sentinel pod is not ready, it is removed from Service load balancers.
* Create a Service and a Headless service for Sentinel statefulset.
* Create a Headless service for Redis statefulset.

Table of Contents
=================

* [redis-operator](#redis-operator)
* [Overview](#overview)
* [Prerequisites](#prerequisites)
* [Features](#features)
* [Quick Start](#quick-start)
* [Deploy redis operator](#deploy-redis-operator)
* [Deploy a sample redis cluster](#deploy-a-sample-redis-cluster)
* [Resize an Redis Cluster](#resize-an-redis-cluster)
* [Create redis cluster with password](#create-redis-cluster-with-password)
* [Dynamically changing redis config](#dynamically-changing-redis-config)
* [Persistence](#persistence)
* [Custom SecurityContext](#custom-securitycontext)
* [Cleanup](#cleanup)
* [Automatic failover details](#automatic-failover-details)

## Prerequisites

* go version v1.12+.
* Access to a Kubernetes v1.11.3+ cluster.
* go version v1.13+.
* Access to a Kubernetes v1.13.10+ cluster.

## Capabilities
## Features
In addition to the sentinel's own capabilities, redis-operator can:

* Push events and update status to the Kubernetes when resources have state changes
Expand All @@ -28,7 +53,7 @@ In addition to the sentinel's own capabilities, redis-operator can:
## Quick Start

### Deploy redis operator
Build and push the redis-operator and e2e test image
Build and push the redis-operator image
```
$ make REGISTRY=you_public_registry build-image
$ make REGISTRY=you_public_registry push
Expand Down Expand Up @@ -92,33 +117,32 @@ Verify that the cluster instances and its components are running.
```
$ kubectl get rediscluster
NAME SIZE STATUS AGE
test 3 Healthy 22h
test 3 Healthy 4m9s
$ kubectl get all -l app.kubernetes.io/managed-by=redis-operator
NAME READY STATUS RESTARTS AGE
pod/redis-cluster-test-0 1/1 Running 0 22h
pod/redis-cluster-test-1 1/1 Running 0 22h
pod/redis-cluster-test-2 1/1 Running 0 22h
pod/redis-sentinel-test-7cbd85785b-6llfp 1/1 Running 0 22h
pod/redis-sentinel-test-7cbd85785b-ggqw4 1/1 Running 0 22h
pod/redis-sentinel-test-7cbd85785b-nxxfc 1/1 Running 0 22h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/redis-sentinel-test ClusterIP xxxxxxxxxx <none> 26379/TCP 22h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/redis-sentinel-test 3/3 3 3 22h
NAME DESIRED CURRENT READY AGE
replicaset.apps/redis-sentinel-test-7cbd85785b 3 3 3 22h
NAME READY AGE
statefulset.apps/redis-cluster-test 3/3 22h
NAME READY STATUS RESTARTS AGE
pod/redis-cluster-test-0 1/1 Running 0 4m16s
pod/redis-cluster-test-1 1/1 Running 0 3m22s
pod/redis-cluster-test-2 1/1 Running 0 2m40s
pod/redis-sentinel-test-0 1/1 Running 0 4m16s
pod/redis-sentinel-test-1 1/1 Running 0 81s
pod/redis-sentinel-test-2 1/1 Running 0 18s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/redis-cluster-test ClusterIP None <none> 6379/TCP 4m16s
service/redis-sentinel-headless-test ClusterIP None <none> 26379/TCP 4m16s
service/redis-sentinel-test ClusterIP 10.22.22.34 <none> 26379/TCP 4m16s
NAME READY AGE
statefulset.apps/redis-cluster-test 3/3 4m16s
statefulset.apps/redis-sentinel-test 3/3 4m16s
```

* redis-cluster-<NAME>: Redis statefulset
* redis-sentinel-<NAME>: Sentinel deployment
* redis-sentinel-<NAME>: Sentinel statefulset
* redis-sentinel-<NAME>: Sentinel service
* redis-sentinel-headless-<NAME>: Sentinel headless service
* redis-cluster-<NAME>: Redis headless service

Describe the Redis Cluster, Viewing Events and Status
```
Expand Down Expand Up @@ -277,18 +301,23 @@ spec:
cpu: 50m
memory: 30Mi
size: 3
# when the disablePersistence set to false, the following configurations will be set automatically:
# disablePersistence: false
# config["save"] = "900 1 300 10"
# config["appendonly"] = "yes"
# config["auto-aof-rewrite-min-size"] = "1gb"
# config["auto-aof-rewrite-min-size"] = "536870912"
# config["repl-diskless-sync"] = "yes"
# config["repl-backlog-size"] = "60mb"
# config["repl-diskless-sync-delay"] = "5"
# config["repl-backlog-size"] = "62914560"
# config["aof-load-truncated"] = "yes"
# config["stop-writes-on-bgsave-error"] = "no"
# when the disablePersistence set to true, the following configurations will be set automatically:
# disablePersistence: true
# config["save"] = ""
# config["appendonly"] = "no"
disablePersistence: false
storage:
# By default, the persistent volume claims will be deleted when the Redis Cluster be delete.
# If this is not the expected usage, a keepAfterDeletion flag can be added under the storage section
Expand Down Expand Up @@ -344,4 +373,37 @@ $ kubectl delete -f deploy/namespace/role.yaml
$ kubectl delete -f deploy/namespace/role_binding.yaml
$ kubectl delete -f deploy/service_account.yaml
$ kubectl delete -f deploy/crds/redis_v1beta1_rediscluster_crd.yaml
```
```

## Automatic failover details

Redis-operator build a **Highly Available Redis cluster with Sentinel**, Sentinel always checks the MASTER and SLAVE
instances in the Redis cluster, checking whether they working as expected. If sentinel detects a failure in the
MASTER node in a given cluster, Sentinel will start a failover process. As a result, Sentinel will pick a SLAVE
instance and promote it to MASTER. Ultimately, the other remaining SLAVE instances will be automatically reconfigured
to use the new MASTER instance.

operator guarantees the following:
* Only one Redis instance as master in a cluster
* Number of Redis instance(masters and replicas) is equal as the set on the RedisCluster specification
* Number of Sentinels is equal as the set on the RedisCluster specification
* All Redis slaves have the same master
* All Sentinels point to the same Redis master
* Sentinel has not dead nodes

But Kubernetes pods are volatile, they can be deleted and recreated, and pods IP will change when pod be recreated,
and also, the IP will be recycled and redistributed to other pods.
Unfortunately, sentinel cannot delete the sentinel list or redis list in its memory when the pods IP changes.
This can be caused because there’s no way of a Sentinel node to self-deregister from the Sentinel Cluster before die,
provoking the Sentinel node list to increase without any control.

To ensure that Sentinel is working properly, operator will send a **RESET(SENTINEL RESET * )** signal to Sentinel node
one by one (if no failover is being running at that moment).
`SENTINEL RESET mastername` command: they'll refresh the list of replicas within the next 10 seconds, only adding the
ones listed as correctly replicating from the current master INFO output.
During this refresh time, `SENTINEL slaves <master name>` command can not get any result from sentinel, so operator sent
RESET signal to Sentinel one by one and wait sentinel status became ok(monitor correct master and has slaves).
Additional, Each Sentinel instance has default ReadinessProbe script to detect whether the current sentinel's status is ok.
When a sentinel pod is not ready, it is removed from Service load balancers.
Operator also create a headless svc for Sentinel statefulset, if you can not get result from `SENTINEL slaves <master name>` command,
You can try polling the headless domain.
Binary file added static/redis-sentinel-readme.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 4f4ef2e

Please sign in to comment.