989 lines
32 KiB
Markdown
989 lines
32 KiB
Markdown
# redis-cluster
|
|
|
|
A production-oriented Helm chart for a native, sharded Redis Cluster on
|
|
Kubernetes. It creates multiple independently writable shard primaries, one or
|
|
more readable failover replicas for every shard, strict cross-node placement,
|
|
persistent storage, optional external cluster-aware access, Prometheus metrics,
|
|
and an optional RedisInsight UI.
|
|
|
|
This chart deploys Redis Cluster mode. It does not deploy Sentinel, a proxy, or
|
|
an active-active database. Every hash slot has exactly one writable primary;
|
|
write capacity scales because different slot ranges belong to different
|
|
primaries.
|
|
|
|
## Contents
|
|
|
|
- [Architecture](#architecture)
|
|
- [Requirements](#requirements)
|
|
- [Quick start](#quick-start)
|
|
- [Connecting clients](#connecting-clients)
|
|
- [Authentication](#authentication)
|
|
- [Persistence](#persistence)
|
|
- [External access](#external-access)
|
|
- [RedisInsight](#redisinsight)
|
|
- [Metrics](#metrics)
|
|
- [Network policy and service mesh](#network-policy-and-service-mesh)
|
|
- [Scheduling and availability](#scheduling-and-availability)
|
|
- [Safe upgrades](#safe-upgrades)
|
|
- [Scaling and topology changes](#scaling-and-topology-changes)
|
|
- [Backup and restore](#backup-and-restore)
|
|
- [Argo CD](#argo-cd)
|
|
- [Configuration reference](#configuration-reference)
|
|
- [Troubleshooting](#troubleshooting)
|
|
- [Uninstalling](#uninstalling)
|
|
|
|
## Architecture
|
|
|
|
The default topology is three shards with one replica per shard:
|
|
|
|
```text
|
|
Kubernetes worker A Kubernetes worker B Kubernetes worker C
|
|
------------------- ------------------- -------------------
|
|
shard 0 member 0 shard 1 member 0 shard 2 member 0
|
|
canonical primary canonical primary canonical primary
|
|
|
|
shard 2 member 1 shard 0 member 1 shard 1 member 1
|
|
replica replica replica
|
|
```
|
|
|
|
That produces six Redis instances:
|
|
|
|
```text
|
|
3 writable primaries + 3 readable/failover replicas = 6 Redis pods
|
|
```
|
|
|
|
Each persistent member is a separate one-replica StatefulSet with its own PVC.
|
|
This design allows Kubernetes to enforce all of the following:
|
|
|
|
- all canonical primaries occupy different `kubernetes.io/hostname` domains;
|
|
- the members of one shard never occupy the same node;
|
|
- each replica is crossed onto the node containing another shard's primary;
|
|
- member identities and Redis node IDs survive pod replacement.
|
|
|
|
At first boot, the shard-0 member-0 pod coordinates cluster creation. It waits
|
|
for every pristine member, introduces all nodes, divides all 16,384 hash slots
|
|
between the primaries, and attaches each replica to its intended primary. It
|
|
refuses to overwrite partially initialized cluster state.
|
|
|
|
Redis can promote a replica after a primary failure. A reconciliation loop later
|
|
returns the cluster to its canonical crossed layout after the failed node comes
|
|
back. Redis Cluster replication is asynchronous; a narrow failure or partition
|
|
window can still lose an acknowledged write.
|
|
|
|
## Requirements
|
|
|
|
- Kubernetes 1.25 or newer.
|
|
- Helm 3.
|
|
- At least `cluster.shards` schedulable worker nodes with distinct
|
|
`kubernetes.io/hostname` labels. The default requires three workers.
|
|
- Enough CPU and memory on those workers for every Redis and exporter pod.
|
|
- A default StorageClass, or an explicit `persistence.storageClass`, when
|
|
persistence is enabled.
|
|
- Working pod DNS and unrestricted Redis client and cluster-bus traffic between
|
|
members.
|
|
|
|
Optional features have additional requirements:
|
|
|
|
- external access requires a controller that implements `type: LoadBalancer`
|
|
Services and populates `.status.loadBalancer.ingress`;
|
|
- `metrics.serviceMonitor.enabled` requires the Prometheus Operator CRDs;
|
|
- RedisInsight Istio ingress requires the Istio networking CRDs;
|
|
- automatic RedisInsight certificate creation requires cert-manager.
|
|
|
|
## Quick start
|
|
|
|
Clone the repository and install an internal-only cluster:
|
|
|
|
```sh
|
|
git clone https://git.mulas.me/corrado/redis-cluster.git
|
|
cd redis-cluster
|
|
|
|
helm install redis . \
|
|
--namespace redis \
|
|
--create-namespace
|
|
```
|
|
|
|
Wait for all six default members:
|
|
|
|
```sh
|
|
kubectl get pods -n redis -w
|
|
```
|
|
|
|
Check cluster health and slot coverage:
|
|
|
|
```sh
|
|
kubectl exec -n redis redis-redis-cluster-s0-n0-0 -c redis -- \
|
|
redis-cli cluster info
|
|
|
|
kubectl exec -n redis redis-redis-cluster-s0-n0-0 -c redis -- \
|
|
redis-cli --cluster check 127.0.0.1:6379
|
|
```
|
|
|
|
`cluster_state:ok`, `cluster_slots_ok:16384`, and zero failed slots indicate a
|
|
healthy cluster.
|
|
|
|
### Recommended production values
|
|
|
|
Create a values file:
|
|
|
|
```yaml
|
|
auth:
|
|
enabled: true
|
|
existingSecret: redis-auth
|
|
existingSecretKey: redis-password
|
|
|
|
persistence:
|
|
storageClass: fast-storage
|
|
size: 100Gi
|
|
|
|
redis:
|
|
maxmemory: 80gb
|
|
resources:
|
|
requests:
|
|
cpu: "1"
|
|
memory: 84Gi
|
|
limits:
|
|
memory: 88Gi
|
|
|
|
metrics:
|
|
serviceMonitor:
|
|
enabled: true
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
|
|
networkPolicy:
|
|
enabled: true
|
|
allowExternal: false
|
|
extraIngress:
|
|
- namespaceSelector:
|
|
matchLabels:
|
|
kubernetes.io/metadata.name: applications
|
|
```
|
|
|
|
Create the password Secret without committing the password:
|
|
|
|
```sh
|
|
kubectl create namespace redis
|
|
kubectl create secret generic redis-auth \
|
|
--namespace redis \
|
|
--from-literal=redis-password="$(openssl rand -base64 32)"
|
|
```
|
|
|
|
Install:
|
|
|
|
```sh
|
|
helm upgrade --install redis . \
|
|
--namespace redis \
|
|
--create-namespace \
|
|
--values production-values.yaml
|
|
```
|
|
|
|
Keep `redis.maxmemory` below the container memory limit. Redis needs headroom
|
|
for replication buffers, client buffers, fork/copy-on-write activity, AOF
|
|
rewrites, allocator overhead, and the operating process.
|
|
|
|
## Connecting clients
|
|
|
|
Redis Cluster requires a cluster-aware client. A standalone or Sentinel client
|
|
will not correctly follow `MOVED` and `ASK` redirects.
|
|
|
|
The internal balanced bootstrap endpoint is:
|
|
|
|
```text
|
|
<release>-redis-cluster.<namespace>.svc:<redis.port>
|
|
```
|
|
|
|
For the example release:
|
|
|
|
```text
|
|
redis-redis-cluster.redis.svc:6379
|
|
```
|
|
|
|
The bootstrap Service only provides an initial node. The client downloads the
|
|
slot map and then connects directly to the member endpoints advertised by
|
|
Redis.
|
|
|
|
### redis-cli
|
|
|
|
```sh
|
|
kubectl run redis-cli --rm -it \
|
|
--namespace redis \
|
|
--image=redis:7.4-alpine \
|
|
-- redis-cli -c -h redis-redis-cluster -p 6379
|
|
```
|
|
|
|
Add `-a "$REDIS_PASSWORD"` when authentication is enabled.
|
|
|
|
### Python
|
|
|
|
```python
|
|
from redis.cluster import RedisCluster
|
|
|
|
client = RedisCluster(
|
|
host="redis-redis-cluster.redis.svc",
|
|
port=6379,
|
|
password="replace-me",
|
|
decode_responses=True,
|
|
)
|
|
|
|
client.set("hello", "world")
|
|
print(client.get("hello"))
|
|
```
|
|
|
|
Set `read_from_replicas=True` only if stale replica reads are acceptable.
|
|
|
|
### Multi-key operations
|
|
|
|
Transactions, Lua scripts, and multi-key commands can only span keys in one hash
|
|
slot. Use a shared hash tag when keys must be colocated:
|
|
|
|
```text
|
|
cart:{customer-42}
|
|
cart-items:{customer-42}
|
|
```
|
|
|
|
## Authentication
|
|
|
|
Authentication is disabled by default so the chart can be evaluated without
|
|
secret bootstrapping. Enable it for production.
|
|
|
|
### Explicit password
|
|
|
|
```yaml
|
|
auth:
|
|
enabled: true
|
|
password: replace-me
|
|
```
|
|
|
|
This places the password in Helm release data and is not recommended for Git.
|
|
|
|
### Existing Secret
|
|
|
|
```yaml
|
|
auth:
|
|
enabled: true
|
|
existingSecret: redis-auth
|
|
existingSecretKey: redis-password
|
|
```
|
|
|
|
The Secret must exist in the release namespace:
|
|
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: Secret
|
|
metadata:
|
|
name: redis-auth
|
|
namespace: redis
|
|
type: Opaque
|
|
stringData:
|
|
redis-password: CHANGE-ME
|
|
```
|
|
|
|
Use SOPS, Sealed Secrets, External Secrets, or another secret manager rather
|
|
than committing the cleartext example.
|
|
|
|
### Generated password
|
|
|
|
When `auth.enabled=true` and neither `password` nor `existingSecret` is set, the
|
|
chart generates a password. A normal Helm upgrade uses `lookup` to reuse the
|
|
existing Secret.
|
|
|
|
Retrieve it with:
|
|
|
|
```sh
|
|
kubectl get secret -n redis redis-redis-cluster \
|
|
-o jsonpath='{.data.redis-password}' | base64 -d
|
|
echo
|
|
```
|
|
|
|
Argo CD renders charts without live-cluster access, so generated passwords need
|
|
the `ignoreDifferences` treatment shown in
|
|
[`argocd/application.yaml`](argocd/application.yaml), or preferably an
|
|
externally managed Secret.
|
|
|
|
The chart configures Redis `requirepass` and `masterauth`. It does not currently
|
|
manage Redis ACL users or Redis TLS.
|
|
|
|
## Persistence
|
|
|
|
Persistence is enabled by default. Every Redis member receives an independent
|
|
PVC and writes an append-only file using `appendfsync everysec`.
|
|
|
|
```yaml
|
|
persistence:
|
|
enabled: true
|
|
storageClass: ""
|
|
accessModes:
|
|
- ReadWriteOnce
|
|
size: 3Gi
|
|
```
|
|
|
|
An empty `storageClass` uses the cluster default. Any valid Kubernetes storage
|
|
quantity is accepted, including `500Mi`, `100Gi`, and `2Ti`. The chart imposes
|
|
no upper size limit. Actual capacity is controlled by the StorageClass,
|
|
provisioner, namespace quotas, and infrastructure.
|
|
|
|
`persistence.size` is per member, not total capacity. With three shards and one
|
|
replica, `100Gi` requests six 100 Gi volumes.
|
|
|
|
When persistence is disabled, Redis uses `emptyDir` and all data on a member is
|
|
lost when its pod is rescheduled.
|
|
|
|
### Expanding existing volumes
|
|
|
|
Increasing the value for a new installation is straightforward. Existing
|
|
StatefulSet `volumeClaimTemplates` are immutable, so changing
|
|
`persistence.size` alone may be rejected and does not reliably resize existing
|
|
claims.
|
|
|
|
To expand an existing deployment:
|
|
|
|
1. Confirm the StorageClass has `allowVolumeExpansion: true`.
|
|
2. Back up the cluster.
|
|
3. Patch each existing PVC to the new requested size.
|
|
4. Wait for the storage controller and filesystem resize to complete.
|
|
5. Keep `persistence.size` at the new value for future members.
|
|
|
|
Never shrink Redis PVCs in place.
|
|
|
|
## External access
|
|
|
|
External access is disabled by default because Redis Cluster cannot be exposed
|
|
correctly through a single generic TCP LoadBalancer.
|
|
|
|
When enabled, the chart creates:
|
|
|
|
- one balanced bootstrap LoadBalancer Service; and
|
|
- one LoadBalancer Service for every Redis member.
|
|
|
|
The default 3x1 topology therefore creates seven LoadBalancers. This can incur
|
|
cloud-provider cost and consume seven addresses.
|
|
|
|
```yaml
|
|
externalAccess:
|
|
enabled: true
|
|
port: 6379
|
|
loadBalancerClass: ""
|
|
externalTrafficPolicy: Cluster
|
|
loadBalancerSourceRanges:
|
|
- 203.0.113.0/24
|
|
```
|
|
|
|
The chart does not assume MetalLB or a particular cloud provider. It waits for
|
|
each member Service to publish either an IP or hostname under
|
|
`.status.loadBalancer.ingress`, then makes that member advertise the discovered
|
|
endpoint. All Services use the same Redis port; there is no fixed IP list and
|
|
no port fan-out.
|
|
|
|
Get the bootstrap endpoint:
|
|
|
|
```sh
|
|
kubectl get service -n redis redis-redis-cluster-external \
|
|
-o jsonpath='{range .status.loadBalancer.ingress[*]}{.ip}{.hostname}{"\n"}{end}'
|
|
```
|
|
|
|
External clients need only that bootstrap endpoint in their startup
|
|
configuration, but they must be able to route to every per-member address
|
|
returned in the slot map. A client that can reach the bootstrap address but not
|
|
the member addresses will connect initially and then fail on redirected keys.
|
|
|
|
Keep the LoadBalancer Services instead of deleting and recreating them. If the
|
|
controller assigns a new member address, restart that member's one-pod
|
|
StatefulSet so its discovery init container records and advertises the new
|
|
endpoint.
|
|
|
|
External Redis traffic is unencrypted. Restrict
|
|
`loadBalancerSourceRanges`, use upstream firewalls/private routing, enable
|
|
authentication, or provide network-layer encryption. Do not expose an
|
|
unauthenticated cluster to the public Internet.
|
|
|
|
### Provider annotations
|
|
|
|
Bootstrap and member annotations are separate:
|
|
|
|
```yaml
|
|
externalAccess:
|
|
bootstrap:
|
|
annotations:
|
|
service.beta.kubernetes.io/example: bootstrap-value
|
|
memberServices:
|
|
annotations:
|
|
service.beta.kubernetes.io/example: member-value
|
|
```
|
|
|
|
Use these for provider-specific address pools, internal LoadBalancers, health
|
|
checks, or DNS controllers.
|
|
|
|
## RedisInsight
|
|
|
|
RedisInsight is optional and disabled by default:
|
|
|
|
```yaml
|
|
redisInsight:
|
|
enabled: true
|
|
```
|
|
|
|
It creates one Deployment, one ClusterIP Service, and optionally a dedicated
|
|
PVC. RedisInsight stores its SQLite state, saved connections, and logs under
|
|
`/data`.
|
|
|
|
### Automatic cluster connection
|
|
|
|
No IP address is required. With `redisInsight.connection.host=""`, the chart
|
|
preconfigures RedisInsight from the release-aware balanced bootstrap endpoint:
|
|
|
|
- with external access disabled, it uses the internal
|
|
`<release>-redis-cluster` Service DNS;
|
|
- with external access enabled, an init container reads the published IP or
|
|
hostname from the `<release>-redis-cluster-external` LoadBalancer Service and
|
|
injects that address before RedisInsight starts.
|
|
|
|
Kubernetes and the LoadBalancer controller therefore own the VIP lifecycle.
|
|
RedisInsight follows the correct release and namespace without a hard-coded
|
|
address. Port `0` automatically follows `redis.port` or
|
|
`externalAccess.port`.
|
|
|
|
```yaml
|
|
redisInsight:
|
|
enabled: true
|
|
connection:
|
|
host: ""
|
|
port: 0
|
|
alias: redis-cluster
|
|
tls: false
|
|
```
|
|
|
|
Set `host` and `port` when RedisInsight should inspect another cluster, or when
|
|
pods cannot hairpin through the external LoadBalancer endpoint. When chart
|
|
authentication is enabled, RedisInsight reads the same password Secret
|
|
automatically.
|
|
|
|
Port-forward the UI:
|
|
|
|
```sh
|
|
kubectl port-forward -n redis \
|
|
service/redis-redis-cluster-insight 5540:80
|
|
```
|
|
|
|
Open <http://127.0.0.1:5540>.
|
|
|
|
### Istio HTTP ingress
|
|
|
|
The chart can create an Istio Gateway and VirtualService for RedisInsight:
|
|
|
|
```yaml
|
|
redisInsight:
|
|
enabled: true
|
|
ingress:
|
|
enabled: true
|
|
host: redisinsight.example.com
|
|
gatewaySelector:
|
|
istio: ingressgateway
|
|
requestTimeout: 60s
|
|
tls:
|
|
enabled: false
|
|
```
|
|
|
|
The host is required when ingress is enabled. Hosting RedisInsight under a
|
|
rewritten path prefix is not supported; give it a host and route `/`.
|
|
|
|
### Istio TLS with an existing Secret
|
|
|
|
```yaml
|
|
redisInsight:
|
|
ingress:
|
|
enabled: true
|
|
host: redisinsight.example.com
|
|
tls:
|
|
enabled: true
|
|
credentialName: redisinsight-tls
|
|
certificate:
|
|
create: false
|
|
```
|
|
|
|
The credential Secret must be available to the selected Istio ingress gateway.
|
|
Exact Secret placement depends on the Istio installation.
|
|
|
|
### Istio TLS with cert-manager
|
|
|
|
```yaml
|
|
redisInsight:
|
|
ingress:
|
|
enabled: true
|
|
host: redisinsight.example.com
|
|
tls:
|
|
enabled: true
|
|
credentialName: redisinsight-tls
|
|
certificate:
|
|
create: true
|
|
issuerName: letsencrypt-production
|
|
issuerKind: ClusterIssuer
|
|
secretNamespace: istio-system
|
|
```
|
|
|
|
Set `secretNamespace` to the namespace from which the ingress gateway reads
|
|
credential Secrets. An empty value uses the Helm release namespace.
|
|
|
|
RedisInsight is an administrative interface and does not provide ingress
|
|
authentication. Put an identity-aware proxy or Istio authorization policy in
|
|
front of it and restrict network exposure.
|
|
|
|
## Metrics
|
|
|
|
The Redis exporter sidecar is enabled by default on every member:
|
|
|
|
```yaml
|
|
metrics:
|
|
enabled: true
|
|
port: 9121
|
|
```
|
|
|
|
Enable a ServiceMonitor when the Prometheus Operator CRDs are installed:
|
|
|
|
```yaml
|
|
metrics:
|
|
serviceMonitor:
|
|
enabled: true
|
|
interval: 30s
|
|
scrapeTimeout: 10s
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
```
|
|
|
|
Use labels selected by your Prometheus installation.
|
|
|
|
## Network policy and service mesh
|
|
|
|
### NetworkPolicy
|
|
|
|
NetworkPolicy creation is disabled by default because policy enforcement and
|
|
default-deny behavior vary between CNIs.
|
|
|
|
```yaml
|
|
networkPolicy:
|
|
enabled: true
|
|
allowExternal: false
|
|
extraIngress:
|
|
- namespaceSelector:
|
|
matchLabels:
|
|
kubernetes.io/metadata.name: applications
|
|
metricsFromNamespaces:
|
|
- monitoring
|
|
```
|
|
|
|
The generated policy always allows Redis client, replication, migration, and
|
|
cluster-bus traffic between release pods. `allowExternal` refers to Redis client
|
|
ingress from outside the selected pods; it does not create a LoadBalancer.
|
|
|
|
### Istio sidecars
|
|
|
|
Redis pods disable Istio injection by default:
|
|
|
|
```yaml
|
|
istio:
|
|
injectSidecar: false
|
|
```
|
|
|
|
Redis Cluster uses direct node-to-node client and cluster-bus connections.
|
|
Namespace-wide strict mTLS can break this traffic. Either keep the Redis
|
|
workload outside strict mesh enforcement, create a scoped PERMISSIVE
|
|
PeerAuthentication, or deliberately enable and validate sidecars before
|
|
production use.
|
|
|
|
## Scheduling and availability
|
|
|
|
The chart uses required pod affinity and anti-affinity, not preferences. It will
|
|
leave pods Pending rather than silently place a primary beside its own replica.
|
|
|
|
For the default topology:
|
|
|
|
- three workers are required;
|
|
- losing one worker removes one primary and a replica of another shard;
|
|
- the lost primary's replica is on a surviving worker;
|
|
- two surviving primaries form the majority needed to authorize promotion.
|
|
|
|
Inspect placement:
|
|
|
|
```sh
|
|
kubectl get pods -n redis \
|
|
-L redis-cluster.shard,redis-cluster.member \
|
|
-o wide
|
|
```
|
|
|
|
The two members of each shard must have different `NODE` values. All member-0
|
|
pods must also occupy different nodes.
|
|
|
|
The global PodDisruptionBudget defaults to `maxUnavailable: 1`. It limits
|
|
cooperating voluntary disruptions but cannot prevent node crashes or an
|
|
administrator from deleting multiple pods.
|
|
|
|
### Failure behavior
|
|
|
|
- Failure detection begins after approximately
|
|
`cluster.nodeTimeoutMilliseconds`.
|
|
- A healthy cross-node replica can be elected primary.
|
|
- Cluster-aware clients refresh the slot map and reconnect.
|
|
- Losing both members of one shard loses that slot range.
|
|
- With `cluster.requireFullCoverage=true`, loss of any slot range makes the
|
|
cluster unavailable until coverage returns.
|
|
- Automatic failover requires a majority of current primaries to communicate.
|
|
|
|
## Safe upgrades
|
|
|
|
Every member StatefulSet uses `updateStrategy: OnDelete`. A Helm or Argo update
|
|
changes the pod template but deliberately does not restart all independent
|
|
members at once.
|
|
|
|
Restart exactly one member at a time:
|
|
|
|
1. Inspect `CLUSTER NODES` and choose a current replica first.
|
|
2. Delete that one pod.
|
|
3. Wait for the replacement to become Ready.
|
|
4. Confirm `cluster_state:ok`.
|
|
5. Continue with the next replica, then primaries one by one.
|
|
|
|
Example:
|
|
|
|
```sh
|
|
kubectl exec -n redis redis-redis-cluster-s0-n0-0 -c redis -- \
|
|
redis-cli cluster nodes
|
|
|
|
kubectl delete pod -n redis redis-redis-cluster-s0-n1-0
|
|
|
|
kubectl wait -n redis \
|
|
--for=condition=Ready \
|
|
pod/redis-redis-cluster-s0-n1-0 \
|
|
--timeout=300s
|
|
|
|
kubectl exec -n redis redis-redis-cluster-s0-n0-0 -c redis -- \
|
|
redis-cli cluster info
|
|
```
|
|
|
|
The preStop hook asks a healthy replica to take over before an intentional
|
|
primary restart. Never delete both members of one shard together.
|
|
|
|
## Scaling and topology changes
|
|
|
|
`cluster.shards` and `cluster.replicasPerShard` define the first-boot topology.
|
|
Changing them in Helm on a populated cluster does not reshard existing data and
|
|
is not a supported scaling procedure.
|
|
|
|
Adding a shard requires an explicit operational workflow:
|
|
|
|
1. create and persist the new members;
|
|
2. join them with `CLUSTER MEET`;
|
|
3. assign replicas;
|
|
4. move slots with `redis-cli --cluster reshard`;
|
|
5. verify slot coverage and failure placement.
|
|
|
|
Removing a shard requires moving all of its slots away before removing its
|
|
nodes. Adding replicas requires joining the nodes and issuing
|
|
`CLUSTER REPLICATE`. Back up first.
|
|
|
|
The chart enforces at least three shards, at least one replica per shard, and
|
|
fewer replicas per shard than shards so the crossed placement remains valid.
|
|
|
|
## Backup and restore
|
|
|
|
This chart does not schedule backups. Redis Cluster stores different slot ranges
|
|
on different primaries, so a backup strategy must cover every shard.
|
|
|
|
Common approaches include:
|
|
|
|
- crash-consistent snapshots of all member volumes;
|
|
- coordinated AOF/RDB copies from one healthy member of every shard;
|
|
- a Redis-aware backup product that understands cluster topology.
|
|
|
|
Test restore procedures. Restoring only one shard or one arbitrary member does
|
|
not reconstruct the complete keyspace. Keep backups outside the cluster and
|
|
verify both data and all 16,384 slots after recovery.
|
|
|
|
## Argo CD
|
|
|
|
[`argocd/application.yaml`](argocd/application.yaml) is a portable example using
|
|
the public repository. Apply it once:
|
|
|
|
```sh
|
|
kubectl apply -f argocd/application.yaml
|
|
```
|
|
|
|
The file is outside `templates/`, so it is not part of the Helm release. If you
|
|
edit its inline `valuesObject`, a normal sync of the Redis Application cannot
|
|
update the Application's own spec. Reapply the file, manage it from an
|
|
app-of-apps, or use an ApplicationSet:
|
|
|
|
```sh
|
|
kubectl apply -f argocd/application.yaml
|
|
```
|
|
|
|
For authentication, prefer an externally managed Secret. If the chart generates
|
|
the password, retain `RespectIgnoreDifferences=true` and the Secret
|
|
`ignoreDifferences` rule from the example.
|
|
|
|
Argo applies new StatefulSet templates, but `OnDelete` intentionally prevents
|
|
automatic Redis pod rolls. Perform the one-member-at-a-time procedure after
|
|
reviewing each change. Additional authentication, external access, and Istio
|
|
examples are in [`argocd/README.md`](argocd/README.md).
|
|
|
|
## Configuration reference
|
|
|
|
The authoritative defaults are in [`values.yaml`](values.yaml), and
|
|
[`values.schema.json`](values.schema.json) validates supported values.
|
|
|
|
### Release and image
|
|
|
|
| Value | Default | Description |
|
|
| --- | --- | --- |
|
|
| `nameOverride` | `""` | Override the chart name portion of resource names. |
|
|
| `fullnameOverride` | `""` | Override the complete release resource prefix; maximum 53 characters. |
|
|
| `image.repository` | `redis` | Redis image repository. |
|
|
| `image.tag` | `7.4-alpine` | Redis image tag. |
|
|
| `image.pullPolicy` | `IfNotPresent` | Redis image pull policy. |
|
|
| `imagePullSecrets` | `[]` | Pod image pull Secret references. |
|
|
|
|
### Cluster
|
|
|
|
| Value | Default | Description |
|
|
| --- | --- | --- |
|
|
| `cluster.shards` | `3` | Independently writable primaries and slot ranges. |
|
|
| `cluster.replicasPerShard` | `1` | Readable failover replicas per shard. |
|
|
| `cluster.busPort` | `16379` | Redis node-to-node cluster bus port. |
|
|
| `cluster.nodeTimeoutMilliseconds` | `10000` | Failure-detection timeout. |
|
|
| `cluster.requireFullCoverage` | `true` | Stop serving when any slot is unavailable. |
|
|
| `cluster.allowReadsWhenDown` | `false` | Permit reads while cluster state is down. |
|
|
| `cluster.migrationBarrier` | `1` | Minimum healthy replicas retained during replica migration. |
|
|
|
|
### Redis and authentication
|
|
|
|
| Value | Default | Description |
|
|
| --- | --- | --- |
|
|
| `auth.enabled` | `false` | Enable password authentication. |
|
|
| `auth.password` | `""` | Explicit password; generated when empty and no Secret is supplied. |
|
|
| `auth.existingSecret` | `""` | Existing password Secret name. |
|
|
| `auth.existingSecretKey` | `redis-password` | Password key in the existing Secret. |
|
|
| `redis.port` | `6379` | Redis client port. |
|
|
| `redis.maxmemory` | `384mb` | Redis memory ceiling. Empty disables the explicit ceiling. |
|
|
| `redis.maxmemoryPolicy` | `noeviction` | Redis eviction policy. |
|
|
| `redis.extraConfig` | `""` | Additional lines appended to `redis.conf`. |
|
|
| `redis.resources` | see values | Redis container requests and limits. |
|
|
|
|
Do not use `redis.extraConfig` to override cluster identity, announce endpoints,
|
|
data directory, authentication, or other settings managed by the chart.
|
|
|
|
### Probes
|
|
|
|
| Value | Default | Description |
|
|
| --- | --- | --- |
|
|
| `probes.redis.startup.initialDelaySeconds` | `10` | Delay before startup checks. |
|
|
| `probes.redis.startup.periodSeconds` | `5` | Startup check interval. |
|
|
| `probes.redis.startup.timeoutSeconds` | `4` | Startup check timeout. |
|
|
| `probes.redis.startup.failureThreshold` | `180` | Failures allowed before restart. |
|
|
| `probes.redis.liveness.periodSeconds` | `15` | Liveness check interval. |
|
|
| `probes.redis.liveness.timeoutSeconds` | `8` | Liveness check timeout. |
|
|
| `probes.redis.liveness.failureThreshold` | `10` | Liveness failures before restart. |
|
|
| `probes.redis.readiness.periodSeconds` | `5` | Readiness check interval. |
|
|
| `probes.redis.readiness.timeoutSeconds` | `4` | Readiness check timeout. |
|
|
| `probes.redis.readiness.failureThreshold` | `3` | Readiness failures before removal from Services. |
|
|
|
|
### Persistence
|
|
|
|
| Value | Default | Description |
|
|
| --- | --- | --- |
|
|
| `persistence.enabled` | `true` | Create one PVC per Redis member. |
|
|
| `persistence.storageClass` | `""` | StorageClass; empty uses the cluster default. |
|
|
| `persistence.accessModes` | `[ReadWriteOnce]` | PVC access modes. |
|
|
| `persistence.size` | `3Gi` | Capacity per member; no chart-side maximum. |
|
|
| `persistence.annotations` | `{}` | PVC template annotations. |
|
|
|
|
### External access
|
|
|
|
| Value | Default | Description |
|
|
| --- | --- | --- |
|
|
| `externalAccess.enabled` | `false` | Create bootstrap and per-member LoadBalancers. |
|
|
| `externalAccess.port` | `6379` | External Redis port on every Service. |
|
|
| `externalAccess.bootstrap.annotations` | `{}` | Bootstrap Service annotations. |
|
|
| `externalAccess.memberServices.annotations` | `{}` | Per-member Service annotations. |
|
|
| `externalAccess.loadBalancerClass` | `""` | Optional LoadBalancer implementation class. |
|
|
| `externalAccess.externalTrafficPolicy` | `Cluster` | Kubernetes external traffic policy. |
|
|
| `externalAccess.loadBalancerSourceRanges` | `[]` | Allowed client CIDRs. |
|
|
| `externalAccess.allocateLoadBalancerNodePorts` | `true` | Allocate backing NodePorts. |
|
|
| `externalAccess.endpointDiscovery.*` | see values | Image/resources for endpoint discovery. |
|
|
|
|
### RedisInsight
|
|
|
|
| Value | Default | Description |
|
|
| --- | --- | --- |
|
|
| `redisInsight.enabled` | `false` | Deploy RedisInsight. |
|
|
| `redisInsight.image.repository` | `redis/redisinsight` | RedisInsight image repository. |
|
|
| `redisInsight.image.tag` | `3.8.0` | RedisInsight image tag. |
|
|
| `redisInsight.image.pullPolicy` | `IfNotPresent` | RedisInsight image pull policy. |
|
|
| `redisInsight.port` | `5540` | RedisInsight container port. |
|
|
| `redisInsight.connection.host` | `""` | Empty discovers the balanced release bootstrap endpoint. |
|
|
| `redisInsight.connection.port` | `0` | Zero follows the selected Redis Service port. |
|
|
| `redisInsight.connection.alias` | `redis-cluster` | Display name in RedisInsight. |
|
|
| `redisInsight.connection.tls` | `false` | Enable TLS for the Redis connection. |
|
|
| `redisInsight.persistence.enabled` | `true` | Persist RedisInsight state. |
|
|
| `redisInsight.persistence.storageClass` | `""` | StorageClass; empty uses default. |
|
|
| `redisInsight.persistence.accessModes` | `[ReadWriteOnce]` | RedisInsight PVC access modes. |
|
|
| `redisInsight.persistence.size` | `1Gi` | RedisInsight PVC size; no chart-side maximum. |
|
|
| `redisInsight.persistence.annotations` | `{}` | RedisInsight PVC annotations. |
|
|
| `redisInsight.resources` | see values | RedisInsight requests and limits. |
|
|
| `redisInsight.podAnnotations` | `{}` | RedisInsight pod annotations. |
|
|
| `redisInsight.nodeSelector` | `{}` | RedisInsight node selector. |
|
|
| `redisInsight.tolerations` | `[]` | RedisInsight pod tolerations. |
|
|
| `redisInsight.ingress.enabled` | `false` | Create Istio Gateway and VirtualService. |
|
|
| `redisInsight.ingress.host` | `""` | Required hostname when ingress is enabled. |
|
|
| `redisInsight.ingress.gatewaySelector` | `{istio: ingressgateway}` | Select the Istio gateway workload. |
|
|
| `redisInsight.ingress.requestTimeout` | `60s` | Istio route timeout. |
|
|
| `redisInsight.ingress.tls.enabled` | `false` | Serve HTTPS on the Gateway. |
|
|
| `redisInsight.ingress.tls.credentialName` | `""` | Istio TLS credential Secret. |
|
|
| `redisInsight.ingress.tls.certificate.create` | `false` | Create the Secret using cert-manager. |
|
|
| `redisInsight.ingress.tls.certificate.*` | see values | Issuer and Secret namespace settings. |
|
|
|
|
### Metrics, policy, and pod settings
|
|
|
|
| Value | Default | Description |
|
|
| --- | --- | --- |
|
|
| `metrics.enabled` | `true` | Add redis_exporter to every member pod. |
|
|
| `metrics.image.repository` | `ghcr.io/oliver006/redis_exporter` | Exporter image repository. |
|
|
| `metrics.image.tag` | `v1.87.0-alpine` | Exporter image tag. |
|
|
| `metrics.image.pullPolicy` | `IfNotPresent` | Exporter pull policy. |
|
|
| `metrics.port` | `9121` | Exporter port. |
|
|
| `metrics.resources` | see values | Exporter requests and limits. |
|
|
| `metrics.serviceMonitor.enabled` | `false` | Create a Prometheus Operator ServiceMonitor. |
|
|
| `metrics.serviceMonitor.interval` | `30s` | Prometheus scrape interval. |
|
|
| `metrics.serviceMonitor.scrapeTimeout` | `10s` | Prometheus scrape timeout. |
|
|
| `metrics.serviceMonitor.labels` | `{}` | Labels used by Prometheus to select the ServiceMonitor. |
|
|
| `networkPolicy.enabled` | `false` | Create an ingress NetworkPolicy. |
|
|
| `networkPolicy.allowExternal` | `true` | Allow Redis client ingress from all sources. |
|
|
| `networkPolicy.extraIngress` | `[]` | Allowed peers when unrestricted ingress is disabled. |
|
|
| `networkPolicy.metricsFromNamespaces` | `[]` | Namespaces allowed to scrape metrics. |
|
|
| `podDisruptionBudget.enabled` | `true` | Create a global Redis PDB. |
|
|
| `podDisruptionBudget.maxUnavailable` | `1` | Maximum voluntary unavailable Redis pods. |
|
|
| `serviceAccount.create` | `true` | Create the endpoint-discovery ServiceAccount. |
|
|
| `serviceAccount.name` | `""` | Existing or generated ServiceAccount name. |
|
|
| `serviceAccount.annotations` | `{}` | ServiceAccount annotations. |
|
|
| `nodeSelector` | `{}` | Redis pod node selector. |
|
|
| `tolerations` | `[]` | Redis pod tolerations. |
|
|
| `topologySpreadConstraints` | `[]` | Additional spread constraints. |
|
|
| `priorityClassName` | `""` | Redis pod priority class. |
|
|
| `podAnnotations` / `podLabels` | `{}` | Additional Redis pod metadata. |
|
|
| `podSecurityContext` | see values | Redis pod-level security context. |
|
|
| `containerSecurityContext` | see values | Redis/exporter container security context. |
|
|
| `istio.injectSidecar` | `false` | Enable sidecars on Redis member pods. |
|
|
| `terminationGracePeriodSeconds` | `30` | Pod shutdown grace period. |
|
|
|
|
## Troubleshooting
|
|
|
|
### Pods remain Pending
|
|
|
|
Inspect scheduling events:
|
|
|
|
```sh
|
|
kubectl describe pod -n redis <pending-pod>
|
|
```
|
|
|
|
Common causes are:
|
|
|
|
- fewer schedulable workers than `cluster.shards`;
|
|
- insufficient CPU or memory;
|
|
- node taints without matching tolerations;
|
|
- node selectors excluding required workers;
|
|
- existing pods violating the required crossed affinity layout;
|
|
- unbound PVCs or a missing default StorageClass.
|
|
|
|
The scheduler message `didn't match pod affinity/anti-affinity rules` means the
|
|
availability invariant cannot currently be satisfied. Do not weaken it unless
|
|
you accept a primary and its replica sharing a failure domain.
|
|
|
|
### External-access pods wait in init
|
|
|
|
```sh
|
|
kubectl logs -n redis <pod> -c discover-external-endpoint
|
|
kubectl get services -n redis
|
|
```
|
|
|
|
The LoadBalancer controller must populate an IP or hostname for every member
|
|
Service. Check controller events, address-pool capacity, provider annotations,
|
|
quota, and `loadBalancerClass`.
|
|
|
|
### `cluster_state:fail`
|
|
|
|
```sh
|
|
kubectl exec -n redis redis-redis-cluster-s0-n0-0 -c redis -- \
|
|
redis-cli cluster nodes
|
|
|
|
kubectl exec -n redis redis-redis-cluster-s0-n0-0 -c redis -- \
|
|
redis-cli cluster info
|
|
```
|
|
|
|
Check for missing members, failed slot ranges, DNS failures, blocked port 16379,
|
|
or a member that cannot read its PVC. Readiness intentionally remains false
|
|
until the complete cluster is healthy; liveness does not restart Redis merely
|
|
for `CLUSTERDOWN`.
|
|
|
|
### Authentication errors
|
|
|
|
Confirm the client is using the current Secret and that all pods reference the
|
|
same password:
|
|
|
|
```sh
|
|
kubectl get secret -n redis redis-redis-cluster \
|
|
-o jsonpath='{.data.redis-password}' | base64 -d
|
|
echo
|
|
```
|
|
|
|
When using an existing Secret, verify its name/key match the Helm values. Restart
|
|
members one at a time if a password Secret was deliberately changed.
|
|
|
|
### External client connects, then fails
|
|
|
|
This almost always means the bootstrap address is reachable but the member
|
|
addresses in `CLUSTER SLOTS`/`CLUSTER SHARDS` are not. Route and permit every
|
|
per-member LoadBalancer endpoint.
|
|
|
|
### RedisInsight is not rendered by Argo CD
|
|
|
|
Confirm the live Application contains `redisInsight.enabled: true`:
|
|
|
|
```sh
|
|
kubectl get application -n argocd redis-cluster -o yaml
|
|
```
|
|
|
|
Changing `argocd/application.yaml` in this repository does not update the live
|
|
Application during its own Helm sync because the file is outside `templates/`.
|
|
Reapply it or manage it from an app-of-apps.
|
|
|
|
### Inspect rendered manifests
|
|
|
|
```sh
|
|
helm lint .
|
|
helm template redis . --namespace redis --debug
|
|
```
|
|
|
|
## Uninstalling
|
|
|
|
Back up first, then remove the release:
|
|
|
|
```sh
|
|
helm uninstall redis --namespace redis
|
|
```
|
|
|
|
PVCs created from StatefulSet volume claim templates are normally retained when
|
|
the StatefulSets are deleted. Verify them before deleting anything:
|
|
|
|
```sh
|
|
kubectl get pvc -n redis
|
|
```
|
|
|
|
The directly managed RedisInsight PVC may be removed with the Helm release,
|
|
depending on Helm and storage-policy behavior. Preserve or snapshot it if its UI
|
|
state matters.
|
|
|
|
Delete retained Redis data PVCs only when data destruction is intentional.
|
|
|
|
## License
|
|
|
|
This project is distributed under the BSD 3-Clause License. See
|
|
[`LICENSE`](LICENSE).
|