【Redis】用Docker-Compose搭建Redis Sentinel

Contents

1. 1. 系统环境
2. 2. docker-compose配置文件
3. 3. redis-master节点配置
4. 4. redis-slave节点配置
5. 5. sentinel节点配置
6. 6. 启动/停止redis主从+哨兵
7. 7. 致谢&参考

在上一篇Blog，【Redis】用Docker-Compose搭建Redis主从复制中，我们介绍了redis主从的配置，接下来这篇文章，我们在主从的基础上，再加上哨兵(Sentinel)机制，来进一步确保redis的高可用。

系统环境

按照惯例，还是列一下我们用到的系统环境和软件版本信息：

• RHEL/CentOS 7
• Dokcer & Docker compose
• Redis

假设我们要搭建1主2从3哨兵的架构，可以假设服务器的目录结构如下：

/app/redis-sentinel             -- 根目录
├── redis                       -- redis节点目录
    ├──conf                     -- redis各节点配置文件目录
        ├──redis-master.conf    -- master节点配置
        ├──redis-slave1.conf    -- slave1节点配置
        ├──redis-slave2.conf    -- slave2节点配置
    ├──data                     -- redis数据目录
├── sentinel                    -- sentinel目录
    ├──conf                     -- sentinel各节点配置文件目录
        ├──sentinel1.conf       -- sentinel1节点配置
        ├──sentinel2.conf       -- sentinel2节点配置
        ├──sentinel3.conf       -- sentinel3节点配置
    ├──data                     -- sentinel数据目录
├── docker-compose.yml          -- compose配置文件

几点说明：

1. sentinel本质上也是一个redis的节点，只不过它只负责监听和配置管理实际节点的主从关系，不实际负责业务数据
2. sentinel本质是配置而不是代理，也就是说，客户端连接时，来sentinel上询问主节点的实际ip和端口，而不是直接连接到了sentinel本身
3. sentinel的data目录中实际只存放log日志

docker-compose配置文件

version: '3'

services:
  # redis-masater
  redis-server-master:
    image: redis
    container_name: redis-server-m
    restart: always
    ports:
      - 6379:6379
    environment:
      TZ: "Asia/Shanghai"
    volumes:
      # configuration and data folder
      - ./redis/conf/redis-master.conf:/usr/local/etc/redis/redis.conf
      - ./redis/data/master/:/data:Z
    command: ["redis-server", "/usr/local/etc/redis/redis.conf"]
  # redis slave 1
  redis-server-slave-1:
    image: redis
    container_name: redis-server-s1
    restart: always
    ports:
      - 6380:6379
    depends_on:
      - redis-server-m
    environment:
      TZ: "Asia/Shanghai"
    volumes:
      - ./redis/conf/redis-slave1.conf:/usr/local/etc/redis/redis.conf
      - ./redis/data/slave1/:/data:Z
    command: ["redis-server", "/usr/local/etc/redis/redis.conf"]
  # redis slave 2
  redis-server-slave-2:
    image: redis
    container_name: redis-server-s2
    restart: always
    ports:
      - 6381:6379
    depends_on:
      - redis-server-m
    environment:
      TZ: "Asia/Shanghai"
    volumes:
      - ./redis/conf/redis-slave2.conf:/usr/local/etc/redis/redis.conf
      - ./redis/data/slave2/:/data:Z
    command: ["redis-server", "/usr/local/etc/redis/redis.conf"]

  # sentinel 1
  redis-sentinel-1:
    image: redis
    container_name: redis-sentinel-1
    restart: always
    environment:
      TZ: "Asia/Shanghai"
    ports:
      - 26379:26379
    depends_on:
      - redis-server-m
    volumes:
      - ./sentinel/conf/sentinel1.conf:/usr/local/etc/redis/sentinel.conf
      - ./sentinel/data/:/data:Z
    command: redis-sentinel /usr/local/etc/redis/sentinel.conf
  # sentinel 2
  redis-sentinel-2:
    image: redis
    container_name: redis-sentinel-2
    restart: always
    environment:
      TZ: "Asia/Shanghai"
    ports:
      - 26380:26379
    depends_on:
      - redis-server-m
    volumes:
      - ./sentinel/conf/sentinel2.conf:/usr/local/etc/redis/sentinel.conf
      - ./sentinel/data/:/data:Z
    command: redis-sentinel /usr/local/etc/redis/sentinel.conf
  # sentinel 3
  redis-sentinel-3:
    image: redis
    container_name: redis-sentinel-3
    restart: always
    environment:
      TZ: "Asia/Shanghai"
    ports:
      - 26381:26379
    depends_on:
      - redis-server-m
    volumes:
      - ./sentinel/conf/sentinel3.conf:/usr/local/etc/redis/sentinel.conf
      - ./sentinel/data/:/data:Z
    command: redis-sentinel /usr/local/etc/redis/sentinel.conf

从compose的配置中可以看到，1主2从3个sentinel节点，一共6个节点，

• 本地的6379-6381作为3个redis server的端口地址，26379-26381作为3个 redis sentinel的端口地址
• 因为sentinel本质上还是一个redis节点，所以docker image镜像文件和redis server是用的同样的image
• command中 redis-sentinel就表示当前节点以sentinel方式运行

redis-master节点配置

redis节点的配置相对于上一个章节中的配置，只是多了主从复制部分的配置；如果没有sentinel，那么master节点是不需要做主从配置方面的修改的，只是从节点需要声明主节点的一些信息，但是有了sentinel，就不一样了：sentinel就表示主从节点的关系是可以发生变化的，所以我们的master就不一定一直是master节点了。

################################# REPLICATION #################################
# If the master is password protected (using the "requirepass" configuration
# directive below) it is possible to tell the replica to authenticate before
# starting the replication synchronization process, otherwise the master will
# refuse the replica request.
#
# masterauth <master-password>
#
# However this is not enough if you are using Redis ACLs (for Redis version
# 6 or greater), and the default user is not capable of running the PSYNC
# command and/or other commands needed for replication. In this case it's
# better to configure a special user to use with replication, and specify the
# masteruser configuration as such:
#
# masteruser <username>
#
# When masteruser is specified, the replica will authenticate against its
# master using the new AUTH form: AUTH <username> <password>.

# With sentinel, each redis node's status would be master or slave
masterauth 123456

# A Redis master is able to list the address and port of the attached
# replicas in different ways. For example the "INFO replication" section
# offers this information, which is used, among other tools, by
# Redis Sentinel in order to discover replica instances.
# Another place where this info is available is in the output of the
# "ROLE" command of a master.
#
# The listed IP address and port normally reported by a replica is
# obtained in the following way:
#
#   IP: The address is auto detected by checking the peer address
#   of the socket used by the replica to connect with the master.
#
#   Port: The port is communicated by the replica during the replication
#   handshake, and is normally the port that the replica is using to
#   listen for connections.
#
# However when port forwarding or Network Address Translation (NAT) is
# used, the replica may actually be reachable via different IP and port
# pairs. The following two options can be used by a replica in order to
# report to its master a specific set of IP and port, so that both INFO
# and ROLE will report those values.
#
# There is no need to use both the options if you need to override just
# the port or the IP address.
#
replica-announce-ip 10.0.0.143
replica-announce-port 6379

• masterauth
  因为master节点可能会变成slave节点，所以就需要申明master的auth密码了

• replica-announce-ip
  同理，需要告知其他节点当前节点的ip和端口，当然，实例中的10.0.0.143是本人实验环境的ip地址，实际以个人不同环境为准

redis-slave节点配置

从节点的配置和上一章节【Redis】用Docker-Compose搭建Redis主从复制的从节点配置保持一致不变就好了。

sentinel节点配置

因为一共有3个sentinel，我们就以其中一个为例，其他两个类似就可以了。

# Example sentinel.conf

# *** IMPORTANT ***
#
# By default Sentinel will not be reachable from interfaces different than
# localhost, either use the 'bind' directive to bind to a list of network
# interfaces, or disable protected mode with "protected-mode no" by
# adding it to this configuration file.
#
# Before doing that MAKE SURE the instance is protected from the outside
# world via firewalling or other means.
#
# For example you may use one of the following:
#
# bind 127.0.0.1 192.168.1.1
#
protected-mode no

# port <sentinel-port>
# The port that this sentinel instance will run on
port 26379

# By default Redis Sentinel does not run as a daemon. Use 'yes' if you need it.
# Note that Redis will write a pid file in /var/run/redis-sentinel.pid when
# daemonized.
daemonize no

# When running daemonized, Redis Sentinel writes a pid file in
# /var/run/redis-sentinel.pid by default. You can specify a custom pid file
# location here.
pidfile /var/run/redis-sentinel.pid

# Specify the log file name. Also the empty string can be used to force
# Sentinel to log on the standard output. Note that if you use standard
# output for logging but daemonize, logs will be sent to /dev/null
logfile /data/sentinel-1.log

# sentinel announce-ip <ip>
# sentinel announce-port <port>
#
# The above two configuration directives are useful in environments where,
# because of NAT, Sentinel is reachable from outside via a non-local address.
#
# When announce-ip is provided, the Sentinel will claim the specified IP address
# in HELLO messages used to gossip its presence, instead of auto-detecting the
# local address as it usually does.
#
# Similarly when announce-port is provided and is valid and non-zero, Sentinel
# will announce the specified TCP port.
#
# The two options don't need to be used together, if only announce-ip is
# provided, the Sentinel will announce the specified IP and the server port
# as specified by the "port" option. If only announce-port is provided, the
# Sentinel will announce the auto-detected local IP and the specified port.
#
# Example:
#
# sentinel announce-ip 1.2.3.4
sentinel announce-ip 10.0.0.134
sentinel announce-port 26379

# dir <working-directory>
# Every long running process should have a well-defined working directory.
# For Redis Sentinel to chdir to /tmp at startup is the simplest thing
# for the process to don't interfere with administrative tasks such as
# unmounting filesystems.
dir /tmp

# sentinel monitor <master-name> <ip> <redis-port> <quorum>
#
# Tells Sentinel to monitor this master, and to consider it in O_DOWN
# (Objectively Down) state only if at least <quorum> sentinels agree.
#
# Note that whatever is the ODOWN quorum, a Sentinel will require to
# be elected by the majority of the known Sentinels in order to
# start a failover, so no failover can be performed in minority.
#
# Replicas are auto-discovered, so you don't need to specify replicas in
# any way. Sentinel itself will rewrite this configuration file adding
# the replicas using additional configuration options.
# Also note that the configuration file is rewritten when a
# replica is promoted to master.
#
# Note: master name should not include special characters or spaces.
# The valid charset is A-z 0-9 and the three characters ".-_".
sentinel monitor mymaster 10.0.0.134 6379 2

# sentinel auth-pass <master-name> <password>
#
# Set the password to use to authenticate with the master and replicas.
# Useful if there is a password set in the Redis instances to monitor.
#
# Note that the master password is also used for replicas, so it is not
# possible to set a different password in masters and replicas instances
# if you want to be able to monitor these instances with Sentinel.
#
# However you can have Redis instances without the authentication enabled
# mixed with Redis instances requiring the authentication (as long as the
# password set is the same for all the instances requiring the password) as
# the AUTH command will have no effect in Redis instances with authentication
# switched off.
#
# Example:
#
# sentinel auth-pass mymaster MySUPER--secret-0123passw0rd
sentinel auth-pass mymaster 123456

# sentinel auth-user <master-name> <username>
#
# This is useful in order to authenticate to instances having ACL capabilities,
# that is, running Redis 6.0 or greater. When just auth-pass is provided the
# Sentinel instance will authenticate to Redis using the old "AUTH <pass>"
# method. When also an username is provided, it will use "AUTH <user> <pass>".
# In the Redis servers side, the ACL to provide just minimal access to
# Sentinel instances, should be configured along the following lines:
#
#     user sentinel-user >somepassword +client +subscribe +publish \
#                        +ping +info +multi +slaveof +config +client +exec on

# sentinel down-after-milliseconds <master-name> <milliseconds>
#
# Number of milliseconds the master (or any attached replica or sentinel) should
# be unreachable (as in, not acceptable reply to PING, continuously, for the
# specified period) in order to consider it in S_DOWN state (Subjectively
# Down).
#
# Default is 30 seconds.
sentinel down-after-milliseconds mymaster 30000

# IMPORTANT NOTE: starting with Redis 6.2 ACL capability is supported for
# Sentinel mode, please refer to the Redis website https://redis.io/topics/acl
# for more details.

# Sentinel's ACL users are defined in the following format:
#
#   user <username> ... acl rules ...
#
# For example:
#
#   user worker +@admin +@connection ~* on >ffa9203c493aa99
#
# For more information about ACL configuration please refer to the Redis
# website at https://redis.io/topics/acl and redis server configuration 
# template redis.conf.

# ACL LOG
#
# The ACL Log tracks failed commands and authentication events associated
# with ACLs. The ACL Log is useful to troubleshoot failed commands blocked 
# by ACLs. The ACL Log is stored in memory. You can reclaim memory with 
# ACL LOG RESET. Define the maximum entry length of the ACL Log below.
acllog-max-len 128

# Using an external ACL file
#
# Instead of configuring users here in this file, it is possible to use
# a stand-alone file just listing users. The two methods cannot be mixed:
# if you configure users here and at the same time you activate the external
# ACL file, the server will refuse to start.
#
# The format of the external ACL user file is exactly the same as the
# format that is used inside redis.conf to describe users.
#
# aclfile /etc/redis/sentinel-users.acl

# requirepass <password>
#
# You can configure Sentinel itself to require a password, however when doing
# so Sentinel will try to authenticate with the same password to all the
# other Sentinels. So you need to configure all your Sentinels in a given
# group with the same "requirepass" password. Check the following documentation
# for more info: https://redis.io/topics/sentinel
#
# IMPORTANT NOTE: starting with Redis 6.2 "requirepass" is a compatibility
# layer on top of the ACL system. The option effect will be just setting
# the password for the default user. Clients will still authenticate using
# AUTH <password> as usually, or more explicitly with AUTH default <password>
# if they follow the new protocol: both will work.
#
# New config files are advised to use separate authentication control for
# incoming connections (via ACL), and for outgoing connections (via
# sentinel-user and sentinel-pass) 
#
# The requirepass is not compatable with aclfile option and the ACL LOAD
# command, these will cause requirepass to be ignored.

# sentinel sentinel-user <username>
#
# You can configure Sentinel to authenticate with other Sentinels with specific
# user name. 

# sentinel sentinel-pass <password>
#
# The password for Sentinel to authenticate with other Sentinels. If sentinel-user
# is not configured, Sentinel will use 'default' user with sentinel-pass to authenticate.

# sentinel parallel-syncs <master-name> <numreplicas>
#
# How many replicas we can reconfigure to point to the new replica simultaneously
# during the failover. Use a low number if you use the replicas to serve query
# to avoid that all the replicas will be unreachable at about the same
# time while performing the synchronization with the master.
sentinel parallel-syncs mymaster 1

# sentinel failover-timeout <master-name> <milliseconds>
#
# Specifies the failover timeout in milliseconds. It is used in many ways:
#
# - The time needed to re-start a failover after a previous failover was
#   already tried against the same master by a given Sentinel, is two
#   times the failover timeout.
#
# - The time needed for a replica replicating to a wrong master according
#   to a Sentinel current configuration, to be forced to replicate
#   with the right master, is exactly the failover timeout (counting since
#   the moment a Sentinel detected the misconfiguration).
#
# - The time needed to cancel a failover that is already in progress but
#   did not produced any configuration change (SLAVEOF NO ONE yet not
#   acknowledged by the promoted replica).
#
# - The maximum time a failover in progress waits for all the replicas to be
#   reconfigured as replicas of the new master. However even after this time
#   the replicas will be reconfigured by the Sentinels anyway, but not with
#   the exact parallel-syncs progression as specified.
#
# Default is 3 minutes.
sentinel failover-timeout mymaster 180000

# SCRIPTS EXECUTION
#
# sentinel notification-script and sentinel reconfig-script are used in order
# to configure scripts that are called to notify the system administrator
# or to reconfigure clients after a failover. The scripts are executed
# with the following rules for error handling:
#
# If script exits with "1" the execution is retried later (up to a maximum
# number of times currently set to 10).
#
# If script exits with "2" (or an higher value) the script execution is
# not retried.
#
# If script terminates because it receives a signal the behavior is the same
# as exit code 1.
#
# A script has a maximum running time of 60 seconds. After this limit is
# reached the script is terminated with a SIGKILL and the execution retried.

# NOTIFICATION SCRIPT
#
# sentinel notification-script <master-name> <script-path>
# 
# Call the specified notification script for any sentinel event that is
# generated in the WARNING level (for instance -sdown, -odown, and so forth).
# This script should notify the system administrator via email, SMS, or any
# other messaging system, that there is something wrong with the monitored
# Redis systems.
#
# The script is called with just two arguments: the first is the event type
# and the second the event description.
#
# The script must exist and be executable in order for sentinel to start if
# this option is provided.
#
# Example:
#
# sentinel notification-script mymaster /var/redis/notify.sh

# CLIENTS RECONFIGURATION SCRIPT
#
# sentinel client-reconfig-script <master-name> <script-path>
#
# When the master changed because of a failover a script can be called in
# order to perform application-specific tasks to notify the clients that the
# configuration has changed and the master is at a different address.
# 
# The following arguments are passed to the script:
#
# <master-name> <role> <state> <from-ip> <from-port> <to-ip> <to-port>
#
# <state> is currently always "failover"
# <role> is either "leader" or "observer"
# 
# The arguments from-ip, from-port, to-ip, to-port are used to communicate
# the old address of the master and the new address of the elected replica
# (now a master).
#
# This script should be resistant to multiple invocations.
#
# Example:
#
# sentinel client-reconfig-script mymaster /var/redis/reconfig.sh

# SECURITY
#
# By default SENTINEL SET will not be able to change the notification-script
# and client-reconfig-script at runtime. This avoids a trivial security issue
# where clients can set the script to anything and trigger a failover in order
# to get the program executed.
sentinel deny-scripts-reconfig yes

# REDIS COMMANDS RENAMING
#
# Sometimes the Redis server has certain commands, that are needed for Sentinel
# to work correctly, renamed to unguessable strings. This is often the case
# of CONFIG and SLAVEOF in the context of providers that provide Redis as
# a service, and don't want the customers to reconfigure the instances outside
# of the administration console.
#
# In such case it is possible to tell Sentinel to use different command names
# instead of the normal ones. For example if the master "mymaster", and the
# associated replicas, have "CONFIG" all renamed to "GUESSME", I could use:
#
# SENTINEL rename-command mymaster CONFIG GUESSME
#
# After such configuration is set, every time Sentinel would use CONFIG it will
# use GUESSME instead. Note that there is no actual need to respect the command
# case, so writing "config guessme" is the same in the example above.
#
# SENTINEL SET can also be used in order to perform this configuration at runtime.
#
# In order to set a command back to its original name (undo the renaming), it
# is possible to just rename a command to itself:
#
# SENTINEL rename-command mymaster CONFIG CONFIG

# HOSTNAMES SUPPORT
#
# Normally Sentinel uses only IP addresses and requires SENTINEL MONITOR
# to specify an IP address. Also, it requires the Redis replica-announce-ip
# keyword to specify only IP addresses.
#
# You may enable hostnames support by enabling resolve-hostnames. Note
# that you must make sure your DNS is configured properly and that DNS
# resolution does not introduce very long delays.
#
SENTINEL resolve-hostnames no

# When resolve-hostnames is enabled, Sentinel still uses IP addresses
# when exposing instances to users, configuration files, etc. If you want
# to retain the hostnames when announced, enable announce-hostnames below.
#
SENTINEL announce-hostnames no

• protected-mode no
  同redis server的含义一样，当需要其他节点访问时，就需要设置成no

• daemonize no
  同理，因为我们是用docker启动的，所以这里也不需要额外的守护进程

• logfile /data/sentinel-1.log
  logfile指定了日志文件的存放位置，这里和上一章节介绍的内容一样，因为我们在compose.yml配置文件中已经指定了data目录的挂载目录，所以在配置文件中，就直接写容器内部的路径就好，而不是设置成宿主机的实际路径

• sentinel announce-ip
  和redis server的声明ip、端口含义一样，这里声明不同sentinel的ip和端口

• sentinel monitor mymaster 10.0.0.134 6379 2
  监听的server ip和端口，别名设置为mymaster，最后的数字2表示，当至少有2个sentinel同意时，才能把master节点置为O_DOWN(Objectively down，客观下线)

• sentinel auth-pass mymaster 123456
  master节点登录密码

• sentinel down-after-milliseconds mymaster 30000
  在多少微秒连接超时后，认为master节点S_DOWN(Subjectively down，主观下线)

• sentinel deny-scripts-reconfig yes
  禁止客户端随意改变reconfig的脚本，防止出现安全问题

关于主观下线和客观下线：

• 主观下线：在心跳检测的定时任务中，如果其他节点超过一定时间没有回复，哨兵节点就会将其进行主观下线。顾名思义，主观下线的意思是一个哨兵节点“主观地”判断下线；与主观下线相对应的是客观下线
• 客观下线：哨兵节点在对主节点进行主观下线后，会通过sentinel is-master-down-by-addr命令询问其他哨兵节点该主节点的状态；如果判断主节点下线的哨兵数量达到一定数值，则对该主节点进行客观下线

启动/停止redis主从+哨兵

配置好了后，我们只需要回到redis所在的目录，然后执行

docker-compose up -d

想要停止redis，只需要下面一条简单的命令即可：

docker-compose stop

至此，简洁快速的redis主从+哨兵配置就搭建好了。

致谢&参考

• docker-compose搭建redis-sentinel
• Docker-compose搭建redis一主双从三哨兵
• Redis Sentinel 高可用服务架构搭建
• 深入学习Redis（4）：哨兵