HOW TO HA A REDIS APPLICATION [Master/Slave] THRU PACEMAKER

Posted on January 4, 2018

0


ENVIRONMENT

OS = RHEL 7.4
Total HA nodes = 2

ENVIRONMENT DETAILS

  • In this environment pacemaker is installed on both machines.
  • Redis is installed on both machines.
  • A Master/Slave resource group is created.
  • 2 HA heartbeats are configured thru corosync.

SUPPOSED ENVIRONMENT

2018-01-04_123758

ACTIVITY

vi /etc/redis.conf
bind 0.0.0.0              # This will make sure that redis will listen on all machines IP address

pcs resource create REDIS ocf:heartbeat:redis
pcs resource master REDIS
pcs resource create VIP ocf:heartbeat:IPaddr2 ip=192.168.0.99 cidr_netmask=32 nic=eth0 op monitor interval=30s

VERIFY

pcs status
Cluster name: REDISCLUSTER
Stack: corosync
Current DC: REDISCOM2 (version 1.1.16-12.el7-94ff4df) – partition with quorum
Last updated: Thu Jan 4 12:57:51 2018
Last change: Thu Jan 4 12:21:17 2018 by root via cibadmin on REDISCOM1

2 nodes configured
2 resources configured

Online: [ REDISCOM1 REDISCOM2 ]

Full list of resources:

Master/Slave Set: REDIS-master [REDIS]
Masters: [ REDISCOM2 ]
Slaves: [ REDISCOM1 ]

Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled

On Master Redis node (REDISCOM2) run the below commands to verify its a Master and (REDISCOM1) is Slave node.

redis-cli -h localhost
localhost:6379>
localhost:6379>INFO                  # Type INFO and check role under replication  section

# Replication
role:master

exit                                                 # Exit from Redis application.

On Slave Redis node (REDISCOM1) run the below commands to verify its a Slave and (REDISCOM2) is Master node.

redis-cli -h localhost
localhost:6379>
localhost:6379>INFO                  # Type INFO and check role under replication  section

exit                                                 # Exit from Redis application.

Restart Master node and check the pcs status to ensure that other HA node become Master. Furthermore make sure the after reboot the node should become Slave of new Master. This should be the case. However as per my experience the rebooted node could not become Slave and shows the status as STOPPED. According to my knowledge redis agent could not able to pick the latest state of rebooted node. As a workaround I added a delay in redis service which may start after pacemaker agent and pic latest state(Seems a agent bug as per my knowledge).

WORKAROUND

systemctl status redis|grep “Loaded: loaded”
Loaded: loaded (/usr/lib/systemd/system/redis.service; enabled; vendor preset: disabled)

vi /usr/lib/systemd/system/redis.service

[Service]
ExecStartPre=/bin/sleep 30
Advertisements
Posted in: Tech News