Fencing (STONITH) is an important mechanism in cluster to avoid the data corruption on shared storage. It also helps to bring the cluster into the known state when there is a split brain occurs between the nodes. Cluster nodes talks to each other over communication channels, which are typically standard network connections, such as Ethernet. Each resources and nodes have “state” (Ex: started , stopped) in the cluster and nodes report every changes that happens on resources. This reporting works well until communication breaks between the nodes. Fencing will come to play when nodes can’t communicate with each other. Majority of nodes will form the cluster based on quorum votes and rest of the nodes will be rebooted or halted based on fencing actions what we have denied.
There are two type of fencing available in pacemaker.
- Resource Level Fencing
- Node Level Fencing
Using the resource level fencing, the cluster can make sure that a node cannot access same resources on both the nodes. The node level fencing makes sure that a node does not run any resources at all. This is usually done in a very simple, yet brutal way: the node is simply reset using a power switch. This may ultimately be necessary because the node may not be responsive at all. In Pacamaker/corosync cluster, we will call the fencing method as “STONITH” (Shoot The Other Node In The Head).
For more information , please visit clusterlabs.org . Here we will see the node level fencing.
Have a look at the cluster setup.
[root@Node1-LAB ~]# pcs status Cluster name: GFSCLUS Last updated: Wed Jan 20 12:43:36 2016 Last change: Wed Jan 20 09:57:06 2016 via cibadmin on Node1 Stack: corosync Current DC: Node1 (1) - partition with quorum Version: 1.1.10-29.el7-368c726 2 Nodes configured 2 Resources configured Online: [ Node1 Node2 ] PCSD Status: Node1: Online Node2: Online Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled [root@Node1-LAB ~]#
In this article ,we will see that how to configure stonith/fencing “fence_xvm” for KVM cluster nodes. The purpose of this setup is to demonstrate the STONITH/FENCING.
Environment: (Demo Purpose only)
- Node 1 & Node 2 – Pacemaker/corosync cluster
- UNIXKB-CP – KVM host which hosts Node1 & Node2
Configure KVM host to use fence_xvm:
1.Login to the KVM host.
2.List the running virtual Machines .
[root@UNIXKB-CP ~]# virsh list Id Name State ---------------------------------------------------- 6 Node1 running 7 Node2 running [root@UNIXKB-CP ~]#
3.Install the required fencing packages on KVM host (Non-Cluster node)
[root@UNIXKB-CP ~]# yum install fence-virt fence-virtd fence-virtd-libvirt fence-virtd-multicast fence-virtd-serial Loaded plugins: langpacks, product-id, subscription-manager This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. Package fence-virt-0.3.0-16.el7.x86_64 already installed and latest version Package fence-virtd-0.3.0-16.el7.x86_64 already installed and latest version Package fence-virtd-libvirt-0.3.0-16.el7.x86_64 already installed and latest version Package fence-virtd-multicast-0.3.0-16.el7.x86_64 already installed and latest version Package fence-virtd-serial-0.3.0-16.el7.x86_64 already installed and latest version Nothing to do [root@UNIXKB-CP ~]#
4. Create the new directory to store the fence key. Create the random key to use for fencing.
[root@UNIXKB-CP ~]# mkdir -p /etc/cluster [root@UNIXKB-CP ~]# cd /etc/cluster/ [root@UNIXKB-CP cluster]# dd if=/dev/urandom of=/etc/cluster/fence_xvm.key bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.000506736 s, 8.1 MB/s [root@UNIXKB-CP cluster]#
5. Copy the fence keys to cluster nodes. (Node1 & Node2)
[root@UNIXKB-CP cluster]# scp -r /etc/cluster/fence_xvm.key root@Node1:/etc/cluster/fence_xvm.key root@node1's password: fence_xvm.key 100% 4096 4.0KB/s 00:00 [root@UNIXKB-CP cluster]# scp -r /etc/cluster/fence_xvm.key root@Node2:/etc/cluster/fence_xvm.key root@node2's password: fence_xvm.key 100% 4096 4.0KB/s 00:00 [root@UNIXKB-CP cluster]#
Note: You must create a “/etc/cluster” directory on the cluster nodes in an order to copy the xvm keys.
6.Use “fence_virtd -c” command to create “/etc/fence_virt.conf” file.
[root@UNIXKB-CP ~]# fence_virtd -c
Module search path [/usr/lib64/fence-virt]:
Available backends:
libvirt 0.1
Available listeners:
multicast 1.2
Listener modules are responsible for accepting requests
from fencing clients.
Listener module [multicast]:
The multicast listener module is designed for use environments
where the guests and hosts may communicate over a network using
multicast.
The multicast address is the address that a client will use to
send fencing requests to fence_virtd.
Multicast IP Address [225.0.0.12]:
Using ipv4 as family.
Multicast IP Port [1229]:
Setting a preferred interface causes fence_virtd to listen only
on that interface. Normally, it listens on all interfaces.
In environments where the virtual machines are using the host
machine as a gateway, this *must* be set (typically to virbr0).
Set to 'none' for no interface.
Interface [virbr0]: br0:1
The key file is the shared key information which is used to
authenticate fencing requests. The contents of this file must
be distributed to each physical host and virtual machine within
a cluster.
Key File [/etc/cluster/fence_xvm.key]:
Backend modules are responsible for routing requests to
the appropriate hypervisor or management layer.
Backend module [libvirt]:
Configuration complete.
=== Begin Configuration ===
backends {
libvirt {
uri = "qemu:///system";
}
}
listeners {
multicast {
port = "1229";
family = "ipv4";
interface = "br0:1";
address = "225.0.0.12";
key_file = "/etc/cluster/fence_xvm.key";
}
}
fence_virtd {
module_path = "/usr/lib64/fence-virt";
backend = "libvirt";
listener = "multicast";
}
=== End Configuration ===
Replace /etc/fence_virt.conf with the above [y/N]? y
[root@UNIXKB-CP ~]#
Make sure that you are proving the correct interface as the bridge. In My setup , I am using br0:1 virtual interface to communicate with KVM guests.
7. Start the fence_virtd service.
[root@UNIXKB-CP ~]# systemctl enable fence_virtd.service
[root@UNIXKB-CP ~]# systemctl start fence_virtd.service
[root@UNIXKB-CP ~]# systemctl status fence_virtd.service
fence_virtd.service - Fence-Virt system host daemon
Loaded: loaded (/usr/lib/systemd/system/fence_virtd.service; enabled)
Active: active (running) since Wed 2016-01-20 23:36:14 IST; 1s ago
Process: 3530 ExecStart=/usr/sbin/fence_virtd $FENCE_VIRTD_ARGS (code=exited, status=0/SUCCESS)
Main PID: 3531 (fence_virtd)
CGroup: /system.slice/fence_virtd.service
└─3531 /usr/sbin/fence_virtd -w
Jan 20 23:36:14 UNIXKB-CP systemd[1]: Starting Fence-Virt system host daemon...
Jan 20 23:36:14 UNIXKB-CP systemd[1]: Started Fence-Virt system host daemon.
Jan 20 23:36:14 UNIXKB-CP fence_virtd[3531]: fence_virtd starting. Listener: libvirt Backend: multicast
[root@UNIXKB-CP ~]#
Configure the Fencing on Cluster Nodes:
1.Login to one of the cluster node.
2.Make sure that both the nodes have “fence_virt” package.
[root@Node1-LAB ~]# rpm -qa fence-virt fence-virt-0.3.0-16.el7.x86_64 [root@Node1-LAB ~]#
3. The following commands much be scuesscced in an order to configure the fencing in cluster.
[root@Node1-LAB ~]# fence_xvm -o list Node1 6daac670-c494-4e02-8d90-96cf900f2be9 on Node2 17707dcb-7bcc-4b36-9498-a5963d86dc2f on [root@Node1-LAB ~]#
4.Cluster nodes entry must be present in the /etc/hosts.
[root@Node1-LAB ~]# cat /etc/hosts |grep Node 192.168.2.10 Node1-LAB Node1 192.168.2.11 Node2-LAB Node2 [root@Node1-LAB ~]#
5.Configure fence_xvm fence agent on pacemaker cluster.
[root@Node1-LAB ~]# pcs stonith create xvmfence fence_xvm key_file=/etc/cluster/fence_xvm.key [root@Node1-LAB ~]# [root@Node1-LAB ~]# pcs stonith xvmfence (stonith:fence_xvm): Started [root@Node1-LAB ~]# [root@Node1-LAB ~]# pcs stonith --full Resource: xvmfence (class=stonith type=fence_xvm) Attributes: key_file=/etc/cluster/fence_xvm.key Operations: monitor interval=60s (xvmfence-monitor-interval-60s) [root@Node1-LAB ~]#
We have successfully configure the fencing on RHEL 7 – Pacemaker/Corosync cluster. (Cluster has been configured between two KVM guests).
Validate the STONITH:
How should I test my “stonith” configuration ? Here is the small demonstration.
1. Login to the one of the cluster node.
2. Try to fence on of the node.
[root@Node1-LAB ~]# pcs stonith fence Node2 Node: Node2 fenced [root@Node1-LAB ~]#
This will eventually reboot Node2 . Reboot happens based on cluster property.
[root@Node1-LAB ~]# pcs property --all |grep stonith-action stonith-action: reboot [root@Node1-LAB ~]#
Stonith also can be ON/OFF using pcs property command.
[root@Node1-LAB ~]# pcs property --all |grep stonith-enabled stonith-enabled: true [root@Node1-LAB ~]#
Hope this article is informative to you.