Home / Pacemaker / Configuring NFS HA using Redhat Cluster – Pacemaker on RHEL 7

Configuring NFS HA using Redhat Cluster – Pacemaker on RHEL 7

This article will help you to setup High Availability NFS server using Pacemaker on Redhat Enterprise Linux 7. From the scratch ,we will build the pacemaker blocks which includes package installation , configuring the HA resources, fencing etc. NFS shares are used for setting up the home directories and sharing the same content across multiple servers. NFS HA will suit for customers who can’t afford NAS storage. You might have followed Pacemaker articles on UnixArena where we have setup Failover KVM VM and GFS earlier. If not , please go through it to understand the various component of pacemaker and how it works. This article is not going to cover in-depth.

 

NFS HA - Pacemaker UnixArena
NFS HA – Pacemaker UnixArena

 

Assumptions:

  • Two servers installed with RHEL 7.x (Hosts- UA-HA1 / UA-HA2)
  • Access to Redhat Repository or Local Repository to install packages.
  • SELINUX & Firewalld can be turned off.

 

1. Login to the each node as root user and install the package .

# yum install pcs fence-agents-all

 

2. Disable SELINUX on both the nodes.

# setenforce 0
setenforce: SELinux is disabled
# cat /etc/selinux/config |grep SELINUX |grep -v "#"
SELINUX=disabled
SELINUXTYPE=targeted

 

3. Disable firewalld on both the hosts

UA-HA# systemctl stop firewalld.service
UA-HA# systemctl disable firewalld.service
UA-HA# iptables --flush
UA-HA#

 

4. Enable and Start the Services on both the Nodes.

# systemctl start pcsd.service
# systemctl enable pcsd.service
# systemctl status pcsd.service

 

5. On each nodes, set the password for hauser.

# passwd hacluster
Changing password for user hacluster.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.

 

6.Login to any of the cluster node and authenticate “hacluster” user.

# pcs cluster auth UA-HA1 UA-HA2

 

7. Create a new cluster using pcs command. The cluster name is “UACLS”.

# pcs cluster setup --name UACLS UA-HA1 UA-HA2

 

8. Start the cluster using pcs command. “–all” will start the cluster on all the configured nodes.

# pcs cluster start –all

 

9. Check the corosync communication status.This command output should show that which IP has been used for heartbeat. Refer configure-redundant-corosync
# corosync-cfgtool –s

 

10. Disable STONITH to avoid issues while configuring the resources. Once we complete the cluster setup, we will enable the fencing back.

#pcs property set stonith-enabled=false
#pcs property show stonith-enabled

 

10. Configure the fencing (STONITH) using ipmilan.

# pcs stonith create UA-HA1_fen fence_ipmilan pcmk_host_list="UA-HA1" ipaddr=192.168.10.24 login=root  passwd=test123 lanplus=1 cipher=1 op monitor interval=60s
#pcs stonith create UA-HA2_fen fence_ipmilan pcmk_host_list="UA-HA2" ipaddr=192.168.10.25 login=root  passwd=test123 lanplus=1 cipher=1 op monitor interval=60s

These IP’s are IDRAC console IP used for fencing.

 

11. Verify the cluster configuration.

# crm_verify -L –V

 

12. Configure the volume group and Logical volume.

# vgcreate UAVG1 /dev/disk_name1
# vgcreate UAVG2 /dev/disk_#name2

# lvcreate -L sizeM -n /dev/UAVG1/UAVOL1
# lvcreate -L sizeM -n /dev/UAVG2/UAVOL2

 

12. Create a filesystem. (Let’s go with XFS)

#mkfs.xfs /dev/UAVG1/UAVOL1
#mkfs.xfs /dev/UAVG2/UAVOL2

 

13. Modify LVM configuration similar to below. Assuming that all the volume groups are used in cluster. If you have root vg , you need to specify in lvm.conf for automatic import.

# grep use_lvmetad /etc/lvm/lvm.conf |grep -v "#"
use_lvmetad = 0

 

14.Configure symmetric cluster property and check the status.

# pcs property set symmetric-cluster=true
[root@UA-HA1 tmp]# pcs status
Cluster name: UACLS
Stack: corosync
Current DC: UA-HA2 (2) - partition with quorum
2 Nodes configured
2 Resources configured

Online: [ UA-HA1 UA-HA2 ]

Full list of resources:

 UA-HA1_fen   (stonith:fence_ipmilan):        Started UA-HA2
 UA-HA2_fen   (stonith:fence_ipmilan):        Started UA-HA1

PCSD Status:
  UA-HA1: Online
  UA-HA2: Online

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled
[root@UA-HA1 tmp]# pcs stonith show
 UA-HA1_fen   (stonith:fence_ipmilan):        Started
 UA-HA2_fen   (stonith:fence_ipmilan):        Started
[root@UA-HA1 tmp]#

 

15.Configure VG & Mount resources .

#pcs resource create UAVG1_res LVM volgrpname="UAVG1" exclusive=true  --group UANFSHA
#pcs resource create UAVOL1_res Filesystem  device="/dev/UAVG1/UAVOL1" directory="/cm/shared" fstype="xfs" --group UANFSHA


#pcs resource create UAVG2_res LVM volgrpname="UAVG2" exclusive=true --group  UANFSHA
#pcs resource create UAVOL2_res Filesystem  device="/dev/UAVG2/UAVOL2" directory="/global/home" fstype="xfs" --group UANFSHA

 

16.Configure VIP for NFS share. This IP will be used on NFS client to mount the shares.

# pcs resource create ClusterIP ocf:heartbeat:IPaddr2 ip=192.168.2.90  nic="eth0" cidr_netmask=24 op monitor interval=30s --group UANFSHA

 

17. Configure NFS server resources.

[root@UA-HA1 ~]# pcs resource create NFS-D nfsserver nfs_shared_infodir=/global/nfsinfo nfs_ip=192.168.2.90  --group UANFSHA

 

18.Check the cluster status.

[root@UA-HA1 ~]# pcs status
Cluster name: UACLS
Last updated: Tue Aug 16 12:39:22 2016
Last change: Tue Aug 16 12:39:19 2016 via cibadmin on UA-HA1
Stack: corosync
Current DC: UA-HA1 (1) - partition with quorum
Version: 1.1.10-29.el7-368c726
2 Nodes configured
8 Resources configured

Online: [ UA-HA1 UA-HA2 ]

Full list of resources:

 UA-HA1_fen   (stonith:fence_ipmilan):        Started UA-HA1
 UA-HA2_fen   (stonith:fence_ipmilan):        Started UA-HA1
 Resource Group: UANFSHA
     UAVG1_res  (ocf::heartbeat:LVM):   Started UA-HA1
     UAVG2_res  (ocf::heartbeat:LVM):   Started UA-HA1
     UAVOL1_res  (ocf::heartbeat:Filesystem):    Started UA-HA1
     UAVOL2_res  (ocf::heartbeat:Filesystem):    Started UA-HA1
     ClusterIP  (ocf::heartbeat:IPaddr2):       Started UA-HA1
     NFS-D      (ocf::heartbeat:nfsserver):     Started UA-HA1 

 

19. Configure the HA NFS shares.

[root@UA-HA1 ~]# pcs resource create nfs-cm-shared exportfs clientspec=192.168.2.0/255.255.255.0 options=rw,sync,no_root_squash directory=/SAP_SOFT fsid=0 --group UANFSHA
[root@UA-HA1 ~]# pcs resource create nfs-global-home  exportfs clientspec=10.248.102.0/255.255.255.0 options=rw,sync,no_root_squash directory=/users1/home fsid=1 --group UANFSHA

 

20.Final cluster status will looks like similar to the following.

[root@UA-HA1 ~]# pcs status
Cluster name: UACLS
Last updated: Tue Aug 16 12:52:43 2016
Last change: Tue Aug 16 12:51:56 2016 via cibadmin on UA-HA1
Stack: corosync
Current DC: UA-HA1 (1) - partition with quorum
Version: 1.1.10-29.el7-368c726
2 Nodes configured
10 Resources configured


Online: [ UA-HA1 UA-HA2 ]

Full list of resources:

 UA-HA1_fen   (stonith:fence_ipmilan):        Started UA-HA1
 UA-HA2_fen   (stonith:fence_ipmilan):        Started UA-HA1
 Resource Group: UANFSHA
     UAVG1_res  (ocf::heartbeat:LVM):   Started UA-HA1
     UAVG2_res  (ocf::heartbeat:LVM):   Started UA-HA1
     UAVOL1_res  (ocf::heartbeat:Filesystem):    Started UA-HA1
     UAVOL2_res  (ocf::heartbeat:Filesystem):    Started UA-HA1
     ClusterIP  (ocf::heartbeat:IPaddr2):       Started UA-HA1
     NFS-D      (ocf::heartbeat:nfsserver):     Started UA-HA1
     nfs-cm-shared      (ocf::heartbeat:exportfs):      Started UA-HA1
     nfs-global-home    (ocf::heartbeat:exportfs):      Started UA-HA1

 

21. Configure resource dependencies.

[root@UA-HA1 ~]# pcs constraint order start UAVG1_res then UAVOL1_res
[root@UA-HA1 ~]# pcs constraint order start UAVG2_res then UAVOL2_res
[root@UA-HA1 ~]# pcs constraint order start UAVOL1_res then ClusterIP
[root@UA-HA1 ~]# pcs constraint order start UAVOL2_res then ClusterIP 
[root@UA-HA1 ~]# pcs constraint order start ClusterIP then NFS-D
[root@UA-HA1 ~]# pcs  constraint order start NFS-D then nfs-cm-shared
[root@UA-HA1 ~]# pcs  constraint order start NFS-D then nfs-global-home
[root@UA-HA1 ~]# pcs constraint
Location Constraints:
  Resource: UANFSHA
    Enabled on: UA-HA1 (role: Started)
Ordering Constraints:
  start UAVG1_res then start UAVOL1_res
  start UAVG2_res then start UAVOL2_res
  start UAVOL1_res then start ClusterIP
  start UAVOL2_res then start ClusterIP
  start ClusterIP then start NFS-D
  start NFS-D then start nfs-cm-shared
  start NFS-D then start nfs-global-home
Colocation Constraints:
[root@UA-HA1 ~]#

 

22. You can also verify the NFS shares using the following command. (Have to execute where the resources are running currently)

[root@UA-HA1 ~]# showmount -e 192.168.2.90
Export list for 192.168.2.90:
/SAP_SOFT   192.168.2.0/255.255.255.0
/users1/home 192.168.2.0/255.255.255.0
[root@UA-HA1 ~]#

[root@UA-HA1 ~]# ifconfig eth0
ib0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 2044
        inet 192.168.2.90  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::7efe:9003:a7:851  prefixlen 64  scopeid 0x20

 

23. Enable the STONITH.

#pcs property set stonith-enabled=true
#pcs property show stonith-enabled

 

24. Login to NFS clients and mount the shares .

# mkdir /users1/home 
# mkdir /SAP_SOFT
# mount -t nfs -o vers=4 192.168.2.90:/SAP_SOFT  /SAP_SOFT
# mount -t nfs -o vers=4 192.168.2.90:/users1/home  /users1/home

We have successfully setup high availability NFS server v4 using pacemaker cluster suite on Redhat Enterprise Linux 7.x.

If you have any trouble with resources , use the following command to clear the state. Resource might be automatically banned if its faulted more than one twice.

 [root@UA-HA1 init.d]# pcs resource clear UANFSHA

Hope this article is informative to you. Share it ! Comment it ! Be Sociable !!!

VMTURBO-CLOUD-CAPACITY

3 comments

  1. Hi,

    I found this article when searching about problems with using NFS with Pacemaker in RHEL 7.2.

    I just wanted to point out that I believe you have missed out the nfsnotify resource in the NFS setup. my understanding is that this is definitely required for a properly functioning HA NFS set up.

    The article is well written and will, I’m sure, be of use to many people but it needs to be correct so please can you update it taking my comments into account.

    Thanks,
    Phil.

  2. Hi,
    really nice article. I follow these procedures and I’m facing this issue:
    In the beginning, having one export, everything works fine. I cold reset the nodes one after another and the services are migrated each time successfuly to the other node. But when I add another export directory (with different fsid than the first one), after the first reboot of the active node, the NFS server does not start on one node or the other. The error that I get is that “rpcbind is not running”. While tailing /var/log/messages I see a repeating message of:
    nfsserver: INFO: Start: rpcbind i:1
    nfsserver: INFO: Start: rpcbind i:2
    nfsserver: INFO: Start: rpcbind i:3
    nfsserver: INFO: Start: rpcbind i:4

    and so one.

    After this, the nfs service never starts again on neither node.

    Thanks

    • To add to the above issue, after a fresh restart of both nodes, when I try to add an nfs server resource again. the error that I get is:
      “Failed to start NFS server: /proc/fs/nfsd/threads”. In the /var/log/messages folder I get: ERROR: nfs-mountd is not running.
      Thanks

Leave a Reply

Your email address will not be published. Required fields are marked *