Home / Pacemaker / RHEL 7 – Pacemaker – Cluster Resources/Group Management – Part 6

RHEL 7 – Pacemaker – Cluster Resources/Group Management – Part 6

In Pacemaker/Corosync  cluster (RHEL 7 HA),  resources management and resource group management are important tasks . Depends on the cluster HA services, you might need to configure N-number of resources. In most of the cases , you might need to start set of resources sequentially, and stop in the reverse order.  To simplify this configuration, Pacemaker supports the concept of groups (Resource groups). For an example, to provide the web-services in HA model, you need resources like , File system(To store website data) , IP (Clustered IP to access website) and Apache (To provide the web-services) . To start the Apache service , you need a filesystem which stores the website data. So the resources must start in the following order ,

  1. IP
  2. File-system
  3. Apache service

 

Let’s see that how to configure the Highly available Apache service (website) in Redhat cluster (Pacemaker/Corosync).  In the previous article, we have already created the IP resource.

[root@UA-HA ~]# pcs status
Cluster name: UABLR
Last updated: Mon Dec 28 18:24:10 2015          Last change: Mon Dec 28 18:09:30 2015 by root via crm_resource on UA-HA
Stack: corosync
Current DC: UA-HA2 (version 1.1.13-10.el7-44eb2dd) - partition with quorum
2 nodes and 1 resource configured

Online: [ UA-HA UA-HA2 ]

Full list of resources:

ClusterIP  (ocf::heartbeat:IPaddr2):       Started UA-HA

PCSD Status:
  UA-HA: Online
  UA-HA2: Online

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled
[root@UA-HA ~]# pcs resource show ClusterIP
 Resource: ClusterIP (class=ocf provider=heartbeat type=IPaddr2)
  Attributes: ip=192.168.203.190 cidr_netmask=24
  Operations: start interval=0s timeout=20s (ClusterIP-start-interval-0s)
              stop interval=0s timeout=20s (ClusterIP-stop-interval-0s)
              monitor interval=30s (ClusterIP-monitor-interval-30s)
[root@UA-HA ~]#

 

Create the File-system and  Apache resources quickly:

 

Filesystem : 

  • Shared LUN – /dev/sdc
  • Volume Group – webvg
  • Volume – webvol1
  • Filesystem Type – ext4

 

Quick Setup for Filesystem resource: 

[root@UA-HA2 ~]# vgcreate webvg /dev/sdc
[root@UA-HA2 ~]# lvcreate -L 90M -n /dev/webvg/webvol1
[root@UA-HA2 ~]# mkfs.ext4 /dev/webvg/webvol1

 

Apache:

  • httpd

Quick Setup:

[root@UA-HA www]# yum install -y httpd

 

Pre-prerequisites for LVM :

(Perform the following changes on both the cluster nodes)

1.Make sure that “use_lvmetad” parameter is set to “0”. This is mandatory when you use “Pacemaker”.

[root@UA-HA ~]# grep use_lvmetad /etc/lvm/lvm.conf |grep -v "#"
    use_lvmetad = 0
[root@UA-HA ~]#

 

2.To prevent the automatic volume group activation, update the volume_list parameter with local VG’s which needs to be activated automatically.

[root@UA-HA ~]# grep volume_list /etc/lvm/lvm.conf |grep -v "#"
        volume_list = [ "nfsvg", "rhel" ]
[root@UA-HA ~]# vgs
  VG    #PV #LV #SN Attr   VSize  VFree
  nfsvg   2   1   0 wz--n-  1.94g 184.00m
  rhel    1   2   0 wz--n- 19.51g      0
  webvg   1   1   0 wz--n- 92.00m      0
[root@UA-HA ~]#

In My case, “webvg” will be managed through cluster.

 

3. Mount the volume in “/var/www”  and create the following directories and files.

[root@UA-HA2 ~]# mount /dev/webvg/webvol1 /var/www
[root@UA-HA2 ~]# cd /var/www
[root@UA-HA2 www]# mkdir errror html cgi-bin
total 3
drwxr-xr-x 2 root root 1024 Dec 28 20:26 cgi-bin
drwxr-xr-x 2 root root 1024 Dec 28 20:26 errror
drwxr-xr-x 2 root root 1024 Dec 28 20:27 html
[root@UA-HA2 www]# cd html/
[root@UA-HA2 html]# vi index.html
Hello, Welcome to UnixArena 

[root@UA-HA2 html]#

 

3.Rebuild the “initramfs” boot image to guarantee that the boot image will not try to activate a volume group controlled by the cluster. Update the initramfs device using the following command.

[root@UA-HA ~]# dracut -H -f /boot/initramfs-$(uname -r).img $(uname -r)
[root@UA-HA ~]#

 

4. Reboot the nodes.

 

 

Create the LVM Cluster resources (vg & lv ), File-system cluster Resources:

 

1.Create the cluster volume group resource.

[root@UA-HA ~]# pcs resource create vgres LVM volgrpname=webvg exclusive=true
[root@UA-HA ~]# pcs resource show vgres
 Resource: vgres (class=ocf provider=heartbeat type=LVM)
  Attributes: volgrpname=webvg exclusive=true
  Operations: start interval=0s timeout=30 (vgres-start-interval-0s)
              stop interval=0s timeout=30 (vgres-stop-interval-0s)
              monitor interval=10 timeout=30 (vgres-monitor-interval-10)
[root@UA-HA ~]#

vgresResource Name (Any Unique Name)
webvgVolume Group

 

2. Create the cluster mount resource.

[root@UA-HA ~]# pcs resource create webvolfs Filesystem  device="/dev/webvg/webvol1" directory="/var/www" fstype="ext4"
[root@UA-HA ~]# pcs resource show webvolfs
 Resource: webvolfs (class=ocf provider=heartbeat type=Filesystem)
  Attributes: device=/dev/webvg/webvol1 directory=/var/www fstype=ext4
  Meta Attrs: 
  Operations: start interval=0s timeout=60 (webvolfs-start-interval-0s)
              stop interval=0s timeout=60 (webvolfs-stop-interval-0s)
              monitor interval=20 timeout=40 (webvolfs-monitor-interval-20)
[root@UA-HA ~]#

 

3. Before adding the resource, you must update the local /etc/httpd/conf/httpd.conf with following contents. This entries required for pacemaker to get the web-server status .

Update apache conf
Update apache conf

 

4. Check the apache server status . (httpd.service). Make sure that httpd.service is stopped & disabled on both the cluster nodes. This service will be managed by cluster.

[root@UA-HA ~]# systemctl status httpd.service
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
     Docs: man:httpd(8)
           man:apachectl(8)

Dec 27 13:55:52 UA-HA systemd[1]: Starting The Apache HTTP Server...
Dec 27 13:55:55 UA-HA httpd[2002]: AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 192.168.203.134. Set the...is message
Dec 27 13:55:55 UA-HA systemd[1]: Started The Apache HTTP Server.
Dec 27 15:16:02 UA-HA httpd[11786]: AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 192.168.203.134. Set th...is message
Dec 27 15:16:02 UA-HA systemd[1]: Reloaded The Apache HTTP Server.
Dec 28 18:06:57 UA-HA systemd[1]: Started The Apache HTTP Server.
Dec 28 20:30:56 UA-HA systemd[1]: Stopping The Apache HTTP Server...
Dec 28 20:30:57 UA-HA systemd[1]: Stopped The Apache HTTP Server.
Hint: Some lines were ellipsized, use -l to show in full.
[root@UA-HA ~]#

 

3. Create the Apache cluster resource.

[root@UA-HA ~]# pcs resource create webres apache configfile="/etc/httpd/conf/httpd.conf" statusurl="http://127.0.0.1/server-status"
[root@UA-HA ~]# pcs status
Cluster name: UABLR
Last updated: Mon Dec 28 20:11:51 2015          Last change: Mon Dec 28 20:11:44 2015 by root via cibadmin on UA-HA
Stack: corosync
Current DC: UA-HA2 (version 1.1.13-10.el7-44eb2dd) - partition with quorum
2 nodes and 4 resources configured

Online: [ UA-HA UA-HA2 ]

Full list of resources:

 vgres  (ocf::heartbeat:LVM):   (target-role:Stopped) Stopped
 webvolfs       (ocf::heartbeat:Filesystem):    (target-role:Stopped) Stopped
 ClusterIP      (ocf::heartbeat:IPaddr2):       Started UA-HA2
 webres (ocf::heartbeat:apache):        Stopped

PCSD Status:
  UA-HA: Online
  UA-HA2: Online

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled
[root@UA-HA ~]#

 

In normal cases, resource group will be created (by specifying –group in the end of command line) when you add the first cluster resource to make the dependency tree. To understand the cluster resources and resource group management concept , I am creating the resource group at the end.

 

If you see any resource was started , just stop it to avoid the errors.

[root@UA-HA ~]# pcs resource disable vgres webvolfs webres stop ClusterIP      
[root@UA-HA ~]# pcs resource
 vgres  (ocf::heartbeat:LVM):                    Stopped
 webvolfs       (ocf::heartbeat:Filesystem):     Stopped
 ClusterIP      (ocf::heartbeat:IPaddr2):        Stopped
 webres (ocf::heartbeat:apache):                 Stopped
[root@UA-HA ~]#

 

4. Create the resource group to form the resource dependencies to stop & start in resources in sequence.

[root@UA-HA ~]# pcs resource group add WEBRG1 ClusterIP vgres webvolfs webres

 

As per the above command , here is the resource start up sequence

  1. ClusterIP – Website URL
  2. vgres – Volume Group
  3. webvolfs – Mount Resource
  4. webres – httpd Resource

 

Stop sequence is just reverse to the start.

  1. webres – httpd Resource
  2. webvolfs – Mount Resource
  3. vgres – Volume Group
  4. ClusterIP – Website URL

 

5. Check the resources status. You should be able to see that all the resources are bundled as one resource group with  “WEBRG1” .

[root@UA-HA ~]# pcs resource
 Resource Group: WEBRG1
     ClusterIP  (ocf::heartbeat:IPaddr2):       Started UA-HA2
     vgres      (ocf::heartbeat:LVM):            Stopped
     webvolfs   (ocf::heartbeat:Filesystem):     Stopped
     webres     (ocf::heartbeat:apache):         Stopped
[root@UA-HA ~]#

 

6. Enable the disabled resources in following sequence.

[root@UA-HA ~]# pcs resource enable ClusterIP
[root@UA-HA ~]# pcs resource enable vgres
[root@UA-HA ~]# pcs resource enable webvolfs
[root@UA-HA ~]# pcs resource enable webres

 

7. Verify the cluster status.

[root@UA-HA ~]# pcs status
Cluster name: UABLR
Last updated: Mon Dec 28 20:54:43 2015          Last change: Mon Dec 28 20:51:30 2015 by root via crm_resource on UA-HA2
Stack: corosync
Current DC: UA-HA2 (version 1.1.13-10.el7-44eb2dd) - partition with quorum
2 nodes and 4 resources configured

Online: [ UA-HA UA-HA2 ]

Full list of resources:

 Resource Group: WEBRG1
     ClusterIP  (ocf::heartbeat:IPaddr2):       Started UA-HA2
     vgres      (ocf::heartbeat:LVM):   Started UA-HA2
     webvolfs   (ocf::heartbeat:Filesystem):    Started UA-HA2
     webres     (ocf::heartbeat:apache):        Started UA-HA2

PCSD Status:
  UA-HA: Online
  UA-HA2: Online

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled
[root@UA-HA ~]#

 

8. Let’s move the resources from UA-HA2 to UA-HA. In this case, we no need to move each resources manually.We just need to move the Resource group since we have bundled the required resource in to that.

[root@UA-HA ~]# pcs resource move WEBRG1 UA-HA
[root@UA-HA ~]# pcs status
Cluster name: UABLR
Last updated: Mon Dec 28 20:58:55 2015          Last change: Mon Dec 28 20:58:41 2015 by root via crm_resource on UA-HA
Stack: corosync
Current DC: UA-HA2 (version 1.1.13-10.el7-44eb2dd) - partition with quorum
2 nodes and 4 resources configured

Online: [ UA-HA UA-HA2 ]

Full list of resources:

 Resource Group: WEBRG1
     ClusterIP  (ocf::heartbeat:IPaddr2):       Started UA-HA
     vgres      (ocf::heartbeat:LVM):   Started UA-HA
     webvolfs   (ocf::heartbeat:Filesystem):    Started UA-HA
     webres     (ocf::heartbeat:apache):        Started UA-HA

PCSD Status:
  UA-HA: Online
  UA-HA2: Online

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled
[root@UA-HA ~]#

 

You should be able to see the webpage like following .

Website Portal
Website Portal

 

9. How to stop the pacemaker resource group ? Just disable the resource group.

[root@UA-HA2 ~]# pcs resource disable WEBRG1
[root@UA-HA2 ~]# pcs status
Cluster name: UABLR
Last updated: Mon Dec 28 21:12:18 2015          Last change: Mon Dec 28 21:12:14 2015 by root via crm_resource on UA-HA2
Stack: corosync
Current DC: UA-HA2 (version 1.1.13-10.el7-44eb2dd) - partition with quorum
2 nodes and 4 resources configured

Online: [ UA-HA UA-HA2 ]

Full list of resources:

 Resource Group: WEBRG1
     ClusterIP  (ocf::heartbeat:IPaddr2):       (target-role:Stopped) Stopped
     vgres      (ocf::heartbeat:LVM):   (target-role:Stopped) Stopped
     webvolfs   (ocf::heartbeat:Filesystem):    (target-role:Stopped) Stopped
     webres     (ocf::heartbeat:apache):        (target-role:Stopped) Stopped

PCSD Status:
  UA-HA: Online
  UA-HA2: Online

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled
[root@UA-HA2 ~]#

 

10. How to start the resource group ? Use enable option for the RG.

[root@UA-HA2 ~]# pcs resource enable WEBRG1
[root@UA-HA2 ~]# pcs status
Cluster name: UABLR
Last updated: Mon Dec 28 21:14:04 2015          Last change: Mon Dec 28 21:14:01 2015 by root via crm_resource on UA-HA2
Stack: corosync
Current DC: UA-HA2 (version 1.1.13-10.el7-44eb2dd) - partition with quorum
2 nodes and 4 resources configured

Online: [ UA-HA UA-HA2 ]

Full list of resources:

 Resource Group: WEBRG1
     ClusterIP  (ocf::heartbeat:IPaddr2):       Started UA-HA2
     vgres      (ocf::heartbeat:LVM):   Started UA-HA2
     webvolfs   (ocf::heartbeat:Filesystem):    Started UA-HA2
     webres     (ocf::heartbeat:apache):        Started UA-HA2

PCSD Status:
  UA-HA: Online
  UA-HA2: Online

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled
[root@UA-HA2 ~]#

 

Note:
Redhat cluster (Pacemaker/corosync) have many parameters like resource stickiness and failure counts.These attributes will play a role that where to start the resources.

 

To clear the errors , use the following command

# pcs resource cleanup 

 

To clear the resource fail counts , use the following command.

 [root@UA-HA2 ~]# pcs resource clear ClusterIP
[root@UA-HA2 ~]# pcs resource clear vgres
[root@UA-HA2 ~]# pcs resource clear webvolfs
[root@UA-HA2 ~]# pcs resource clear webres
[root@UA-HA2 ~]#

 

Hope this article is informative to you.

 

Share it ! Comment it !! Be Sociable !!!

VMTURBO-CLOUD-CAPACITY

One comment

  1. in this post you have not mentioned anything about the DLM and CLVM like it is necessary for GFS2 filesystem, in this post you have taken EXT4 as a filesystem so does this mean that we don’t need to use DLM and CLVM in the pacemaker cluster resource for EXT4 filesystem.

Leave a Reply

Your email address will not be published. Required fields are marked *