I have been trying from so long time to add the KVM virtual guest under redhat cluster for High availability virtual guest.So that virtual machines can automatically move from one node to another if anything happen to the parent server.Its similar to VMWARE ESX cluster where virtual machines can migrate automatically from one server to another. After long try , I have successfully configured the service for KVM guest and its working like charm. Using my blogger platform i got an opportunity to share the same with you.
Prerequisite:
1.Install KVM package on Cluster nodes to create a virtual machine.
2.Install Redhat cluster software on both the cluster nodes.
3.Configure the Redhat Cluster from LUCI
4.Install CLVM2 and GFS2 package and create a new shared filesystem
5.Create a new KVM guest in any one of the cluster node on the shared filesystem
6.Configure the ssh keyless authentication between two Linux nodes for root.
Once you have everything in place, you can start adding the KVM guest in to our redhat cluster.
Info:
Redhat Cluster Node names : uagl1h, uagl2h (OS: Redhat Linux 6.3)
KVM virtual Machine Name: VMG (OS: Redhat Linux 6.3)
Cluster Name: hacl1
Service Name:VMG
1. Login to luci console and navigate to cluster.(Prerequisite:3)
2.Make sure you have configured the failover domain with two cluster nodes.
It just simple that by clicking the add button.Here the failover the domain name is “HAVM”.
3.From the services tab,click “+” to add the new service.Enter the service name as “VMG” and click the failover domain as “HAVM”. You can also select the recovery ploicy as you desired. Its up to you to provide the details about Maximum number of restart and length of time in seconds.Once its done, then click on “Add a resource” tab.
4.Click on the “Global resources” drop down list and select “Virtual Machine” from that list.
5.Enter the shared volume name in “Path to XML file use to create VM”. (/havol1)
6. Click on submit to complete the KVM guest addition in to redhat cluster.
7.By default , the service(VMG) will be in disabled state.The above image was taken after enabling the service from redhat cluster.
8.Let us see the status from the command line.
[root@uagl1 ~]# clustat
Cluster Status for uacl1 @ Thu Oct 24 19:36:44 2013
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
uagl1h 1 Online, Local, rgmanager
uagl2h 2 Online, rgmanager
Service Name Owner (Last) State
------- ---- ----- ------ -----
vm:VMG (uagl1h) disabled
9.Start the “VMG” service on uagl1h node .
# clusvcadm -e vm:VMG -m uagl1
Member uagl1h trying to enable vm:VMG...Success
vm:VMG is now running on uagl1h
[root@uagl1 ~]#
10.Check the cluster status again from using clustat command.
[root@uagl1 ~]# clustat
Cluster Status for uacl1 @ Thu Oct 24 19:39:46 2013
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
uagl1h 1 Online, Local, rgmanager
uagl2h 2 Online, rgmanager
Service Name Owner (Last) State
------- ---- ----- ------ -----
vm:VMG uagl1h started
11.Verify the cluster machine status using KVM commands.
[root@uagl1 New Volume]# virsh list
Id Name State
----------------------------------------------------
26 VMG running
[root@uagl1 New Volume]# ssh uagl2h virsh list
Id Name State
----------------------------------------------------
[root@uagl1 New Volume]#
We have successfully added the KVM guest in to redhat cluster and started the virtual machine via redhat cluster.
Troubleshooting:
I have faced lot issue to get the above results.After adding virtual machine , Redhat cluster was unable to boot the KVM guest machine.
Issue:1
In Step no:5, I have used “VM configuration file path” instead of using “path to xmlfile used to create the VM” . So what happened that redhat cluster assumed the virtual machine type is “Xen” and its not initiating the KVM commands in the background.
Oct 22 23:01:42 uagl1 rgmanager[25959]: vm:VMG was added to the config, but I am not initializing it.
Oct 22 23:01:57 uagl1 rgmanager[25959]: Starting disabled service vm:VMG
Oct 22 23:01:58 uagl1 rgmanager[25959]: start on vm "VMG" returned 1 (generic error)
Oct 22 23:01:58 uagl1 rgmanager[25959]: #68: Failed to start vm:VMG; return value: 1
Oct 22 23:01:58 uagl1 rgmanager[25959]: Stopping service vm:VMG
Oct 22 23:01:58 uagl1 rgmanager[25959]: Service vm:VMG is recovering
Oct 22 23:01:58 uagl1 rgmanager[25959]: #71: Relocating failed service vm:VMG
Oct 22 23:02:00 uagl1 rgmanager[25959]: Service vm:VMG is stopped
Oct 22 23:02:48 uagl1 rgmanager[25959]: Starting stopped service vm:VMG
Oct 22 23:02:48 uagl1 rgmanager[25959]: start on vm "VMG" returned 1 (generic error)
Oct 22 23:02:48 uagl1 rgmanager[25959]: #68: Failed to start vm:VMG; return value: 1
Oct 22 23:02:48 uagl1 rgmanager[25959]: Stopping service vm:VMG
Oct 22 23:02:49 uagl1 rgmanager[25959]: Service vm:VMG is recovering
Oct 22 23:02:49 uagl1 rgmanager[25959]: #71: Relocating failed service vm:VMG
Oct 22 23:02:50 uagl1 rgmanager[25959]: Service vm:VMG is stopped
Oct 22 23:04:29 uagl1 libvirtd: Could not find keytab file: /etc/libvirt/krb5.tab: Permission denied
Oct 22 23:05:53 uagl1 rgmanager[25959]: Starting stopped service vm:VMG
Oct 22 23:05:53 uagl1 rgmanager[25959]: start on vm "VMG" returned 1 (generic error)
Oct 22 23:05:53 uagl1 rgmanager[25959]: #68: Failed to start vm:VMG; return value: 1
So i have removed the ““VM configuration file path” and provided the “path to xmlfile used to create the VM” (/havol1) in the KVM guest resource addition.(Step:5)
There is small test to confirm the configuration.
#rg_test test /etc/cluster/cluster.conf start vm VMG
Issue:2
This is absolutely stupid from me.What i have done is that, i have created kVM guest on different path(Not on shared volume) and i thought of moving the “VMG.img”(Virtual Machine Image file) later to /havol1(shared FS). But i forget to do it and keep on trying to enable the VMG service and virtual machine is not starting .I got the below error when i did the stupid thing.
Oct 24 00:03:14 uagl1 rgmanager[20185]: Starting stopped service vm:VMG
Oct 24 00:03:14 uagl1 rgmanager[20185]: start on vm "VMG" returned 1 (generic error)
Oct 24 00:03:14 uagl1 rgmanager[20185]: #68: Failed to start vm:VMG; return value: 1
Oct 24 00:03:14 uagl1 rgmanager[20185]: Stopping service vm:VMG
Oct 24 00:03:15 uagl1 rgmanager[20185]: Service vm:VMG is recovering
Oct 24 00:03:15 uagl1 rgmanager[20185]: #71: Relocating failed service vm:VMG
Oct 24 00:03:15 uagl1 rgmanager[20185]: Service vm:VMG is stopped
Oct 24 00:03:17 uagl1 rgmanager[20185]: State change: uagl2h UP
Oct 24 00:03:17 uagl1 rgmanager[20185]: Starting stopped service vm:VMG
Oct 24 00:03:17 uagl1 rgmanager[20185]: start on vm "VMG" returned 1 (generic error)
Oct 24 00:03:17 uagl1 rgmanager[20185]: #68: Failed to start vm:VMG; return value: 1
Oct 24 00:03:17 uagl1 rgmanager[20185]: Stopping service vm:VMG
Oct 24 00:03:18 uagl1 rgmanager[20185]: Service vm:VMG is recovering
Oct 24 00:03:18 uagl1 rgmanager[20185]: #71: Relocating failed service vm:VMG
Oct 24 00:03:38 uagl1 rgmanager[20185]: #2: Service vm:VMG returned failure code. Last Owner: uagl2h
Oct 24 00:03:38 uagl1 rgmanager[20185]: #4: Administrator intervention required.
Oct 24 00:03:38 uagl1 rgmanager[20185]: #2: Service vm:VMG returned failure code. Last Owner: uagl2h
Oct 24 00:03:38 uagl1 rgmanager[20185]: #4: Administrator intervention required.
I have moved the virtual machine image file to shared filesystem(/havol1) and enable the service.Worked like charm. 🙂
I know that you are just wondering how to migrate the KVM guest (virtual machine) from one node to another node and also migrating the Live machine from one to another node without downtime. Check out here.
Thank you for visiting UnixArena.