Recently i have faced strange issue in HP-UX v11.31 with VXVM 5.0 . When we tried to extend one of the file system,vxresize command returns the following error messages.
UX:vxfs fsadm: ERROR: V-3-20295: device record checksum is invalid
UX:vxfs fsadm: ERROR: V-3-20291: could not initialize devices
VxVM vxresize ERROR V-5-1-7514 Problem running fsadm command for volume test5, in diskgroup testdg
UX:vxfs fsadm: ERROR: V-3-20291: could not initialize devices
VxVM vxresize ERROR V-5-1-7514 Problem running fsadm command for volume test5, in diskgroup testdg
Many of the command returns with the same errors on that specific filesystems.
The following commands are to check for the need of defrag, does not actually do the defrag.
The following commands are to check for the need of defrag, does not actually do the defrag.
# fsadm -D /test5
fsadm: /etc/default/fs is used for determining the file system type
UX:vxfs fsadm: ERROR: V-3-20295: device record checksum is invalid.
UX:vxfs fsadm: ERROR: V-3-20291: could not initialize devices.
root @HP-UX/11.31[/root]
# fsadm -E /test5
fsadm: /etc/default/fs is used for determining the file system type
UX:vxfs fsadm: ERROR: V-3-20295: device record checksum is invalid.
UX:vxfs fsadm: ERROR: V-3-20291: could not initialize devices.
To fix the above issue , vendor recommended to upgrade the VXVM from 5.0 to 5.1 .But we don’t to go for that major upgrade.
Actually we are getting above error because of filesystem corruption.Symantec provided two workaround to fix the above issue.
1.Copy the new fsck binary from HP-UX – VXVM 5.1 and ran fsck on the corrupted fielsystem.
2.Run dd command to fix the corrupted region .
Symantec recommended to go with second option because our FS corruption was minor.
Before going to either of method we should verify from Symantec that what data we will loose if run fsck or dd .To confirm that ,we ran metasave command and provided the file to vendor to verify .
# cd /opt/VRTSspt/FS/MetaSave
# ./metasave_11.23_ia64 -f /var/tmp/fsprddata02-meta.out /dev/vx/rdsk/testdg/test5
# tar -cvf meta.tar fsprddata02-meta.out fsprddata01-meta.out
a fsprddata02-meta.out 30084 blocks
# ftp ftp.veritas.com
Name (ftp.veritas.com:): symsupport
331 User symsupport okay, need password.
Password:
230 Restricted user logged in.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> cd /incoming
250 "/incoming" is new cwd.
ftp> put metasave-out.tar
local: meta.tar remote: metasaveout.tar
227 Entering Passive Mode (216,10,194,202,167,144)
150 Data connection accepted from 64.86.140.34:28883; transfer starting.
226 Transfer completed.
386048000 bytes sent in 59 seconds (6.4e+03 Kbytes/s)
ftp> bye
221 Goodbye.
Using the metasave output, Symantec filesystem expert will test in the below manner .
# fsck -F vxfs ./test5.replay
log replay in progress
log replay failed to clean file system
file system is not clean, full fsck required
full file system check required, exiting ...
# fsck -F vxfs -o full ./test5.replay
log replay in progress
pass0 - checking structural files
fileset 1 primary-ilist inode 42 (DEV 2)
failed validation clear? (ynq)y
pass1 - checking inode sanity and blocks
rebuild structural files? (ynq)y
pass0 - checking structural files
pass1 - checking inode sanity and blocks
pass2 - checking directory linkage
pass3 - checking reference counts
pass4 - checking resource maps
au 6 emap incorrect - fix? (ynq)y
au 6 summary incorrect - fix? (ynq)y
free extent vector incorrect fix? (ynq)y
OK to clear log? (ynq)y
flush fileset headers? (ynq)y
set state to CLEAN? (ynq)y
Symantec don’t know when or why the IFDEV inodes were corrupted and they said that could have happened at any time in the past. Normal VxFS operations will not be affected by this corruption. Only at resize or reorg time do we actually go out and check the IFDEV unused extent.
Finally We have fixed our FS corruption using the below method.
To fix the IFDEV inode 42 corruption, they requested to run the below commands and its fixed our FS issue.Don’t use the below commands blindly. You need to provide the matasave to vendor and they will let you know in which block you need to run it.
#dd if=/dev/zero of=/dev/vx/rdsk/testdg/test5 bs=1024 count=256 oseek=256 conv=notrunc
#dd if=/dev/zero of=/dev/vx/rdsk/testdg/test5 bs=1024 count=256 oseek=1527 conv=notrunc
Once you run the above command ,you have to run fsck to create a that structural inodes which we nullify using dd on that FS.
#/opt/VRTS/bin/fsck -o full -y /dev/vx/rdsk/testdg/test5
log replay in progress
pass0 - checking structural files
pass1 - checking inode sanity and blocks
pass2 - checking directory linkage
pass3 - checking reference counts
pass4 - checking resource maps
OK to clear log? (ynq)y
flush fileset headers? (ynq)y
set state to CLEAN? (ynq)y
#
Thank you for reading this article.Please leave a comment if you have any doubt ,i will get back to you as soon as possible.