Overview<\/p>\n
Logical Volume Management (LVM) provides a high level, flexible view of a server’s disk storage. Though robust, problems can occur. The purpose of this document is to review the recovery process when a disk is missing or damaged, and then apply that process to plausible examples. When a disk is accidentally removed or damaged in some way that adversely affects the logical volume, the general recovery process is:<\/p>\n
Replace the failed or missing disk
\nRestore the missing disk’s UUID
\nRestore the LVM meta data
\nRepair the file system on the LVM device
\nThe recovery process will be demonstrated in three specific cases:<\/p>\n
A disk belonging to a logical volume group is removed from the server
\nThe LVM meta data is damaged or corrupted
\nOne disk in a multi-disk volume group has been permanently removed
\nThis article discusses how to restore the LVM meta data. This is a risky proposition. If you restore invalid information, you can loose all the data on the LVM device. An important part of LVM recovery is having backups of the meta data to begin with, and knowing how it’s supposed to look when everything is running smoothly. LVM keeps backup and archive copies of it’s meta data in \/etc\/lvm\/backup and \/etc\/lvm\/archive. Backup these directories regularly, and be familiar with their contents. You should also manually backup the LVM meta data with vgcfgbackup before starting any maintenance projects on your LVM volumes.<\/p>\n
If you are planning on removing a disk from the server that belongs to a volume group, you should refer to the LVM HOWTO before doing so.<\/p>\n
Server Configuration<\/p>\n
In all three examples, a server with SUSE Linux Enterprise Server 10 with Service Pack 1 (SLES10 SP1) will be used with LVM version 2. The examples will use a volume group called “sales” with a linear logical volume called “reports”. The logical volume and it’s mount point are shown below. You will need to substitute your mount points and volume names as needed to match your specific environment.<\/p>\n
ls-lvm:~ # cat \/proc\/partitions
\nmajor minor #blocks name<\/p>\n
8 0 4194304 sda
\n 8 1 514048 sda1
\n 8 2 1052257 sda2
\n 8 3 1 sda3
\n 8 5 248976 sda5
\n 8 16 524288 sdb
\n 8 32 524288 sdc
\n 8 48 524288 sdd<\/p>\n
ls-lvm:~ # pvcreate \/dev\/sda5 \/dev\/sd[b-d]
\n Physical volume “\/dev\/sda5” successfully created
\n Physical volume “\/dev\/sdb” successfully created
\n Physical volume “\/dev\/sdc” successfully created
\n Physical volume “\/dev\/sdd” successfully created<\/p>\n
ls-lvm:~ # vgcreate sales \/dev\/sda5 \/dev\/sd[b-d]
\n Volume group “sales” successfully created<\/p>\n
ls-lvm:~ # lvcreate -n reports -L +1G sales
\n Logical volume “reports” created<\/p>\n
ls-lvm:~ # pvscan
\n PV \/dev\/sda5 VG sales lvm2 [240.00 MB \/ 240.00 MB free]
\n PV \/dev\/sdb VG sales lvm2 [508.00 MB \/ 0 free]
\n PV \/dev\/sdc VG sales lvm2 [508.00 MB \/ 0 free]
\n PV \/dev\/sdd VG sales lvm2 [508.00 MB \/ 500.00 MB free]
\n Total: 4 [1.72 GB] \/ in use: 4 [1.72 GB] \/ in no VG: 0 [0 ]<\/p>\n
ls-lvm:~ # vgs
\n VG #PV #LV #SN Attr VSize VFree
\n sales 4 1 0 wz–n- 1.72G 740.00M<\/p>\n
ls-lvm:~ # lvs
\n LV VG Attr LSize Origin Snap% Move Log Copy%
\n reports sales -wi-ao 1.00G<\/p>\n
ls-lvm:~ # mount | grep sales
\n\/dev\/mapper\/sales-reports on \/sales\/reports type ext3 (rw)<\/p>\n
ls-lvm:~ # df -h \/sales\/reports
\nFilesystem Size Used Avail Use% Mounted on
\n\/dev\/mapper\/sales-reports
\n 1008M 33M 925M 4% \/sales\/reports
\nDisk Belonging to a Volume Group Removed<\/p>\n
Removing a disk, belonging to a logical volume group, from the server may sound a bit strange, but with Storage Area Networks (SAN) or fast paced schedules, it happens.<\/p>\n
Symptom:<\/p>\n
The first thing you may notice when the server boots are messages like:<\/p>\n
“Couldn’t find all physical volumes for volume group sales.”
\n“Couldn’t find device with uuid ’56pgEk-0zLS-cKBc-z9vJ-kP65-DUBI-hwZPSu’.”
\n‘Volume group “sales” not found’<\/p>\n
Type root’s password.
\nEdit the \/etc\/fstab file.
\nComment out the line with \/dev\/sales\/report
\nReboot
\nThe LVM symptom is a missing sales volume group. Typing cat \/proc\/partitions confirms the server is missing one of it’s disks.<\/p>\n
ls-lvm:~ # cat \/proc\/partitions
\nmajor minor #blocks name<\/p>\n
8 0 4194304 sda
\n 8 1 514048 sda1
\n 8 2 1052257 sda2
\n 8 3 1 sda3
\n 8 5 248976 sda5
\n 8 16 524288 sdb
\n 8 32 524288 sdc<\/p>\n
ls-lvm:~ # pvscan
\n Couldn’t find device with uuid ’56ogEk-OzLS-cKBc-z9vJ-kP65-DUBI-hwZPSu’.
\n Couldn’t find device with uuid ’56ogEk-OzLS-cKBc-z9vJ-kP65-DUBI-hwZPSu’.
\n Couldn’t find device with uuid ’56ogEk-OzLS-cKBc-z9vJ-kP65-DUBI-hwZPSu’.
\n Couldn’t find device with uuid ’56ogEk-OzLS-cKBc-z9vJ-kP65-DUBI-hwZPSu’.
\n Couldn’t find device with uuid ’56ogEk-OzLS-cKBc-z9vJ-kP65-DUBI-hwZPSu’.
\n Couldn’t find device with uuid ’56ogEk-OzLS-cKBc-z9vJ-kP65-DUBI-hwZPSu’.
\n PV \/dev\/sda5 VG sales lvm2 [240.00 MB \/ 240.00 MB free]
\n PV \/dev\/sdb VG sales lvm2 [508.00 MB \/ 0 free]
\n PV unknown device VG sales lvm2 [508.00 MB \/ 0 free]
\n PV \/dev\/sdc VG sales lvm2 [508.00 MB \/ 500.00 MB free]
\n Total: 4 [1.72 GB] \/ in use: 4 [1.72 GB] \/ in no VG: 0 [0 ]
\nSolution:<\/p>\n
Fortunately, the meta data and file system on the disk that was \/dev\/sdc are intact.
\nSo the recovery is to just put the disk back.
\nReboot the server.
\nThe \/etc\/init.d\/boot.lvm start script will scan and activate the volume group at boot time.
\nDon’t forget to uncomment the \/dev\/sales\/reports device in the \/etc\/fstab file.
\nIf this procedure does not work, then you may have corrupt LVM meta data.<\/p>\n
Corrupted LVM Meta Data<\/p>\n
The LVM meta data does not get corrupted very often; but when it does, the file system on the LVM logical volume should also be considered unstable. The goal is to recover the LVM volume, and then check file system integrity.<\/p>\n
Symptom 1:<\/p>\n
Attempting to activate the volume group gives the following:<\/p>\n
ls-lvm:~ # vgchange -ay sales
\n \/dev\/sdc: Checksum error
\n \/dev\/sdc: Checksum error
\n \/dev\/sdc: Checksum error
\n \/dev\/sdc: Checksum error
\n \/dev\/sdc: Checksum error
\n \/dev\/sdc: Checksum error
\n \/dev\/sdc: Checksum error
\n \/dev\/sdc: Checksum error
\n \/dev\/sdc: Checksum error
\n \/dev\/sdc: Checksum error
\n \/dev\/sdc: Checksum error
\n \/dev\/sdc: Checksum error
\n \/dev\/sdc: Checksum error
\n Couldn’t read volume group metadata.
\n Volume group sales metadata is inconsistent
\n Volume group for uuid not found: m4Cg2vkBVSGe1qSMNDf63v3fDHqN4uEkmWoTq5TpHpRQwmnAGD18r44OshLdHj05
\n 0 logical volume(s) in volume group “sales” now active
\nThis symptom is the result of a minor change in the meta data. In fact, only three bytes were overwritten. Since only a portion of the meta data was damaged, LVM can compare it’s internal check sum against the meta data on the device and know it’s wrong. There is enough meta data for LVM to know that the “sales” volume group and devices exit, but are unreadable.<\/p>\n
ls-lvm:~ # pvscan
\n \/dev\/sdc: Checksum error
\n \/dev\/sdc: Checksum error
\n \/dev\/sdc: Checksum error
\n \/dev\/sdc: Checksum error
\n \/dev\/sdc: Checksum error
\n \/dev\/sdc: Checksum error
\n \/dev\/sdc: Checksum error
\n \/dev\/sdc: Checksum error
\n PV \/dev\/sda5 VG sales lvm2 [240.00 MB \/ 240.00 MB free]
\n PV \/dev\/sdb VG sales lvm2 [508.00 MB \/ 0 free]
\n PV \/dev\/sdc VG sales lvm2 [508.00 MB \/ 0 free]
\n PV \/dev\/sdd VG sales lvm2 [508.00 MB \/ 500.00 MB free]
\n Total: 4 [1.72 GB] \/ in use: 4 [1.72 GB] \/ in no VG: 0 [0 ]
\nNotice pvscan shows all devices present and associated with the sales volume group. It’s not the device UUID that is not found, but the volume group UUID.<\/p>\n
Solution 1:<\/p>\n
Since the disk was never removed, leave it as is.
\nThere were no device UUID errors, so don’t attempt to restore the UUIDs.
\nThis is a good candidate to just try restoring the LVM meta data.
\nls-lvm:~ # vgcfgrestore sales
\n \/dev\/sdc: Checksum error
\n \/dev\/sdc: Checksum error
\n Restored volume group sales<\/p>\n
ls-lvm:~ # vgchange -ay sales
\n 1 logical volume(s) in volume group “sales” now active<\/p>\n
ls-lvm:~ # pvscan
\n PV \/dev\/sda5 VG sales lvm2 [240.00 MB \/ 240.00 MB free]
\n PV \/dev\/sdb VG sales lvm2 [508.00 MB \/ 0 free]
\n PV \/dev\/sdc VG sales lvm2 [508.00 MB \/ 0 free]
\n PV \/dev\/sdd VG sales lvm2 [508.00 MB \/ 500.00 MB free]
\n Total: 4 [1.72 GB] \/ in use: 4 [1.72 GB] \/ in no VG: 0 [0 ]
\nRun a file system check on \/dev\/sales\/reports.
\nls-lvm:~ # e2fsck \/dev\/sales\/reports
\ne2fsck 1.38 (30-Jun-2005)
\n\/dev\/sales\/reports: clean, 961\/131072 files, 257431\/262144 blocks<\/p>\n
ls-lvm:~ # mount \/dev\/sales\/reports \/sales\/reports\/<\/p>\n
ls-lvm:~ # df -h \/sales\/reports\/
\nFilesystem Size Used Avail Use% Mounted on
\n\/dev\/mapper\/sales-reports
\n 1008M 990M 0 100% \/sales\/reports
\nSymptom 2:<\/p>\n
Minor damage to the LVM meta data is easily fixed with vgcfgrestore. If the meta data is gone, or severely damaged, then LVM will consider that disk as an “unknown device.” If the volume group contains only one disk, then the volume group and it’s logical volumes will simply be gone. In this case the symptom is the same as if the disk was accidentally removed, with the exception of the device name. Since \/dev\/sdc was not actually removed from the server, the devices are still labeled a through d.<\/p>\n
ls-lvm:~ # pvscan
\n Couldn’t find device with uuid ’56ogEk-OzLS-cKBc-z9vJ-kP65-DUBI-hwZPSu’.
\n Couldn’t find device with uuid ’56ogEk-OzLS-cKBc-z9vJ-kP65-DUBI-hwZPSu’.
\n Couldn’t find device with uuid ’56ogEk-OzLS-cKBc-z9vJ-kP65-DUBI-hwZPSu’.
\n Couldn’t find device with uuid ’56ogEk-OzLS-cKBc-z9vJ-kP65-DUBI-hwZPSu’.
\n Couldn’t find device with uuid ’56ogEk-OzLS-cKBc-z9vJ-kP65-DUBI-hwZPSu’.
\n Couldn’t find device with uuid ’56ogEk-OzLS-cKBc-z9vJ-kP65-DUBI-hwZPSu’.
\n PV \/dev\/sda5 VG sales lvm2 [240.00 MB \/ 240.00 MB free]
\n PV \/dev\/sdb VG sales lvm2 [508.00 MB \/ 0 free]
\n PV unknown device VG sales lvm2 [508.00 MB \/ 0 free]
\n PV \/dev\/sdd VG sales lvm2 [508.00 MB \/ 500.00 MB free]
\n Total: 4 [1.72 GB] \/ in use: 4 [1.72 GB] \/ in no VG: 0 [0 ]
\nSolution 2:<\/p>\n
First, replace the disk. Most likely the disk is already there, just damaged.
\nSince the UUID on \/dev\/sdc is not there, a vgcfgrestore will not work.
\nls-lvm:~ # vgcfgrestore sales
\n Couldn’t find device with uuid ’56ogEk-OzLS-cKBc-z9vJ-kP65-DUBI-hwZPSu’.
\n Couldn’t find all physical volumes for volume group sales.
\n Restore failed.
\nComparing the output of cat \/proc\/partitions and pvscan shows the missing device is \/dev\/sdc, and pvscan shows which UUID it needs for that device. So, copy and paste the UUID that pvscan shows for \/dev\/sdc.
\nls-lvm:~ # pvcreate –uuid 56ogEk-OzLS-cKBc-z9vJ-kP65-DUBI-hwZPSu \/dev\/sdc
\n Physical volume “\/dev\/sdc” successfully created
\nRestore the LVM meta data
\nls-lvm:~ # vgcfgrestore sales
\n Restored volume group sales<\/p>\n
ls-lvm:~ # vgscan
\n Reading all physical volumes. This may take a while…
\n Found volume group “sales” using metadata type lvm2<\/p>\n
ls-lvm:~ # vgchange -ay sales
\n 1 logical volume(s) in volume group “sales” now active
\nRun a file system check on \/dev\/sales\/reports.
\nls-lvm:~ # e2fsck \/dev\/sales\/reports
\ne2fsck 1.38 (30-Jun-2005)
\n\/dev\/sales\/reports: clean, 961\/131072 files, 257431\/262144 blocks<\/p>\n
ls-lvm:~ # mount \/dev\/sales\/reports \/sales\/reports\/<\/p>\n
ls-lvm:~ # df -h \/sales\/reports
\nFilesystem Size Used Avail Use% Mounted on
\n\/dev\/mapper\/sales-reports
\n 1008M 990M 0 100% \/sales\/reports<\/p>\n
Disk Permanently Removed<\/p>\n
This is the most severe case. Obviously if the disk is gone and unrecoverable, the data on that disk is likewise unrecoverable. This is a great time to feel good knowing you have a solid backup to rely on. However, if the good feelings are gone, and there is no backup, how do you recover as much data as possible from the remaining disks in the volume group? No attempt will be made to address the data on the unrecoverable disk; this topic will be left to the data recovery experts.<\/p>\n
Symptom:<\/p>\n
The symptom will be the same as Symptom 2 in the Corrupted LVM Meta Data section above. You will see errors about an “unknown device” and missing device with UUID.<\/p>\n
Solution:<\/p>\n
Add a replacement disk to the server. Make sure the disk is empty.
\nCreate the LVM meta data on the new disk using the old disk’s UUID that pvscan displays.
\nls-lvm:~ # pvcreate –uuid 56ogEk-OzLS-cKBc-z9vJ-kP65-DUBI-hwZPSu \/dev\/sdc
\n Physical volume “\/dev\/sdc” successfully created
\nRestore the backup copy of the LVM meta data for the sales volume group.
\nls-lvm:~ # vgcfgrestore sales
\n Restored volume group sales<\/p>\n
ls-lvm:~ # vgscan
\n Reading all physical volumes. This may take a while…
\n Found volume group “sales” using metadata type lvm2<\/p>\n
ls-lvm:~ # vgchange -ay sales
\n 1 logical volume(s) in volume group “sales” now active
\nRun a file system check to rebuild the file system.
\nls-lvm:~ # e2fsck -y \/dev\/sales\/reports
\ne2fsck 1.38 (30-Jun-2005)
\n–snip–
\nFree inodes count wrong for group #5 (16258, counted=16384).
\nFix? yes<\/p>\n
Free inodes count wrong (130111, counted=130237).
\nFix? yes<\/p>\n
\/dev\/sales\/reports: ***** FILE SYSTEM WAS MODIFIED *****
\n\/dev\/sales\/reports: 835\/131072 files (5.7% non-contiguous), 137213\/262144 blocks
\nMount the file system and recover as much data as possible.
\nNOTE: If the missing disk contains the beginning of the file system, then the file system’s superblock will be missing. You will need to rebuild or use an alternate superblock. Restoring a file system superblock is outside the scope of this article, please refer to your file system’s documentation.
\nConclusion<\/p>\n
LVM by default keeps backup copies of it’s meta data for all LVM devices. These backup files are stored in \/etc\/lvm\/backup and \/etc\/lvm\/archive. If a disk is removed or the meta data gets damaged in some way, it can be easily restored, if you have backups of the meta data. This is why it is highly recommended to never turn off LVM’s auto backup feature. Even if a disk is permanently removed from the volume group, it can be reconstructed, and often times the remaining data on the file system recovered.<\/p>\n","protected":false},"excerpt":{"rendered":"
Overview<\/p>\n
Logical Volume Management (LVM) provides a high level, flexible view of a server’s disk storage. Though robust, problems can occur. The purpose of this document is to review the recovery process when a disk is missing or damaged, and then apply that process to plausible examples. When a disk is accidentally removed or damaged […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4,17],"tags":[],"_links":{"self":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/4124"}],"collection":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=4124"}],"version-history":[{"count":1,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/4124\/revisions"}],"predecessor-version":[{"id":4125,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/4124\/revisions\/4125"}],"wp:attachment":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=4124"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=4124"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=4124"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}