{"id":1308,"date":"2012-09-05T14:44:53","date_gmt":"2012-09-05T06:44:53","guid":{"rendered":"http:\/\/rmohan.com\/?p=1308"},"modified":"2012-09-05T15:18:40","modified_gmt":"2012-09-05T07:18:40","slug":"adding-and-removing-san-disks-from-linux-device-manager-multipath-systems","status":"publish","type":"post","link":"https:\/\/mohan.sg\/?p=1308","title":{"rendered":"Adding and Removing SAN Disks from Linux Device Manager Multipath Systems"},"content":{"rendered":"<h3><strong>Add a new disk<\/strong><\/h3>\n<p>This assumes you have created an array on the <a href=\"http:\/\/www.novell.com\/communities\/glossary\/term\/3059\"><acronym title=\"Storage area networkIn ZENworks Server Management, clustered nodes together with their shared disk system and shared volumes.\">SAN<\/acronym><\/a> and allocated space to a <a href=\"http:\/\/www.novell.com\/communities\/glossary\/term\/1670\"><acronym title=\"A volume that is set to a specific size or that can grow dynamically according to the amount of physical space assigned to the storage pool that contains the volume.\">logical volume<\/acronym><\/a> on it; you have mapped a <a href=\"http:\/\/www.novell.com\/communities\/glossary\/term\/3339\"><acronym title=\"Logical Unit Number In computer storage, a logical unit number or LUN is an address for an individual disk drive and by extension, the disk device itself. The term originated in the SCSI protocol as a way of differentiating individual disk drives within a common SCSI target device like a disk array.\">LUN<\/acronym><\/a> pointing that logical volume to that host, and that the host is correctly zoned to see the SAN in the fibre channel fabric.<\/p>\n<ul>\n<li>Before anything, run <code>multipath -ll<\/code> to see what is there currently.<\/li>\n<li>See how many HBAs are connected (and zoned) to the SAN &#8211; you need to repeat the commands for each one. For example:\n<pre>          echo 1 &gt; \/sys\/class\/fc_host\/host0\/issue_lip\r\n          echo 1 &gt; \/sys\/class\/fc_host\/host1\/issue_lip\r\n          echo \"- - -\" &gt; \/sys\/class\/scsi_host\/host0\/scan\r\n          echo \"- - -\" &gt; \/sys\/class\/scsi_host\/host0\/scan\r\n\t\t  \r\n<\/pre>\n<\/li>\n<li>After running those commands, check that something happened by using <code>dmesg<\/code> and <code>\/var\/log\/messages<\/code>.<\/li>\n<li>Run <code>multipath -v2<\/code>to get multipath to pick it up &#8211; you can then compare the listing to the previously run command.\n<div>Note the scsi devices for the new disk, it will be sdX and sdY or whatever.<\/div>\n<\/li>\n<li>Edit <code>\/etc\/lvm\/lvm.conf<\/code> and make sure these are being filtered to remove duplicates &#8211; use <code>vgdisplay -vv<\/code>to show what LVM considers duplicate.\n<div>FYI: device mapper \/ multipath create multiple device handles for the same device, this can cause delays with LVM2 and severely impact <a href=\"http:\/\/www.novell.com\/communities\/glossary\/term\/2470\"><acronym title=\"The total amount of useful data that is processed or communicated during a specific time period between a source and a destination.\">throughput<\/acronym><\/a>.<\/div>\n<\/li>\n<li>Now you can <code>pvcreate \/dev\/dm-XX<\/code>, <code>vgextend VolGroup dev\/dm-XX<\/code>, etc.<\/li>\n<\/ul>\n<h3><strong>Remove a disk<\/strong><\/h3>\n<ul>\n<li>Run the <code>multipath -ll<\/code> command, note the UUID (the big hex number), LUN and sdX device of the disk, eg in the example below it is LUN 2:, and they are <code>\/dev\/sdf<\/code> and <code>\/dev\/sde<\/code>&#8211; you will need this info for the procedure. Just confirm this is in fact the one you want to remove &#8211; cross-check the LUN and size of the volume on the SAN before proceeding&#8230;\n<pre>      3600a0b80000fb6e50000000e487b02f5 dm-10 IBM,1742\r\n      [size=1.6T][features=1 queue_if_no_path][hwhandler=1 rdac]\r\n      \\_ round-robin 0 [prio=6][active]\r\n      \\_ 1:0:0:2 sdf 8:80 [active][ready]\r\n      \\_ round-robin 0 [prio=1][enabled]\r\n      \\_ 0:0:0:2 sde 8:64 [active][ghost]\r\n<\/pre>\n<div>Note &#8211; the <code>dm-XX<\/code> is not permanent and may change when you&#8217;ve added or removed disks, so don&#8217;t rely on old info &#8211; check each time.<\/div>\n<\/li>\n<li>Also <code>cat \/proc\/scsi\/scsi<\/code> to see match up the kernel <a href=\"http:\/\/www.novell.com\/communities\/glossary\/term\/2287\"><acronym title=\"Small Computer Systems InterfaceAn industry standard that provides high-speed access to peripheral devices such as hard disk drives, tape drives, and other hardware components. Computers and peripheral devices designed to meet SCSI specifications have a large degree of compatibility. SCSI is often implemented in servers and other high-level machines because of its high-speed performance.\">SCSI<\/acronym><\/a> devices with the SCSI IDs and LUNs of the SAN disks.<\/li>\n<\/ul>\n<ul>\n<li>First you need to remove the disk from the volume group.\n<ul>\n<li>If the disk is in use, either delete what is on it (if there is a logical volume limited to that disk), or use <code>pvmove<\/code>. (this of course assumes you have sufficient space to move everything off the original disk)\n<div>NB &#8211; with <code>pvmove<\/code> on SUSE10 sp2, there is a bug where if you are moving from a bigger disk to smaller disk(s), it may complain there isn&#8217;t enough space. Just move as many extents as are on the first smaller disk, the you can move the rest on to the second, eg: <code>pvmove \/dev\/dm-1:0-20000 \/dev\/dm-2<\/code>.<\/div>\n<\/li>\n<li>Once stuff is deleted\/removed: <code>vgreduce VolGroup dev\/dm-XX<\/code> and <code>pvremove \/dev\/dm-XX<\/code>.<\/li>\n<\/ul>\n<\/li>\n<li>Using the disk ID for the next command (the dm-xx isn&#8217;t recognised), from multipath -ll:<br \/> <code>dmsetup remove 3600a0b80000f7b270000000b47b15c26<\/code>. (of course you need to use your own disk ID)<\/li>\n<li>Now, finally, you remove the SCSI devices from the kernel:<\/li>\n<\/ul>\n<pre>      echo 1 &gt; \/sys\/block\/sdX\/device\/delete\r\n      echo 1 &gt; \/sys\/block\/sdY\/device\/delete\r\n\t  \r\n<\/pre>\n<p>You should now have all traces removed, you can run <code>multipath -ll<\/code> and <code>cat \/proc\/scsi\/scsi<\/code> to cross check. You can now remove the mapping from the SAN and delete the logical volume if required.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Add a new disk <\/p>\n<p>This assumes you have created an array on the SAN and allocated space to a logical volume on it; you have mapped a LUN pointing that logical volume to that host, and that the host is correctly zoned to see the SAN in the fibre channel fabric.<\/p>\n<p> Before anything, run multipath [&#8230;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[23],"tags":[],"_links":{"self":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/1308"}],"collection":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1308"}],"version-history":[{"count":4,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/1308\/revisions"}],"predecessor-version":[{"id":1310,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/1308\/revisions\/1310"}],"wp:attachment":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1308"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1308"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1308"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}