{"id":3136,"date":"2014-06-13T19:09:35","date_gmt":"2014-06-13T11:09:35","guid":{"rendered":"http:\/\/rmohan.com\/?p=3136"},"modified":"2014-06-13T19:09:35","modified_gmt":"2014-06-13T11:09:35","slug":"detect-failing-storage-lun-on-linux-multipath","status":"publish","type":"post","link":"https:\/\/mohan.sg\/?p=3136","title":{"rendered":"Detect failing storage LUN on Linux \u2013 multipath"},"content":{"rendered":"<p style=\"font: 13px\/18.81px 'Lucida Grande', Verdana, Arial, sans-serif; text-align: justify; color: #333333; text-transform: none; text-indent: 0px; letter-spacing: normal; word-spacing: 0px; white-space: normal; background-color: #d5d6d7; -webkit-text-stroke-width: 0px;\">f you login to server and after running<span class=\"Apple-converted-space\">\u00a0<\/span><strong>dmesg<\/strong><span class=\"Apple-converted-space\">\u00a0<\/span>\u2013 show kernel log command you get tons of messages like:<\/p>\n<p style=\"font: 13px\/18.81px 'Lucida Grande', Verdana, Arial, sans-serif; text-align: justify; color: #333333; text-transform: none; text-indent: 0px; letter-spacing: normal; word-spacing: 0px; white-space: normal; background-color: #d5d6d7; -webkit-text-stroke-width: 0px;\"><code style=\"background: #fafafa; font: 1.1em\/normal 'Courier New', Courier, monospace; padding: 5px 10px 10px; border: 1px dotted #cccccc; color: #000000; display: block;\">#<span class=\"Apple-converted-space\">\u00a0<\/span><strong>dmesg<\/strong><\/code><\/p>\n<p style=\"font: 13px\/18.81px 'Lucida Grande', Verdana, Arial, sans-serif; text-align: justify; color: #333333; text-transform: none; text-indent: 0px; letter-spacing: normal; word-spacing: 0px; white-space: normal; background-color: #d5d6d7; -webkit-text-stroke-width: 0px;\">\n<blockquote style=\"font: 13px\/18.81px 'Lucida Grande', Verdana, Arial, sans-serif; margin: 15px 30px 0px 10px; text-align: justify; color: #777777; text-transform: none; text-indent: 0px; letter-spacing: normal; padding-left: 20px; word-spacing: 0px; border-left-color: #dddddd; border-left-width: 5px; border-left-style: solid; white-space: normal; background-color: #d5d6d7; -webkit-text-stroke-width: 0px;\"><p>end_request: I\/O error, dev sdc, sector 0<br \/>\nsd 0:0:0:1: SCSI error: return code = 0\u00d700010000<br \/>\nend_request: I\/O error, dev sdb, sector 0<br \/>\nsd 0:0:0:0: SCSI error: return code = 0\u00d700010000<br \/>\nend_request: I\/O error, dev sda, sector 0<br \/>\nsd 0:0:0:2: SCSI error: return code = 0\u00d700010000<br \/>\nend_request: I\/O error, dev sdc, sector 0<br \/>\nsd 0:0:0:1: SCSI error: return code = 0\u00d700010000<br \/>\nend_request: I\/O error, dev sdb, sector 0<br \/>\nsd 0:0:0:0: SCSI error: return code = 0\u00d700010000<br \/>\nend_request: I\/O error, dev sda, sector 0<br \/>\nsd 0:0:0:2: SCSI error: return code = 0\u00d700010000<br \/>\nend_request: I\/O error, dev sdc, sector 0<br \/>\nsd 0:0:0:1: SCSI error: return code = 0\u00d700010000<br \/>\nend_request: I\/O error, dev sdb, sector 0<br \/>\nsd 0:0:0:0: SCSI error: return code = 0\u00d700010000<br \/>\nend_request: I\/O error, dev sda, sector 0<\/p><\/blockquote>\n<p style=\"font: 13px\/18.81px 'Lucida Grande', Verdana, Arial, sans-serif; text-align: justify; color: #333333; text-transform: none; text-indent: 0px; letter-spacing: normal; word-spacing: 0px; white-space: normal; background-color: #d5d6d7; -webkit-text-stroke-width: 0px;\">\n<p style=\"font: 13px\/18.81px 'Lucida Grande', Verdana, Arial, sans-serif; text-align: justify; color: #333333; text-transform: none; text-indent: 0px; letter-spacing: normal; word-spacing: 0px; white-space: normal; background-color: #d5d6d7; -webkit-text-stroke-width: 0px;\">In<span class=\"Apple-converted-space\">\u00a0<\/span><strong>\/var\/log\/messages<\/strong><span class=\"Apple-converted-space\">\u00a0<\/span>there are also log messages present like:<\/p>\n<p style=\"font: 13px\/18.81px 'Lucida Grande', Verdana, Arial, sans-serif; text-align: justify; color: #333333; text-transform: none; text-indent: 0px; letter-spacing: normal; word-spacing: 0px; white-space: normal; background-color: #d5d6d7; -webkit-text-stroke-width: 0px;\"><code style=\"background: #fafafa; font: 1.1em\/normal 'Courier New', Courier, monospace; padding: 5px 10px 10px; border: 1px dotted #cccccc; color: #000000; display: block;\"><strong># cat \/var\/log\/messages<\/strong><br \/>\n...<\/code><\/p>\n<p style=\"font: 13px\/18.81px 'Lucida Grande', Verdana, Arial, sans-serif; text-align: justify; color: #333333; text-transform: none; text-indent: 0px; letter-spacing: normal; word-spacing: 0px; white-space: normal; background-color: #d5d6d7; -webkit-text-stroke-width: 0px;\">\n<blockquote style=\"font: 13px\/18.81px 'Lucida Grande', Verdana, Arial, sans-serif; margin: 15px 30px 0px 10px; text-align: justify; color: #777777; text-transform: none; text-indent: 0px; letter-spacing: normal; padding-left: 20px; word-spacing: 0px; border-left-color: #dddddd; border-left-width: 5px; border-left-style: solid; white-space: normal; background-color: #d5d6d7; -webkit-text-stroke-width: 0px;\"><p>Apr 13 09:45:49 sl02785 kernel: end_request: I\/O error, dev sda, sector 0<br \/>\nApr 13 09:45:49 sl02785 kernel: klogd 1.4.1, \u2014\u2014\u2014- state change \u2014\u2014\u2014-<br \/>\nApr 13 09:45:50 sl02785 kernel: sd 0:0:0:1: SCSI error: return code = 0\u00d700010000<br \/>\nApr 13 09:45:50 sl02785 kernel: end_request: I\/O error, dev sdb, sector 0<br \/>\nApr 13 09:45:55 sl02785 kernel: sd 0:0:0:2: SCSI error: return code = 0\u00d700010000<br \/>\nApr 13 09:45:55 sl02785 kernel: end_request: I\/O error, dev sdc, sector 0<\/p><\/blockquote>\n<p style=\"font: 13px\/18.81px 'Lucida Grande', Verdana, Arial, sans-serif; text-align: justify; color: #333333; text-transform: none; text-indent: 0px; letter-spacing: normal; word-spacing: 0px; white-space: normal; background-color: #d5d6d7; -webkit-text-stroke-width: 0px;\">\n<p style=\"font: 13px\/18.81px 'Lucida Grande', Verdana, Arial, sans-serif; text-align: justify; color: #333333; text-transform: none; text-indent: 0px; letter-spacing: normal; word-spacing: 0px; white-space: normal; background-color: #d5d6d7; -webkit-text-stroke-width: 0px;\">\n<p style=\"font: 13px\/18.81px 'Lucida Grande', Verdana, Arial, sans-serif; text-align: justify; color: #333333; text-transform: none; text-indent: 0px; letter-spacing: normal; word-spacing: 0px; white-space: normal; background-color: #d5d6d7; -webkit-text-stroke-width: 0px;\">This is a sure sign something is wrong with<span class=\"Apple-converted-space\">\u00a0<\/span><b>SCSI hard<\/b><span class=\"Apple-converted-space\">\u00a0<\/span>drives or<b>SCSI controller<\/b>, to further debug the situation you can use the<b>multipath<\/b><span class=\"Apple-converted-space\">\u00a0<\/span>command, i.e.:<\/p>\n<p style=\"font: 13px\/18.81px 'Lucida Grande', Verdana, Arial, sans-serif; text-align: justify; color: #333333; text-transform: none; text-indent: 0px; letter-spacing: normal; word-spacing: 0px; white-space: normal; background-color: #d5d6d7; -webkit-text-stroke-width: 0px;\">\n<code style=\"background: #fafafa; font: 1.1em\/normal 'Courier New', Courier, monospace; padding: 5px 10px 10px; border: 1px dotted #cccccc; color: #000000; display: block;\"><strong>multipath -ll | grep -i failed<br \/>\n_ 0:0:0:2 sdc 8:32 [failed][faulty]<br \/>\n_ 0:0:0:1 sdb 8:16 [failed][faulty]<br \/>\n_ 0:0:0:0 sda 8:0 [failed][faulty]<\/strong><\/code>\n<\/p>\n<p style=\"font: 13px\/18.81px 'Lucida Grande', Verdana, Arial, sans-serif; text-align: justify; color: #333333; text-transform: none; text-indent: 0px; letter-spacing: normal; word-spacing: 0px; white-space: normal; background-color: #d5d6d7; -webkit-text-stroke-width: 0px;\">As you can see all<span class=\"Apple-converted-space\">\u00a0<\/span><i>3 drives merged (<strong>sdc<\/strong>,<span class=\"Apple-converted-space\">\u00a0<\/span><strong>sdb<\/strong><span class=\"Apple-converted-space\">\u00a0<\/span>and<span class=\"Apple-converted-space\">\u00a0<\/span><strong>sda<\/strong>) to show up on 1 physical drive via the remote network connected<\/i><span class=\"Apple-converted-space\">\u00a0<\/span>\u2013<span class=\"Apple-converted-space\">\u00a0<\/span><b>LUN<\/b><span class=\"Apple-converted-space\">\u00a0<\/span>to the server is showing as faulty. This is a clear sign something is either wrong with 3 hard drive members of<span class=\"Apple-converted-space\">\u00a0<\/span><strong>LUN<\/strong><span class=\"Apple-converted-space\">\u00a0<\/span>\u2013 (<em>Logical Unit Number<\/em>) (which is less probable)\u00a0 or most likely there is problem with the LUN network\u00a0 SCSI controller. It is common error that<span class=\"Apple-converted-space\">\u00a0<\/span><em>LUN SCSI controller optics cable gets dirty and needs a physical clean up to solve it<\/em>.<\/p>\n<p style=\"font: 13px\/18.81px 'Lucida Grande', Verdana, Arial, sans-serif; text-align: justify; color: #333333; text-transform: none; text-indent: 0px; letter-spacing: normal; word-spacing: 0px; white-space: normal; background-color: #d5d6d7; -webkit-text-stroke-width: 0px;\">In case you don&#8217;t know<span class=\"Apple-converted-space\">\u00a0<\/span><em><strong>what is storage LUN?<\/strong><\/em><span class=\"Apple-converted-space\">\u00a0<\/span>\u2013 In computer storage, a logical unit number, or<span class=\"Apple-converted-space\">\u00a0<\/span><em><strong>LUN<\/strong><\/em>, is a number used to identify a logical unit, which is a device addressed by the<span class=\"Apple-converted-space\">\u00a0<\/span><em>SCSI protocol<\/em><span class=\"Apple-converted-space\">\u00a0<\/span>or protocols which encapsulate SCSI, such as<em><span class=\"Apple-converted-space\">\u00a0<\/span>Fibre Channel<\/em><span class=\"Apple-converted-space\">\u00a0<\/span>or<span class=\"Apple-converted-space\">\u00a0<\/span><em>iSCSI<\/em>. A<span class=\"Apple-converted-space\">\u00a0<\/span><em>LUN<\/em><span class=\"Apple-converted-space\">\u00a0<\/span>may be used with any device which supports read\/write operations, such as a tape drive, but is most often used to refer to a logical disk as created on a SAN. Though not technically correct, the term &#8220;<em>LUN<\/em>&#8221; is often also used to refer to the logical disk itself.<\/p>\n<p style=\"font: 13px\/18.81px 'Lucida Grande', Verdana, Arial, sans-serif; text-align: justify; color: #333333; text-transform: none; text-indent: 0px; letter-spacing: normal; word-spacing: 0px; white-space: normal; background-color: #d5d6d7; -webkit-text-stroke-width: 0px;\">What LUN&#8217;s does is very similar to Linux&#8217;s Software<span class=\"Apple-converted-space\">\u00a0<\/span><i>LVM (Logical Volume Manager)<\/i>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p style=\"font: 13px\/18.81px 'Lucida Grande', Verdana, Arial, sans-serif; text-align: justify; color: #333333; text-transform: none; text-indent: 0px; letter-spacing: normal; word-spacing: 0px; white-space: normal; background-color: #d5d6d7; -webkit-text-stroke-width: 0px;\">f you login to server and after running dmesg \u2013 show kernel log command you get tons of messages like:<\/p>\n<p style=\"font: 13px\/18.81px 'Lucida Grande', Verdana, Arial, sans-serif; text-align: justify; [...] \n\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[],"_links":{"self":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/3136"}],"collection":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3136"}],"version-history":[{"count":2,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/3136\/revisions"}],"predecessor-version":[{"id":3138,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/3136\/revisions\/3138"}],"wp:attachment":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3136"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3136"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3136"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}