{"id":3132,"date":"2014-06-12T22:48:43","date_gmt":"2014-06-12T14:48:43","guid":{"rendered":"http:\/\/rmohan.com\/?p=3132"},"modified":"2014-06-12T22:48:43","modified_gmt":"2014-06-12T14:48:43","slug":"mount-point-suddenly-turned-into-read-only","status":"publish","type":"post","link":"https:\/\/mohan.sg\/?p=3132","title":{"rendered":"Mount point suddenly turned into READ ONLY"},"content":{"rendered":"

Mount point suddenly turned into READ ONLY<\/h3>\n
<\/div>\n
\n

A RAC system, the GRID mount point on eighth node suddenly turned into read-only mode:<\/span><\/p>\n

\n
[oracle@node8 ~]$ cd \/prod\/grid\r\n[oracle@node8 grid]$ touch 1\r\ntouch: cannot touch `1': Read-only file system<\/pre>\n<\/div>\n
Instance is down due to that.
\nWe can see below error in OS log:<\/span><\/p>\n
\n
Sep  5 03:26:13 node8 kernel: EXT3-fs error (device dm-226): ext3_lookup: unlinked inode 426196 in dir #2244752\r\nSep  5 03:26:13 node8 kernel: EXT3-fs error (device dm-226): ext3_lookup: unlinked inode 426199 in dir #2244752\r\nSep  5 03:26:13 node8 kernel: ext3_abort called.\r\nSep  5 03:26:13 node8 kernel: EXT3-fs error (device dm-226): ext3_journal_start_sb: Detected aborted journal\r\nSep  5 03:26:13 node8 kernel: Remounting filesystem read-only<\/pre>\n<\/div>\n
Something wrong with inode in dir 2244752.
\nTo found out what dir 2244752 is, we can use linux kernel command:
\ndebugfs<\/strong><\/span><\/span><\/p>\n
But in this case i don’t care what the dir 2244752 is, we only want the issue to be fixed.<\/span><\/p>\n
So we need to un-mount the mount point, use fsck to fix the Inode issue, and then re-mount the mount point.<\/p>\n
We use\u00a0<\/span>losf\/fuser<\/strong>\u00a0<\/span>command to find out all related processes and kill them, and through\u00a0<\/span>losetup<\/strong>\u00a0<\/span>command we ensure no resclusive device for this mount point, but still when we try to\u00a0umount the mount point through “mount -f”, it reports mount point busy:\u00a0<\/span><\/p>\n
\n
[root@node8 log]# fuser \/prod\/grid\r\n[root@node8 log]#\r\n[root@node8 log]# lsof |grep \/prod\/grid\r\n[root@node8 log]#\r\n[root@node8 log]# losetup -a\r\n[root@node8 log]#<\/pre>\n<\/div>\n

\nAbove case means the mount point is holding by kernel, not a process.\u00a0In this case,\u00a0we can\u00a0use \u201cumount -l\u201d to\u00a0trigger a lazy umount, but with lazy umount, it will damage the file system when we try to fsck.
\nAlso it may prevent us when we want to mount it back.<\/span><\/span><\/p>\n
A clean and safe way is to reboot the server. After that we succeed fixed the mount point read only issue.<\/p>\n<\/div>\n
ext3_lookup finds an “unlinked inode” and calls ext3_abort which results in a read-only filesystem<\/p>\n
Issue<\/p>\n
The ext3-filesystem went read-only. In \/var\/log\/messages the following errors are logged:
\nEXT3-fs error (device dm-4): ext3_lookup: unlinked inode 91626760 in dir #19039379
\nAborting journal on device dm-4.
\next3_abort called.
\nEXT3-fs error (device dm-4): ext3_journal_start_sb: Detected aborted journal
\nRemounting filesystem read-only
\nA couple of days later (after restoring with as written in the resolution):
\nEXT3-fs error (device dm-4): ext3_free_blocks_sb: bit already cleared for block 43951098
\nAborting journal on device dm-4.
\nEXT3-fs error (device dm-4) in ext3_free_blocks_sb: Journal has aborted
\nEXT3-fs error (device dm-4) in ext3_reserve_inode_write: Journal has aborted
\nEXT3-fs error (device dm-4) in ext3_truncate: Journal has aborted
\nEXT3-fs error (device dm-4) in ext3_reserve_inode_write: Journal has aborted
\nEXT3-fs error (device dm-4) in ext3_orphan_del: Journal has aborted
\nEXT3-fs error (device dm-4) in ext3_reserve_inode_write: Journal has aborted
\nEXT3-fs error (device dm-4) in ext3_delete_inode: Journal has aborted
\next3_abort called.
\nEXT3-fs error (device dm-4): ext3_journal_start_sb: Detected aborted journal
\nRemounting filesystem read-only
\n__journal_remove_journal_head: freeing b_committed_data
\nnfs ext3 file system going read-only on two servers which are nfs cross mounted to each other.
\nAug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339312 in dir #318520
\nAug 16 11:57:17 server1 kernel: Aborting journal on device dm-10.
\nAug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339315 in dir #318520
\nAug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339293 in dir #318520
\nAug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339321 in dir #318520
\nAug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339323 in dir #318520
\nAug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339276 in dir #318520
\nAug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339322 in dir #318520
\nAug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339291 in dir #318520
\nAug 16 11:57:17 server1 kernel: ext3_abort called.
\nAug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_journal_start_sb: Detected aborted journal
\nAug 16 11:57:17 server1 kernel: Remounting filesystem read-only
\nAug 16 12:02:10 server1 multipathd: dm-10: umount map (uevent)
\nAug 16 12:02:11 server1 kernel: ext3_abort called.
\nAug 16 12:02:11 server1 kernel: EXT3-fs error (device dm-10): ext3_put_super: Couldn’t clean up the journal
\nAug 16 12:03:55 server1 multipathd: dm-10: remove map (uevent)
\nAug 16 12:05:01 server1 multipathd: dm-10: add map (uevent)
\nAug 16 12:05:22 server1 kernel: kjournald starting. Commit interval 5 seconds
\nAug 16 12:05:22 server1 kernel: EXT3-fs warning (device dm-10): ext3_clear_journal_err: Filesystem error recorded from previous mount: IO failure
\nAug 16 12:05:22 server1 kernel: EXT3-fs warning (device dm-10): ext3_clear_journal_err: Marking fs in need of filesystem check.
\nAug 16 12:05:22 server1 kernel: EXT3-fs warning: mounting fs with errors, running e2fsck is recommended<\/p>\n
 <\/p>\n
 <\/p>\n
 <\/p>\n
Solution for this problem<\/p>\n
 <\/p>\n
Run this<\/span><\/p>\n
\n
Quote:<\/div>\n\n\n\n
umount -l \/tmp
\ne2fsck \/usr\/tmpDSK
\nmount -o loop,rw,noexec,nosuid \/usr\/tmpDSK \/tmp
\nchmod 1777 \/tmp<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n","protected":false},"excerpt":{"rendered":"
Mount point suddenly turned into READ ONLY <\/p>\n
A RAC system, the GRID mount point on eighth node suddenly turned into read-only mode:<\/p>\n
 [oracle@node8 ~]$ cd \/prod\/grid [oracle@node8 grid]$ touch 1 touch: cannot touch `1′: Read-only file system <\/p>\n
Instance is down due to that. We can see below error in OS log:<\/p>\n
 Sep 5 03:26:13 node8 […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[],"_links":{"self":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/3132"}],"collection":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3132"}],"version-history":[{"count":2,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/3132\/revisions"}],"predecessor-version":[{"id":3134,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/3132\/revisions\/3134"}],"wp:attachment":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3132"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3132"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3132"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}