Mount point suddenly turned into READ ONLY
A RAC system, the GRID mount point on eighth node suddenly turned into read-only mode:
[oracle@node8 ~]$ cd /prod/grid [oracle@node8 grid]$ touch 1 touch: cannot touch `1': Read-only file system
Instance is down due to that.
We can see below error in OS log:
Sep 5 03:26:13 node8 kernel: EXT3-fs error (device dm-226): ext3_lookup: unlinked inode 426196 in dir #2244752 Sep 5 03:26:13 node8 kernel: EXT3-fs error (device dm-226): ext3_lookup: unlinked inode 426199 in dir #2244752 Sep 5 03:26:13 node8 kernel: ext3_abort called. Sep 5 03:26:13 node8 kernel: EXT3-fs error (device dm-226): ext3_journal_start_sb: Detected aborted journal Sep 5 03:26:13 node8 kernel: Remounting filesystem read-only
Something wrong with inode in dir 2244752.
To found out what dir 2244752 is, we can use linux kernel command:
debugfs
But in this case i don’t care what the dir 2244752 is, we only want the issue to be fixed.
So we need to un-mount the mount point, use fsck to fix the Inode issue, and then re-mount the mount point.
We use losf/fuser command to find out all related processes and kill them, and through losetup command we ensure no resclusive device for this mount point, but still when we try to umount the mount point through “mount -f”, it reports mount point busy:
[root@node8 log]# fuser /prod/grid [root@node8 log]# [root@node8 log]# lsof |grep /prod/grid [root@node8 log]# [root@node8 log]# losetup -a [root@node8 log]#
Above case means the mount point is holding by kernel, not a process. In this case, we can use “umount -l” to trigger a lazy umount, but with lazy umount, it will damage the file system when we try to fsck.
Also it may prevent us when we want to mount it back.
A clean and safe way is to reboot the server. After that we succeed fixed the mount point read only issue.
ext3_lookup finds an “unlinked inode” and calls ext3_abort which results in a read-only filesystem
Issue
The ext3-filesystem went read-only. In /var/log/messages the following errors are logged:
EXT3-fs error (device dm-4): ext3_lookup: unlinked inode 91626760 in dir #19039379
Aborting journal on device dm-4.
ext3_abort called.
EXT3-fs error (device dm-4): ext3_journal_start_sb: Detected aborted journal
Remounting filesystem read-only
A couple of days later (after restoring with as written in the resolution):
EXT3-fs error (device dm-4): ext3_free_blocks_sb: bit already cleared for block 43951098
Aborting journal on device dm-4.
EXT3-fs error (device dm-4) in ext3_free_blocks_sb: Journal has aborted
EXT3-fs error (device dm-4) in ext3_reserve_inode_write: Journal has aborted
EXT3-fs error (device dm-4) in ext3_truncate: Journal has aborted
EXT3-fs error (device dm-4) in ext3_reserve_inode_write: Journal has aborted
EXT3-fs error (device dm-4) in ext3_orphan_del: Journal has aborted
EXT3-fs error (device dm-4) in ext3_reserve_inode_write: Journal has aborted
EXT3-fs error (device dm-4) in ext3_delete_inode: Journal has aborted
ext3_abort called.
EXT3-fs error (device dm-4): ext3_journal_start_sb: Detected aborted journal
Remounting filesystem read-only
__journal_remove_journal_head: freeing b_committed_data
nfs ext3 file system going read-only on two servers which are nfs cross mounted to each other.
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339312 in dir #318520
Aug 16 11:57:17 server1 kernel: Aborting journal on device dm-10.
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339315 in dir #318520
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339293 in dir #318520
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339321 in dir #318520
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339323 in dir #318520
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339276 in dir #318520
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339322 in dir #318520
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339291 in dir #318520
Aug 16 11:57:17 server1 kernel: ext3_abort called.
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_journal_start_sb: Detected aborted journal
Aug 16 11:57:17 server1 kernel: Remounting filesystem read-only
Aug 16 12:02:10 server1 multipathd: dm-10: umount map (uevent)
Aug 16 12:02:11 server1 kernel: ext3_abort called.
Aug 16 12:02:11 server1 kernel: EXT3-fs error (device dm-10): ext3_put_super: Couldn’t clean up the journal
Aug 16 12:03:55 server1 multipathd: dm-10: remove map (uevent)
Aug 16 12:05:01 server1 multipathd: dm-10: add map (uevent)
Aug 16 12:05:22 server1 kernel: kjournald starting. Commit interval 5 seconds
Aug 16 12:05:22 server1 kernel: EXT3-fs warning (device dm-10): ext3_clear_journal_err: Filesystem error recorded from previous mount: IO failure
Aug 16 12:05:22 server1 kernel: EXT3-fs warning (device dm-10): ext3_clear_journal_err: Marking fs in need of filesystem check.
Aug 16 12:05:22 server1 kernel: EXT3-fs warning: mounting fs with errors, running e2fsck is recommended
Solution for this problem
Run this
| umount -l /tmp e2fsck /usr/tmpDSK mount -o loop,rw,noexec,nosuid /usr/tmpDSK /tmp chmod 1777 /tmp |
Recent Comments