{"id":529,"date":"2012-07-10T10:01:18","date_gmt":"2012-07-10T02:01:18","guid":{"rendered":"http:\/\/rmohan.com\/?p=529"},"modified":"2012-07-10T10:01:18","modified_gmt":"2012-07-10T02:01:18","slug":"grub-fallback-boot-good-kernel-if-new-one-crashes","status":"publish","type":"post","link":"https:\/\/mohan.sg\/?p=529","title":{"rendered":"Grub Fallback: Boot good kernel if new one crashes"},"content":{"rendered":"

Grub Fallback: Boot good kernel if new one crashes<\/p>\n

t\u2019s hard to believe but I didn\u2019t know about Grub fallback<\/strong>feature. So every time when I needed to reboot remote server into a new kernel I had to test it on local server to make sure it won\u2019t panic on remote unit. And if kernel panic still happened I had to ask somebody who has physical access to the server to reboot the hardware choose proper kernel in Grub. It\u2019s all boring and not\u00a0healthful \u2013 it\u2019s much better to use Grub\u2019s native fallback feature.<\/p>\n

Grub is default boot loader in most Linux distributions today, at least major distros like Centos\/Fedora\/RedHat, Debian\/Ubuntu\/Mint, Arch use Grub. This makes it possible to use Grub fallback<\/a> feature just out of the box. Here is example scenario.<\/p>\n

There is remote server hosted in New Zealand and you (sitting in Denmark) have access to it over the network only (no console server). In this case you cannot afford that the new kernel makes server unreachable, e.g. if new kernel crash during boot it won\u2019t load network interface drivers so your Linux box won\u2019t appear online until somebody reboots it into workable kernel. Thankfully Grub can be configured to try loading new kernel once and if it fails Grub will load another kernel according to configuration. You can see my example grub.conf below:<\/p>\n

\n
\n
default=saved\r\ntimeout=5\r\nsplashimage=(hd0,1)\/boot\/grub\/splash.xpm.gz\r\nhiddenmenu\r\nfallback 0 1\r\ntitle Fedora OpenVZ (2.6.32-042stab053.5)\r\n        root (hd0,1)\r\n        kernel \/boot\/vmlinuz-2.6.32-042stab053.5 ro root=UUID=6fbdddf9-307c-49eb-83f5-ca1a4a63f584 rd_MD_UUID=1b9dc11a:d5a084b5:83f6d993:3366bbe4 rd_NO_LUKS rd_NO_LVM rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=sv-latin1 rhgb quiet crashkernel=auto\r\n        initrd \/boot\/initramfs-2.6.32-042stab053.5.img\r\n        savedefault fallback\r\ntitle Fedora (2.6.35.12-88.fc14.i686)\r\n        root (hd0,1)\r\n        kernel \/boot\/vmlinuz-2.6.35.12-88.fc14.i686 ro root=UUID=6fbdddf9-307c-49eb-83f5-ca1a4a63f584 rd_MD_UUID=1b9dc11a:d5a084b5:83f6d993:3366bbe4 rd_NO_LUKS rd_NO_LVM rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=sv-latin1 rhgb quiet\r\n        initrd \/boot\/initramfs-2.6.35.12-88.fc14.i686.img\r\n        savedefault fallback<\/pre>\n<\/div>\n<\/div>\n
According to this configuration Grub will try to load \u2018Fedora OpenVZ\u2019 kernel once and if it fails system will be loaded into good \u2018Fedora\u2019 kernel. If \u2018Fedora OpenVZ\u2019 loads well you\u2019ll be able to reach the server over the network after reboot. Notice lines \u2018default=saved\u2019<\/em> and \u2018savedefault fallback\u2019<\/em> which are mandatory to make fallback feature working.<\/p>\n
Alternative way<\/h4>\n
I\u2019ve heard that official Grub fallback feature may work incorrectly on RHEL5 (and Centos 5) so there is elegant workaround (found here<\/a>):<\/p>\n
1. Add param \u2018panic=5? to your new kernel line so it looks like below:<\/p>\n
\n
\n
title Fedora OpenVZ (2.6.32-042stab053.5)\r\n        root (hd0,1)\r\n        kernel \/boot\/vmlinuz-2.6.32-042stab053.5 ro root=UUID=6fbdddf9-307c-49eb-83f5-ca1a4a63f584 rd_MD_UUID=1b9dc11a:d5a084b5:83f6d993:3366bbe4 rd_NO_LUKS rd_NO_LVM rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=sv-latin1 rhgb quiet crashkernel=auto panic=5\r\n        initrd \/boot\/initramfs-2.6.32-042stab053.5.img<\/pre>\n<\/div>\n<\/div>\n
This param will make crashed kernel to reboot itself in 5 seconds.<\/p>\n
2. Point default<\/em> Grub param to good kernel, e.g. \u2018default=0?.<\/p>\n
3. Type in the following commands (good kernel appears in grub.conf first and new kernel is second one):<\/p>\n
\n
\n
# grub\r\n\r\ngrub> savedefault --default=1 --once\r\nsavedefault --default=1 --once\r\ngrub> quit<\/pre>\n<\/div>\n<\/div>\n
This will make Grub to boot into new kernel once and if it fails it will load good kernel. Now you can reboot the server and make sure it will 100% appear online in a few minutes. I usually prefer native Grub fallback feature but if you see it doesn\u2019t work for you it makes sense to try above mentioned workaround.<\/p>\n","protected":false},"excerpt":{"rendered":"
Grub Fallback: Boot good kernel if new one crashes<\/p>\n
t\u2019s hard to believe but I didn\u2019t know about Grub fallbackfeature. So every time when I needed to reboot remote server into a new kernel I had to test it on local server to make sure it won\u2019t panic on remote unit. And if kernel panic […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[28],"tags":[],"_links":{"self":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/529"}],"collection":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=529"}],"version-history":[{"count":4,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/529\/revisions"}],"predecessor-version":[{"id":533,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/529\/revisions\/533"}],"wp:attachment":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=529"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=529"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=529"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}