{"id":7663,"date":"2018-07-05T10:11:00","date_gmt":"2018-07-05T02:11:00","guid":{"rendered":"http:\/\/rmohan.com\/?p=7663"},"modified":"2018-07-05T10:11:12","modified_gmt":"2018-07-05T02:11:12","slug":"migrating-to-amazon-linux-2","status":"publish","type":"post","link":"https:\/\/mohan.sg\/?p=7663","title":{"rendered":"Migrating to Amazon Linux 2"},"content":{"rendered":"<p><a href=\"https:\/\/forums.aws.amazon.com\/ann.jspa?annID=5653\" target=\"_blank\" rel=\"external noopener\">AWS also announced<\/a> that Amazon Linux 2018.03 is the last release for the current generation of Amazon Linux and will be supported until June 30, 2020. Therefore, you have to come up with a migration plan.<\/p>\n<p>Amazon Linux 2 comes with the same benefits as Amazon Linux, but it adds some new capabilities:<\/p>\n<ul>\n<li>long-term support: Amazon Linux 2 supports each LTS release for five years<\/li>\n<li>on-premises support: virtual machine images for on-premises development and testing are available<\/li>\n<li>systemd: replacing SystemVinit<\/li>\n<li>extras library: provides up-to-date versions of software bundles such as nginx<\/li>\n<\/ul>\n<p>Let\u2019s dive into some of the changes in more detail. At the end of the post, I will also outline some pitfalls I encountered when migrating our <a href=\"https:\/\/github.com\/widdix\/aws-cf-templates\/pull\/145\" target=\"_blank\" rel=\"external noopener\">Free Templates for AWS CloudFormation<\/a> to Amazon Linux 2.<\/p>\n<p>Further reading: <a href=\"https:\/\/aws.amazon.com\/amazon-linux-2\/release-notes\/\" target=\"_blank\" rel=\"external noopener\">Release Notes<\/a>, <a href=\"https:\/\/aws.amazon.com\/amazon-linux-2\/faqs\/\" target=\"_blank\" rel=\"external noopener\">FAQs<\/a>, <a href=\"https:\/\/aws.amazon.com\/blogs\/aws\/amazon-linux-2-modern-stable-and-enterprise-friendly\/\" target=\"_blank\" rel=\"external noopener\">AWS Blog Post<\/a>, <a href=\"https:\/\/aws.amazon.com\/about-aws\/whats-new\/2017\/12\/introducing-amazon-linux-2\" target=\"_blank\" rel=\"external noopener\">Announcement<\/a><\/p>\n<h2 id=\"Long-term-support\">Long-term support<\/h2>\n<p>The Amazon Linux delivers a continuous flow of updates that allow you to roll from one version of the Amazon Linux AMI to the most recent. A <code>yum update<\/code> always moves your system to the latest Amazon Linux version. There were no versions of Amazon Linux available, only snapshots.<\/p>\n<p>Amazon Linux 2 changes this. You will have Amazon Linux 2 versions that are supplied with updates for five years. Once a new Amazon Linux 2 LTS release becomes available, no breaking changes will be introduced by AWS for this release.<\/p>\n<h2 id=\"systemd\">systemd<\/h2>\n<p>Amazon Linux uses <a href=\"https:\/\/en.wikipedia.org\/wiki\/Init\" target=\"_blank\" rel=\"external noopener\">SysVinit<\/a> to bootstrap the Linux user space and to manage system processes after booting. This procedure is usually called init. One of the major drawbacks of <code>SysVinit<\/code> is that it starts tasks serially, waiting for each to finish loading before moving on to the next. This can result in long delays during boot.<\/p>\n<p>Amazon Linux 2 uses <a href=\"https:\/\/en.wikipedia.org\/wiki\/Systemd\" target=\"_blank\" rel=\"external noopener\">systemd<\/a> as the init system. <code>systemd<\/code> executes elements of its startup sequence in parallel, which is faster than the traditional serial approach from <code>SysVinit<\/code>. <code>systemd<\/code> can also ensure that a service is running (e.g., it restarts a service if it crashed).<\/p>\n<p><code>systemd<\/code> is not just the name of the init system daemon but also refers to the entire software bundle around it, which includes:<\/p>\n<ul>\n<li><code>journald<\/code>: responsible for event logging (replaces syslog)<\/li>\n<li><code>udevd<\/code>: device manager for the Linux kernel, which handles the \/dev directory and all user space actions when adding\/removing devices<\/li>\n<li><code>logind<\/code>: manages user logins and seats in various ways.<\/li>\n<\/ul>\n<p>I will not cover <code>udevd<\/code> and <code>logind<\/code> in this post. You should not get in touch with them as a normal user like me. Keep in mind that networking configuration is not controlled by <code>networkd<\/code> (also part of <code>systemd<\/code> software bundle). Instead, networking configuration is controlled by <a href=\"https:\/\/cloudinit.readthedocs.io\" target=\"_blank\" rel=\"external noopener\">cloud-init<\/a> which is triggered by <code>systemd<\/code> <a href=\"https:\/\/cloudinit.readthedocs.io\/en\/latest\/topics\/boot.html\" target=\"_blank\" rel=\"external noopener\">several times during boot<\/a>. <code>cloud-init<\/code> handles early initialization of an EC2 instance (also works with other vendors).<\/p>\n<p>Further reading: <a href=\"https:\/\/www.freedesktop.org\/software\/systemd\/man\/systemd.html\" target=\"_blank\" rel=\"external noopener\">systemd man page<\/a><\/p>\n<h3 id=\"Reading-logs-from-journald\">Reading logs from journald<\/h3>\n<p>To read all system logs (journal in journald terminology), starting with the oldest entry, run <code>journalctl<\/code>. The output is paged through <code>less<\/code> by default. Which means you can scroll down \/ up an entry with the <code>DOWN<\/code> \/ <code>UP<\/code> arrow keys, or scroll a full page down\/up with the <code>SPACE<\/code> \/ <code>b<\/code> keys. Press the <code>q<\/code> key to quit. To reverse the order, run <code>journalctl -r<\/code>.<\/p>\n<p>To show only the most recent journal entries, and continuously print new entries, run <code>journalctl -f<\/code> (like a <code>tail -f<\/code>).<\/p>\n<p>There are many ways to filter the output. Based on priority, run <code>journalctl -p err<\/code> to get levels alert, crit, and err (using syslog log levels). Based on the unit, run <code>journalctl -u sshd<\/code> to get all entries for <code>sshd<\/code>. Check the further reading links for more information.<\/p>\n<p>Keep in mind that some applications still write logs to <code>\/var\/log<\/code>. Journald also forwards logs to <code>rsyslog<\/code> which is configured (<code>\/etc\/rsyslog.conf<\/code>) to write some of them to files in <code>\/var\/log<\/code>.<\/p>\n<p>Further reading: <a href=\"https:\/\/www.freedesktop.org\/software\/systemd\/man\/journalctl.html\" target=\"_blank\" rel=\"external noopener\">journalctl man page<\/a><\/p>\n<h3 id=\"Controlling-systemd-services\">Controlling systemd services<\/h3>\n<p>To start a service (unit in systemd terminology), you run:<\/p>\n<figure class=\"highlight crmsh\">\n<table>\n<tbody>\n<tr>\n<td class=\"code\">\n<pre><span class=\"line\">systemctl <span class=\"literal\">start<\/span> awslogsd.service<\/span><\/pre>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n<p>To make sure a service (unit in systemd terminology) is started during boot\/reboot, you run:<\/p>\n<figure class=\"highlight routeros\">\n<table>\n<tbody>\n<tr>\n<td class=\"code\">\n<pre><span class=\"line\">systemctl <span class=\"builtin-name\">enable<\/span> awslogsd.service<\/span><\/pre>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n<p>There are many other commands. E.g., you can also reboot the system:<\/p>\n<figure class=\"highlight ebnf\">\n<table>\n<tbody>\n<tr>\n<td class=\"code\">\n<pre><span class=\"line\"><span class=\"attribute\">systemctl reboot<\/span><\/span><\/pre>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n<p>Further reading: <a href=\"https:\/\/www.freedesktop.org\/software\/systemd\/man\/systemctl.html\" target=\"_blank\" rel=\"external noopener\">systemctl man page<\/a><\/p>\n<h2 id=\"Extras-Library\">Extras Library<\/h2>\n<p>The Extras Library (aka Amazon Linux Extras Repository or Extras mechanism), provides a way to install up-to-date software bundles (topics in Amazon Linux 2 terminology) without impacting the stability of the rest of the operating system.<\/p>\n<blockquote><p>Extras Library is not covered by LTS!<\/p><\/blockquote>\n<p>To get a list of available topics, run:<\/p>\n<figure class=\"highlight bash\">\n<table>\n<tbody>\n<tr>\n<td class=\"code\">\n<pre><span class=\"line\">$ amazon-linux-extras list<\/span>\r\n<span class=\"line\">  0  ansible2                 available  [ =2.4.2 ]<\/span>\r\n<span class=\"line\">  1  emacs                    available  [ =25.3 ]<\/span>\r\n<span class=\"line\">  2  memcached1.5             available  [ =1.5.1 ]<\/span>\r\n<span class=\"line\">  3  nginx1.12                available  [ =1.12.2 ]<\/span>\r\n<span class=\"line\">  4  postgresql9.6            available  [ =9.6.6  =9.6.8 ]<\/span>\r\n<span class=\"line\">  5  python3                  available  [ =3.6.2 ]<\/span>\r\n<span class=\"line\">  6  redis4.0                 available  [ =4.0.5 ]<\/span>\r\n<span class=\"line\">  7  R3.4                     available  [ =3.4.3 ]<\/span>\r\n<span class=\"line\">  8  rust1                    available  [ =1.22.1  =1.26.0 ]<\/span>\r\n<span class=\"line\">  9  vim                      available  [ =8.0 ]<\/span>\r\n<span class=\"line\"> 10  golang1.9                available  [ =1.9.2 ]<\/span>\r\n<span class=\"line\"> 11  ruby2.4                  available  [ =2.4.2  =2.4.4 ]<\/span>\r\n<span class=\"line\"> 12  nano                     available  [ =2.9.1 ]<\/span>\r\n<span class=\"line\"> 13  php7.2                   available  [ =7.2.0  =7.2.4  =7.2.5 ]<\/span>\r\n<span class=\"line\"> 14  lamp-mariadb10.2-php7.2  available  [ =10.2.10_7.2.0  =10.2.10_7.2.4  =10.2.10_7.2.5 ]<\/span>\r\n<span class=\"line\"> 15  libreoffice              available  [ =5.0.6.2_15 ]<\/span>\r\n<span class=\"line\"> 16  gimp                     available  [ =2.8.22 ]<\/span>\r\n<span class=\"line\"> 17  docker=latest            enabled    [ =17.12.1  =18.03.1 ]<\/span>\r\n<span class=\"line\"> 18  mate-desktop1.x          available  [ =1.19.0  =1.20.0 ]<\/span>\r\n<span class=\"line\"> 19  GraphicsMagick1.3        available  [ =1.3.29 ]<\/span>\r\n<span class=\"line\"> 20  tomcat8.5                available  [ =8.5.31 ]<\/span><\/pre>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n<p>To install an topic, run <code>amazon-linux-extras install &lt;topic&gt;<\/code> (e.g., <code>amazon-linux-extras install ruby2.4<\/code>).<\/p>\n<p>If you install (or only enable) a topic, a new repository (plus two for sources and debuginfo) is configured in <code>\/etc\/yum.repos.d\/amzn2-extras.repo<\/code>.<\/p>\n<h2 id=\"Pitfalls\">Pitfalls<\/h2>\n<p>I migrated <a href=\"https:\/\/github.com\/widdix\/aws-cf-templates\/pull\/145\" target=\"_blank\" rel=\"external noopener\">Free Templates for AWS CloudFormation<\/a> to Amazon Linux 2. In the following, I will outline the problems I was faced with and how I worked around them.<\/p>\n<h3 id=\"The-awslogs-agent-was-renamed\">The awslogs agent was renamed<\/h3>\n<p>The <code>awslogs<\/code> agent was renamed to <code>awslogsd<\/code> but you still install it via <code>yum install awslogs<\/code>.<\/p>\n<p>You can start (activate in systemd terminology) awslogs with <code>systemctl start awslogsd.service<\/code> (shortcut: <code>systemctl start awslogsd<\/code>).<\/p>\n<h3 id=\"The-awslogs-agent-does-not-support-journald\">The awslogs agent does not support journald<\/h3>\n<p>awslogs agent cannot read logs directly from the journal. <code>journald<\/code> fowards all logs to <code>rsyslog<\/code> which is configured (<code>\/etc\/rsyslog.conf<\/code>) to write some of the logs to files in <code>\/var\/log<\/code> from where the awslogs agent can pick them up.<\/p>\n<h3 id=\"Where-are-the-log-files\">Where are the log files?<\/h3>\n<p><code>\/var\/log<\/code> does not contain all system logs anymore.<\/p>\n<p>If in doubt, you can access all system logs with <code>journalctl<\/code>.<\/p>\n<h3 id=\"Ruby-is-missing\">Ruby is missing<\/h3>\n<p>Ruby is no longer installed by default. This breaks <code>cfn-init<\/code> if you want to install RubyGems.<\/p>\n<p>You can install Ruby 2.0 with <code>yum install ruby<\/code> or Ruby 2.4 with <code>amazon-linux-extras install ruby2.4<\/code>.<\/p>\n<h3 id=\"netcat-is-missing\">netcat is missing<\/h3>\n<p>netcat (or <code>nc<\/code>) is no longer installed by default.<\/p>\n<p>You can install <code>ncat<\/code> with <code>yum install nmap-ncat<\/code>, but this will install nmap based ncat which behaves differently (e.g., no <code>-z<\/code> flag anymore). <a href=\"https:\/\/unix.stackexchange.com\/questions\/368155\/what-are-the-differences-between-ncat-nc-and-netcat\" target=\"_blank\" rel=\"external noopener\">Learn more<\/a><\/p>\n<h3 id=\"Nginx-package-not-available-by-default\">Nginx package not available by default<\/h3>\n<p><code>nginx<\/code> is no longer part of the default repository.<\/p>\n<figure class=\"highlight bash\">\n<table>\n<tbody>\n<tr>\n<td class=\"code\">\n<pre><span class=\"line\">$ yum install nginx<\/span>\r\n<span class=\"line\">Failed to <span class=\"built_in\">set<\/span> locale, defaulting to C<\/span>\r\n<span class=\"line\">Loaded plugins: langpacks, update-motd<\/span>\r\n<span class=\"line\">No package nginx available.<\/span>\r\n<span class=\"line\">Error: Nothing to <span class=\"keyword\">do<\/span><\/span><\/pre>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n<p>To install nginx, use the new Amazon Linux Extras Repository <code>amazon-linux-extras install nginx1.12<\/code>.<\/p>\n<h3 id=\"EPEL-repository-is-missing\">EPEL repository is missing<\/h3>\n<p>The EPEL repository (Extra Packages for Enterprise Linux) is no longer installed by default or available to install. The Extras Library replaces the EPEL repository but contains only a fraction of the packages which may causes troubles during your migration.<\/p>\n<h3 id=\"NAT-and-ECS-optimized-AMIs-are-missing\">NAT and ECS optimized AMIs are missing<\/h3>\n<p>NAT and ECS optimized AMI are not available. You can replace your NAT instances with NAT Gateways to get around this problem. But for ECS workloads there is no easy workaround. I advise waiting for news from AWS regarding the ECS optimized AMI.<\/p>\n<h3 id=\"cfn-init-is-not-integrated-with-the-Extras-Library\">cfn-init is not integrated with the Extras Library<\/h3>\n<p>You can not install packages from the Extras Library with the package mechanism in <code>cfn-init<\/code> easily. <code>cfn-init<\/code> is the way how you can install software onto EC2 instances managed by CloudFormation.<\/p>\n<p>There can either run <code>amazon-linux-extras enable &lt;topic&gt;<\/code> before running <code>cfn-init<\/code> which than can install the package by using the package mechanism. Or you can use two config sets. The first config sets uses the command mechanism to enable the topic. The second config set uses the package mechanism to install the enabled package. You have to use two config sets because commands run after package installation. Here is an example:<\/p>\n<figure class=\"highlight yaml\">\n<table>\n<tbody>\n<tr>\n<td class=\"code\">\n<pre><span class=\"line\"><span class=\"attr\">AutoScalingGroup:<\/span><\/span>\r\n<span class=\"line\"><span class=\"attr\">  Type:<\/span> <span class=\"string\">'AWS::AutoScaling::AutoScalingGroup'<\/span><\/span>\r\n<span class=\"line\"><span class=\"attr\">  Properties:<\/span><\/span>\r\n<span class=\"line\">    <span class=\"comment\"># [...]<\/span><\/span>\r\n<span class=\"line\"><span class=\"attr\">LaunchConfiguration:<\/span><\/span>\r\n<span class=\"line\"><span class=\"attr\">  Type:<\/span> <span class=\"string\">'AWS::AutoScaling::LaunchConfiguration'<\/span><\/span>\r\n<span class=\"line\"><span class=\"attr\">  Metadata:<\/span><\/span>\r\n<span class=\"line\">    <span class=\"string\">'AWS::CloudFormation::Init'<\/span><span class=\"string\">:<\/span><\/span>\r\n<span class=\"line\"><span class=\"attr\">      configSets:<\/span><\/span>\r\n<span class=\"line\"><span class=\"attr\">        default:<\/span> <span class=\"string\">[extras,<\/span> <span class=\"string\">config]<\/span><\/span>\r\n<span class=\"line\"><span class=\"attr\">      extras:<\/span><\/span>\r\n<span class=\"line\"><span class=\"attr\">        commands:<\/span><\/span>\r\n<span class=\"line\"><span class=\"attr\">          a_enable_nginx:<\/span><\/span>\r\n<span class=\"line\"><span class=\"attr\">            command:<\/span> <span class=\"string\">'amazon-linux-extras enable nginx1.12'<\/span><\/span>\r\n<span class=\"line\"><span class=\"attr\">            test:<\/span> <span class=\"string\">\"[ ! grep -Fxq '[amzn2extra-nginx1.12]' \/etc\/yum.repos.d\/amzn2-extras.repo ]\"<\/span><\/span>\r\n<span class=\"line\"><span class=\"attr\">      config:<\/span><\/span>\r\n<span class=\"line\"><span class=\"attr\">        packages:<\/span><\/span>\r\n<span class=\"line\"><span class=\"attr\">          yum:<\/span><\/span>\r\n<span class=\"line\"><span class=\"attr\">            nginx:<\/span> <span class=\"string\">[]<\/span> <span class=\"comment\"># will install nginx1.12<\/span><\/span>\r\n<span class=\"line\"><span class=\"attr\">  Properties:<\/span><\/span>\r\n<span class=\"line\">    <span class=\"comment\"># [...]<\/span><\/span>\r\n<span class=\"line\"><span class=\"attr\">    UserData:<\/span><\/span>\r\n<span class=\"line\">      <span class=\"string\">'Fn::Base64'<\/span><span class=\"string\">:<\/span> <span class=\"string\">!Sub<\/span> <span class=\"string\">|<\/span><\/span>\r\n<span class=\"line\"><span class=\"string\">        #!\/bin\/bash -x<\/span><\/span>\r\n<span class=\"line\"><span class=\"string\">        \/opt\/aws\/bin\/cfn-init -v --stack ${AWS::StackName} --resource LaunchConfiguration --region ${AWS::Region}<\/span><\/span>\r\n<span class=\"line\"><span class=\"string\">        \/opt\/aws\/bin\/cfn-signal -e $? --stack ${AWS::StackName} --resource AutoScalingGroup --region ${AWS::Region}<\/span><\/span><\/pre>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n<h2 id=\"Summary\">Summary<\/h2>\n<p>Amazon Linux 2 is the new default for running Linux workloads on AWS. Amazon Linux 2 benefits from systemd, LTS, and a new extras library. There are a few pain points when migrating, most notably the missing EPEL repository. Besides that, you should spend some time to understand how systemd works, because that\u2019s central in modern Linux operating systems.<\/p>\n<p>It\u2019s time to plan your migration from Amazon Linux now!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>AWS also announced that Amazon Linux 2018.03 is the last release for the current generation of Amazon Linux and will be supported until June 30, 2020. Therefore, you have to come up with a migration plan.<\/p>\n<p>Amazon Linux 2 comes with the same benefits as Amazon Linux, but it adds some new capabilities:<\/p>\n<p> long-term support: [&#8230;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[49],"tags":[],"_links":{"self":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/7663"}],"collection":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=7663"}],"version-history":[{"count":1,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/7663\/revisions"}],"predecessor-version":[{"id":7664,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/7663\/revisions\/7664"}],"wp:attachment":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=7663"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=7663"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=7663"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}