Rug pulling
This involves AWS EC2 AMI deployment/setup automation, and if that makes you shudder then look away now.
Last week I was completing some automation that takes a blank EC2 through a carefully scripted sequence of steps to produce a production-ready platform for a specific live service. It’s not Chef, or Puppet or any of a number of config/build automation solutions. It’s just a simple shell script that incrementally adds functionality either to enhance its own configuration/build abilities and/or support the target setup. It’s close enough to the OS to support the granularity of control that I need, while being abstract enough to be reasonably compact. The current script is just shy of a thousand lines.
This script starts with the “reasonable” assumption that it has the minimal functionality provided by the default OS (Amazon Machine Image with Amazon Linux 2, AKA AL2), does a quick “yum update“, mounts drives, defines swap space, adds a repository of very useful tools via “amazon-linux-extras install epel” and continues to add more tools, libraries, directories and much more, through multiple OS reboots where necessary until eventually I have a working system.
Setting up the initial EC2 for testing can be automated so that there are real cloud instances to use during development of this scripted process. However, I have found it far more flexible (and efficient in time and money) to deploy an AL2 instance on a local VirtualBox during this time. This is something that Amazon intentionally supports. The free OS images are available to download and I have my own script that will create new Virtual Machine instances from these images in a few seconds, ready to test-run my platform installation script.
Last week I had reached the point where the entire automation was reliable for all the use cases that were required. Now the testing needed to move from VirtualBox instances to EC2 instances. To do my first scripted installation I needed a fresh EC2 and I decided to manually create one using the AWS console. It only takes a minute to get an instance set up.
This is the point where the rug was pulled from under me.
Having clicked the “Launch Instance” button from the EC2 part of the AWS console, I was presented with the Application and OS Images options, and I expected that the AWS Amazon Linux 2 would be the initial (default) selection. Instead, I was presented with this new default option:
Amazon Linux 2023 AMI
ami-09dd5f12915cfb387 (64-bit (x86), uefi-preferred) / ami-0de2a2552e7fe4b4d (64-bit (Arm), uefi)
Virtualization: hvm  ENA enabled: true  Root device type: ebs
Hello?
Amazon Linux 2 is now the second option on the list of Amazon Linux variations. While I had been busy creating an installation for AL2 (which I had also tested on RedHat-like environments such as CentOS and Rocky) and using an Amazon-supplied VirtualBox image, they had been busy launching a new version of the OS, Amazon Linux 2023, together with a schedule for the next few years offering a new version every 2 years.
There are a number of differences that I must address:
- SELinux is now enabled by default. I’m OK with that as I try to make use of whatever security features are present, but as it was not used by default in AL2 it is not obvious if the custom installation will trip up the AL2023 security. I will have to monitor the SELinux logs. (Thankfully it defaults to permissive.) Still, this is wading into unexplored territory.
- Amazon Linux is no longer compatible with any particular release of Fedora. This could further limit my deployment/development options as I like to work with multiple distros to limit lock-in.
- The package manager changed from yum to dnf, or if you want to be pedantic: from the modified version of the original Yellowdog UPdater (YUP) known as Yellowdog Updater, Modified (YUM) to the Dandified Yellowdog Updater Modified (DNF). Give me a break! But seriously, while there is a strong family resemblance, replacing yum with dnf is not all plain sailing.
- EPEL is gone! The Fedora Extra Packages for Enterprise Linux won’t work on AL2023 because of a pile of compatibility issues. AL2 was much the same as CentOS 7, and therefore most of the EPEL packages were compatible, but not so for AL2023. This is an extra headache because a lot of my scripted installs would make use of EPEL.
Dealing with yum/dnf should be easy enough, with the help of a bit of abstraction. Especially if I want to keep some backwards compatibility with the AL2 installations. The loss of EPEL could be a bigger headache.
In the meantime, I have one even bigger (hopefully short-term) headache: despite the written promises, Amazon has not (yet) released an on-prem deployable image for VirtualBox or equivalent. People have been complaining for weeks about this. Until I can get an image to use locally I have no choice but to experiment and develop within the cloud itself.
As for my AL2 deployment that was ready to go, that’s now on hold because of the demotion of AL2. I will be reading the AL202X Release Notes intensively to see what other changes and potential pitfalls I have to deal with before I restart the scripted deployment activity. Nothing like a bit of light reading on a sunny evening…
Categorised as: Coding, Operating Systems, Technology