Over the weekend of 8th Feb 2020 I upgraded our AWS production instances to the new nitro series 3 platform plumping for the AMD based T3a family as this attracted a 10 percent discount and we are looking to reduce our AWS costs as they are increasing month by month.
Whereas our customers upgrades (using Ubuntu 14.04 like ours) went smoothly, ours of course didn’t. Turned out to be a bug in 14.04 which didn’t add nvme drivers to initrd. In nitro instances the storage is presented as nvme despite using magnetic disks or ssd so you need an nvme driver.
So to prepare an instance for upgrade to nitro we need to install drivers for the nitro network card and nvme interface. This is done using the guide here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking-ena.html but I’ll pick out the relevant bits here.
Fortunately the nitro drivers and nvme drivers are included in all recent kernels so all we have to do is install them:
sudo apt install linux-aws
Amazon provides a script to test whether the drivers have been loaded here: https://aws.amazon.com/premiumsupport/knowledge-center/boot-error-linux-m5-c5/
Unfortunately as mentioned earlier Ubuntu 14.04 fails to put the nvme driver into initrd so we have to explicitly tell it to:
sudo su echo nvme >> /etc/initramfs-tools/modules update-initramfs -c -k all
On a new nitro install Amazon also puts nvme_core.io_timeout=4294967295 into grub so I suggest we do the same:
sudo nano /etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0 nvme_core.io_timeout=4294967295 zswap.enabled=1 zswap.compressor=lz4" sudo update-grub (RHEL/CENTOS: sudo grub2-mkconfig -o /boot/grub2/grub.cfg)
Ubuntu lets anything in /etc/default/grub.d/ override the default so you might have to add the entries to a file similar to /etc/default/grub.d/50-cloudimg-settings.cfg too, remembering to sudo update-grub afterwards.
If there are things already in there like console commands then leave them there. Since zswap is running in every new kernel I’ve also added entries to turn it on. Whilst swap isn’t really needed in powerful instances, money can still be saved by using a smaller instance and swap. Zswap compresses pages in memory instead of paging to disk but still needs a disk based backup swap file to work. Zswap uses a fifth of the RAM so the swap file should be a fifth of the RAM too – or thereabouts. Its often easier to do a quarter of the RAM.
sudo su echo lz4 >> /etc/initramfs-tools/modules echo lz4_compress >> /etc/initramfs-tools/modules update-initramfs -c -k all fallocate -l 1G /swapfile chmod 600 /swapfile mkswap /swapfile
And add the swap file to fstab:
sudo nano /etc/fstab /swapfile swap swap defaults 0 0
After rebooting you can see if zswap is enabled:
grep -R . /sys/module/zswap/parameters /sys/module/zswap/parameters/zpool:zbud /sys/module/zswap/parameters/max_pool_percent:20 /sys/module/zswap/parameters/enabled:Y /sys/module/zswap/parameters/compressor:lz4
To enable ENA on AWS the user guide says we need to shutdown the instance and use the AWS CLI.
export AWS_PROFILE=xyze aws ec2 modify-instance-attribute --instance-id i-xxxxxxxx --ena-support
Now the actual upgrade is easy – but I’d still take a snapshot to be on the safe side. To upgrade we close the instance down, change the instance type to the same as it was except in the t3a series – so t2.micro becomes t3a.micro and then reboot. New nitro installs turn on the T2/T3 Unlimited by default so you may have to enable this on the AWS Console too. Then turn it back on – booting seems much quicker on nitro.
Postscript
On AWS & Google cloud LZ4 doesn’t load at boot time so the following trick in a crontab will load it:
root@instance-1:~# crontab -e . . . # m h dom mon dow command @reboot echo lz4 > /sys/module/zswap/parameters/compressor
Using zstd in place of lz4 gives better compression although decompression is two thirds slower. In theory better compression would take up less RAM leaving more to play with.