sudo apt install python3-pip python3-venv
python3 -m venv env
source env/bin/activate
pip3 install ansible

The first thing I created was a hosts.ini file:

cat hosts.ini

[xyze]
backup.xyze       ansible_ssh_extra_args="-Jxyze@bastion.xyze"
git.xyze          ansible_ssh_extra_args="-Jxyze@bastion.xyze"
icinga.xyze       ansible_ssh_extra_args="-Jxyze@bastion.xyze"
tickets.xyze      ansible_ssh_extra_args="-Jxyze@bastion.xyze"
bastion.xyze

[client1]
website.client1

[client2]
website.client2
.
.
.

Our DNS is on AWS’s Route 53 and I’ve created private zones for each of our clients and us, so when we’re logged into our VPN the addresses above can be resolved. Our machines can only be accessed via an intermediate bastion host so the jump host is included in the hosts file.

Then I created some apt playbooks:

cat apt-safe-upgrade.yml 
---
- hosts: xyze, client1, client2
  remote_user: xyze
  become: true
  become_method: sudo
  gather_facts: false

  tasks:
    - name: apt-get dist-upgrade
      apt:
        update_cache: true
        upgrade: safe

cat apt-reboot.yml 
---
- hosts: xyze, client1, client2
  remote_user: xyze
  become: yes
  become_method: sudo
  gather_facts: no

  tasks:
  - name: Check if a reboot is required
    register: reboot_required_file
    stat: path=/var/run/reboot-required get_md5=no

  - name: Reboot box if kernel/libs updated and requested by the system
    shell: sleep 10 && /sbin/shutdown -r now 'Rebooting box to update system libs/kernel as needed'
    args:
        removes: /var/run/reboot-required
    async: 300
    poll: 0
    ignore_errors: true
    when: reboot_required_file.stat.exists == true

cat apt-autoremove.yml 
---
- hosts: xyze, client1, client2
  remote_user: xyze
  become: true
  become_method: sudo
  gather_facts: false

  tasks:
    - name: apt autoremove
      apt:
        autoremove: true

Unfortunately some of my playbooks had errors which were found by the Ansible Lint tool. This can be installed via pip and installs yamllint which can be run first:

pip3 install ansible-lint

Playbooks are executed in the following manner:

ansible-playbook -i hosts.ini -l xyze,client1,client2 apt-safe-upgrade.yml -f10

Running ‘ansible-playbook’ on its own will give a list of all the options or passing ‘-h’ or’–help’ to it will display the same thing.

In our example above we are using hosts.ini as the inventory, and limiting the hosts it runs on to a subset. This can be a single host within the hosts.ini file or a group of hosts like xyze,client1 or it’ll run on all the hosts if you pass the limiting option ‘all’. Normally it’ll fork into 5 parallel processes but I’ve doubled that to 10 in this example.

A check or test run can be performed and this is useful for testing if a machine needs a reboot after patching. A machine or group of machines need rebooting if it shows ‘ok=2’:

ansible-playbook --check -i hosts.ini -l xyze apt-reboot.yml

To perform the actual reboots the playbook is run again without the ‘–check’.

I wanted to start an instance from my terminal rather than going onto the AWS console. Firstly choose the profile, then search for the instance:

export AWS_PROFILE=xyze
aws ec2 describe-instances --output table | grep -B150 backup | grep InstanceId

–output table presents the output as a nice table rather than json. Grep searches for the name I gave the instance, and B150 prints the lines before we get to the name. I’m grepping this to find the InstanceId so I can launch the machine.

Its a good idea to check you’ve got the right one:

aws ec2 describe-instances --output table --instance-ids i-044c47167ba728a23

And then you can start it:

aws ec2 start-instances --instance-ids i-044c47167ba728a23

Amazon’s guide to AWS Backup is here: https://docs.aws.amazon.com/aws-backup/latest/devguide/whatisbackup.html

Our Backup plan is called xyze-prod-backup and can be viewed by clicking on ‘Manage Backup plans’ from the AWS Backup dashboard.

I’ve set up a backup rule called ‘DailyBackups’ which are kept for 7 days in EBS as most recent backups are likely to be the ones that may be required to restore a server. I’ve written a boto3 script using their API to copy weekly backups to their ‘Deep Archive’ tape backups. Keeping the weekly backups on the Deep Archive for 12 months will save about $140 per month from our AWS bill and can be restored in less than 12 hours if needed. The bucket lifecycle rules are set to: keep for a year and then delete, and the backups will transition from standard to deep storage right away.

They’re based on snapshots which are also stored as AMI’s which allow for easy restores instead of messing with snapshots and volumes like before.

What to backup (aka Resource assignments) hasn’t changed and uses the ‘Daily’ tags. To add an instance to the backup plan all we do is tag an instance as ‘Daily’ under the Backup tag.

My boto3 script requires the latest python so the easiest thing to do is to run it in a virtual environment which can be set up as shown. In the end I had added it to Lambda running once a week which meant we didn’t have to run an additional backup server like before. Unfortunately Lambda only lets your function operate for 15 mins at a time which wasn’t enough for the whole script to run as I’d introduced a wait between each one as there was a maximum gigabytes of concurrent copying that you could use at a time. I may modify the script but for now I’ve created a new instance which, like the previous one, is called backup.cloud.xyze. The script runs once a week as a cronjob on the ubuntu user. Apparently lifecycle transitions are queued before midnight UTC so 10pm was chosen so standard storage will cost very little. This failed once so the time was put back to 7pm. I’ve set it to power down after the backup completes and sends an email. The minutes sleep in the crontab is because Postfix needs some time to send us the email. I’ve created a small Lambda function called boot-backup-instance and created a cronjob in EventBridge which starts backup.cloud.xyze at 6.45pm every Friday which should be enough time for it to initialise before the backup starts using the cronjob below:

# m h  dom mon dow   command
0 19 * * 5 cd /home/ubuntu/newscripts && source env/bin/activate && python3 ./copy-ami.py 2>&1 | /usr/bin/mailx -s "Weekly backup to glacier. Please keep in public-support-internal" support@xyze.co.uk && /usr/bin/sleep 60 && /usr/bin/sudo /usr/sbin/poweroff

At first the script failed as ‘source’ is only in bash not dash so I had to reconfigure using ‘sudo dpkg-reconfigure dash’ (which is preferable to alternatives according to the internet) and then it failed again as the backup user needed the additional IAM permissions detailed here:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebsapi-permissions.html
I was using my permissions originally and Lambda was internal so that’s why its different. The backup user has EC2 & S3 full access too.

sudo apt-get install python3-venv
python3 -m venv env
source env/bin/activate
pip3 install boto3
export AWS_PROFILE=xyze

To run the copy-ami.py script below we need to export our AWS credentials.

#!/usr/bin/env python
#James Holland September 2021

import boto3
import datetime
import json

#Uncomment for Lambda
#def lambda_handler(event, context):

# A counter was added for readability when cron sends email
count = 0
s3 = boto3.client('s3')
client = boto3.client('ec2')

#Get yesterdays date as today's backups might not have been done yet
date_filter = (datetime.datetime.now() - datetime.timedelta(days=1))
#This is only needed if archiving legacy backups as was done initially and is always one less day than the above date
date_filter_minus = (datetime.datetime.now() - datetime.timedelta(days=0))

#Documented here: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#EC2.Client.describe_images
response = client.describe_images(
Filters=[
       {
           'Name': 'tag:Backup',
           'Values': [
               'Daily',
           ]
       },
   ],
)

for ami in response['Images']:

   ami_creation_date_str = ami['CreationDate']
   ami_creation_date = datetime.datetime.strptime(ami_creation_date_str, "%Y-%m-%dT%H:%M:%S.%fZ")
   ami_image_id = eval(json.dumps(ami['ImageId']))
   ami_image_name = ami['Name']
   ami_image_id_bin = eval(json.dumps(ami['ImageId'])) + ".bin"
   name = [tag['Value'] for tag in ami['Tags'] if tag['Key'] == 'Name'][0]
   if datetime.datetime.timestamp(ami_creation_date) > datetime.datetime.timestamp(date_filter) and datetime.datetime.timestamp(ami_creation_date) < datetime.datetime.timestamp(date_filter_minus):
#Print the result to standard output to maybe send by email - but not implemented yet
       count = count + 1
       print(count, ami_image_id_bin, ami_creation_date, name)

#Documented here: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#EC2.Client.create_store_image_task
       copyami = client.create_store_image_task(
       ImageId=ami_image_id,
       Bucket='xyze-backups',
       S3ObjectTags=[
       {
               'Key': 'Name',
               'Value': name
           },
           {
               'Key': 'Backup-Date',
               'Value': ami_creation_date_str
           },
       ],
       DryRun=False
       )

#A waiter was added because doing them all at once exceeded the limit imposed by Amazon
#Now the bucket is polled every 30 seconds to see if the backup file is there before doing the next one
#I've included an hour's worth of checking because sometimes the copying pauses for minutes at a time
#Documented here: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html?highlight=waiter#S3.Waiter.ObjectExists
       waiter = s3.get_waiter('object_exists')
       waiter.wait(
       Bucket='xyze-backups',
       WaiterConfig={
          'Delay': 30,
          'MaxAttempts': 120
       },
       Key=ami_image_id_bin
       )

Restore from backup

Hopefully the backup you need will be from the last 7 days. All you need to do here is to go to the AMI images on the EC2 dashboard on the AWS console, sort for the date you want, choose the image and launch it as a new instance. You can check what the instance family is from the current instance.

Restoring from cold storage is a bit more involved particularly as this method is new and Amazon hasn’t developed it much yet. Lets say you want to restore a previous copy of our backup box. This xargs cludge will search through all the backups and grep the results to your terminal as shown:

aws s3 ls s3://xyze-backups | awk '{print $4}' | xargs -n1 --verbose -I {} aws s3api get-object-tagging --bucket xyze-backups --key {} | grep -B5 backup.cloud.xyze

aws s3api get-object-tagging --bucket xyze-backups --key ami-00323c135ac547f5b.bin
aws s3api get-object-tagging --bucket xyze-backups --key ami-006955f1847ecac96.bin
aws s3api get-object-tagging --bucket xyze-backups --key ami-018110dd5440eeb60.bin
aws s3api get-object-tagging --bucket xyze-backups --key ami-01c721b7baa2f09d9.bin
aws s3api get-object-tagging --bucket xyze-backups --key ami-037206e2c482b8c07.bin
aws s3api get-object-tagging --bucket xyze-backups --key ami-0423091a04eed102e.bin
aws s3api get-object-tagging --bucket xyze-backups --key ami-054d2d2434c6acd87.bin
           "Key": "Backup-Date",
           "Value": "2021-09-01T07:07:54.000Z"
       },
       {
           "Key": "Name",
           "Value": "backup.cloud.xyze"
aws s3api get-object-tagging --bucket xyze-backups --key ami-05ce0d7981dfa3afb.bin
aws s3api get-object-tagging --bucket xyze-backups --key ami-0615d33a9898f5a41.bin
aws s3api get-object-tagging --bucket xyze-backups --key ami-06eed532a0d19e117.bin
aws s3api get-object-tagging --bucket xyze-backups --key ami-07f0a939f8d408d3f.bin

Choose the date you’re after and the ami key is shown above the grepped date. In our example we then use the aws cli to restore the ami image. Again we’re utilising the s3api. I’ve set the days the restored object will expire to 3 days when I am next on shift. A restored object is charged at the standard rate so don’t set it any more than necessary. If you click on the object in the s3 dashboard you will see a ‘Restoration in progress’ dialog. The restoration is normally complete within 12 hours but in practice can be shorter. You can check on its progress using ‘s3api head-object’ as shown.

When its complete you can run the ‘create-restore-image-task’ and the object will soon appear in the AMI images on the EC2 dashboard and as before you can now launch a new instance from the backup.

aws s3api restore-object --bucket xyze-backups --key ami-054d2d2434c6acd87.bin --restore-request '{"Days":3,"GlacierJobParameters":{"Tier":"Standard"}}'

aws s3api head-object --bucket xyze-backups --key ami-054d2d2434c6acd87.bin

aws ec2 create-restore-image-task --bucket xyze-backups --name ami-backup.cloud.xyze --object-key ami-054d2d2434c6acd87.bin

I use Amazon’s Daily Backup service which creates Amazon Machine Images from snapshots which I keep for a year in S3 Deep Archive and then automatically delete using a Lifecycle Rule (Daily backups.) However I’ve upgraded the Git server and I want to keep a copy of the old one in a bucket of old instances. I want to try using the cli again rather than the AWS console. Firstly choose the profile, then search for the ami:

export AWS_PROFILE=xyze
aws ec2 describe-images --owners 000000000000 --output table | grep -B50 git | grep ImageId

This uses an option ‘owners’ which is your AWS account. Help with the command can be viewed using the help:

aws ec2 describe-images help

As previously (AWS CLI start instance) its useful to check we’ve got the right one:

aws ec2 describe-images --output table --image-ids ami-0905c88505b225019

This command stores the image in s3:

aws ec2 create-store-image-task --s3-object-tags Key=Name,Value=old-git --image-id ami-0905c88505b225019 --bucket old-instances

And this renames the file and moves it into Glacier Deep Archive:

aws s3 mv s3://old-instances/ami-0905c88505b225019.bin s3://old-instances/old-git.bin --storage-class DEEP_ARCHIVE

I’m using pip to install the latest awscli as the version in apt won’t work for the new ‘Deep Archive’ however awscli v2 is now available and should be used instead of the old one: https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html

pip3 install awscli

Now we need to configure the cli by installing authentication from AWS. Log into the AWS account and go to IAM. Click on Users, then your name, and then Create access key as shown below. Save the file as you won’t be able to get it again and will have to create a new one.

Now add the keys using the default region: eu-west-1

aws configure --profile xyze

To use the xyze profile for the whole of your current terminal session:

export AWS_PROFILE=xyze

Over the weekend of 8th Feb 2020 I upgraded our AWS production instances to the new nitro series 3 platform plumping for the AMD based T3a family as this attracted a 10 percent discount and we are looking to reduce our AWS costs as they are increasing month by month.

Whereas our customers upgrades (using Ubuntu 14.04 like ours) went smoothly, ours of course didn’t. Turned out to be a bug in 14.04 which didn’t add nvme drivers to initrd. In nitro instances the storage is presented as nvme despite using magnetic disks or ssd so you need an nvme driver.

So to prepare an instance for upgrade to nitro we need to install drivers for the nitro network card and nvme interface. This is done using the guide here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking-ena.html but I’ll pick out the relevant bits here.

Fortunately the nitro drivers and nvme drivers are included in all recent kernels so all we have to do is install them:

sudo apt install linux-aws

Amazon provides a script to test whether the drivers have been loaded here: https://aws.amazon.com/premiumsupport/knowledge-center/boot-error-linux-m5-c5/

Unfortunately as mentioned earlier Ubuntu 14.04 fails to put the nvme driver into initrd so we have to explicitly tell it to:

sudo su
echo nvme >> /etc/initramfs-tools/modules
update-initramfs -c -k all

On a new nitro install Amazon also puts nvme_core.io_timeout=4294967295 into grub so I suggest we do the same:

sudo nano /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0 nvme_core.io_timeout=4294967295 zswap.enabled=1 zswap.compressor=lz4"
sudo update-grub
(RHEL/CENTOS: sudo grub2-mkconfig -o /boot/grub2/grub.cfg)

Ubuntu lets anything in /etc/default/grub.d/ override the default so you might have to add the entries to a file similar to /etc/default/grub.d/50-cloudimg-settings.cfg too, remembering to sudo update-grub afterwards.

If there are things already in there like console commands then leave them there. Since zswap is running in every new kernel I’ve also added entries to turn it on. Whilst swap isn’t really needed in powerful instances, money can still be saved by using a smaller instance and swap. Zswap compresses pages in memory instead of paging to disk but still needs a disk based backup swap file to work. Zswap uses a fifth of the RAM so the swap file should be a fifth of the RAM too – or thereabouts. Its often easier to do a quarter of the RAM.

sudo su
echo lz4 >> /etc/initramfs-tools/modules
echo lz4_compress >> /etc/initramfs-tools/modules
update-initramfs -c -k all
fallocate -l 1G /swapfile
chmod 600 /swapfile
mkswap /swapfile

And add the swap file to fstab:

sudo nano /etc/fstab
/swapfile swap swap defaults 0 0

After rebooting you can see if zswap is enabled:

grep -R . /sys/module/zswap/parameters
/sys/module/zswap/parameters/zpool:zbud
/sys/module/zswap/parameters/max_pool_percent:20
/sys/module/zswap/parameters/enabled:Y
/sys/module/zswap/parameters/compressor:lz4

To enable ENA on AWS the user guide says we need to shutdown the instance and use the AWS CLI.

export AWS_PROFILE=xyze
aws ec2 modify-instance-attribute --instance-id i-xxxxxxxx --ena-support

Now the actual upgrade is easy – but I’d still take a snapshot to be on the safe side. To upgrade we close the instance down, change the instance type to the same as it was except in the t3a series – so t2.micro becomes t3a.micro and then reboot. New nitro installs turn on the T2/T3 Unlimited by default so you may have to enable this on the AWS Console too. Then turn it back on – booting seems much quicker on nitro.

Postscript

On AWS & Google cloud LZ4 doesn’t load at boot time so the following trick in a crontab will load it:

root@instance-1:~# crontab -e
.
.
.
# m h  dom mon dow   command
@reboot echo lz4 > /sys/module/zswap/parameters/compressor

Using zstd in place of lz4 gives better compression although decompression is two thirds slower. In theory better compression would take up less RAM leaving more to play with.