Windows 11 install kept failing at 77% despite Secure Boot and UEFI being enabled. I decided to try and install Windows 10 again but this also failed. Searching Google brought up a Reddit post which said that Windows installations couldn’t cope with another hard drive being present as when doing its many reboots it wouldn’t be able to tell which hard disk to boot from! It recommended disabling the SATA interface in the BIOS. I wanted to install it to the SATA drive, so disabled the NVMe interface instead. This time Windows 11 installed without hiccup despite taking 3 hours to install and do the updates, this compares with about 20 minutes for an Ubuntu install (including updates) on the same hardware. Why it needs to reboot several times is a mystery particularly when Ubuntu doesn’t need to reboot at all apart from into the newly installed system at the end of its installation.

I used a usb stick on which I’d installed an open source loader called Ventoy. It had its own UEFI key which needed to be installed into the BIOS where I discovered that the Windows license key was installed. Ventoy seems to have binary blobs in its source code tree on github but everyone seems to be using it, and it works very well.

Windows installed a new copy of the EFI partition so I had to copy its contents onto my original EFI partition on the NVMe drive and on rebooting it, loaded the normal Grub menu again. Unfortunately my wifi was no longer working in Ubuntu and another Google search revealed that the Windows 11 quick boot enabled the suspension of the wifi which is why Ubuntu no longer detected it. Disabling the quick boot in the power settings of the control panel in Windows 11 fixed it.

Another Google search revealed that the local account settings have been well hidden and Microsoft wants you to link it to an online login, but choosing to set it up for a college or workplace and choosing to add the install to a domain allows you to create a local account.

sudo find / -not -path "/home/*" -size +10M

Find files containing text in a particular path:

grep -rnw '/path/to/somewhere/' -e 'pattern'
sudo grep -rnw /etc -e james

The challenge was to get it shoehorned into one of Google Cloud Engine’s e2-micros. Most of the site was static so could be served from Cloudflare, but there were too many images so I decided to offload them onto an AWS bucket and mount using s3fs fuse. Unfortunately s3fs is very slow and searching for alternatives I happened upon JuiceFS.

In common with a lot of open source projects they offer a community edition which can be used for free and is extensively documented here: https://juicefs.com/docs/community/introduction/

Installation is detailed here: https://juicefs.com/docs/community/getting-started/installation and I installed via their PPA:

sudo apt install software-properties-common
sudo add-apt-repository ppa:juicefs/ppa
sudo apt install juicefs
sudo ln -s /usr/bin/juicefs /usr/local/bin/

I then formatted my aws s3 storage using: https://juicefs.com/docs/community/getting-started/standalone#hands-on-practice-2

# Replace relevant options with the actual object storage being used
juicefs format --storage s3 \
    --bucket https://myjfs.s3.us-west-1.amazonaws.com \
    --access-key ABCDEFGHIJKLMNopqXYZ \
    --secret-key ZYXwvutsrqpoNMLkJiHgfeDCBA \
    sqlite3://myjfs.db myjfs

Unfortunately sqlite kept crashing so I installed redis for the metadata storage. Fortunately you can reconfigure your metadata storage without having to reformat s3: https://juicefs.com/docs/community/metadata_dump_load#recovery-and-migration

apt install redis-server
juicefs dump sqlite3://myjfs.db meta-dump
juicefs load redis://127.0.0.1:6379 meta-dump
juicefs config --storage s3 --bucket https://myjfs.s3.us-west-1.amazonaws.com --access-key ABCDEFGHIJKLMNopqXYZ --secret-key ZYXwvutsrqpoNMLkJiHgfeDCBA redis://127.0.0.1:6379

It can generate an fstab entry which is nice:

juicefs mount --update-fstab --cache-size 2048 --cache-partial-only redis://127.0.0.1:6379 /var/www/html/wordpress/wp-content/uploads/

As its a low traffic site I can do it on one of Google’s smallest servers, again using an e2-micro configured with Ubuntu 24.04 Minimal.

Caddy installation is detailed here: https://caddyserver.com/docs/install#debian-ubuntu-raspbian

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install caddy

I then installed the stuff that I need that’s missing from Ubuntu Minimal:

sudo apt install bash-completion nano htop cron

I always change the history size in the shell so it keeps more, changing HISTSIZE=100000
HISTFILESIZE=20000:

nano .bashrc

With only 1GB ram, I always install zswap:

sudo nano /etc/default/grub.d/50-cloudimg-settings.cfg
GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS0,115200 zswap.enabled=1 zswap.shrinker_enabled=1 zswap.compressor=zstd zswap.zpool=zsmalloc"
sudo update-grub

sudo fallocate -l 1G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo nano /etc/fstab
/swapfile swap swap defaults 0 0

sudo crontab -e
@reboot echo zstd > /sys/module/zswap/parameters/compressor

If using btrfs then this has to be used as it needs to be created without copy-on-write:
sudo btrfs filesystem mkswapfile –size 1G /swapfile

Use swapon or reboot to activate. For some reason the zswap compressor isn’t enabled at boot time, so the kludge in a crontab activates it.
It can be checked with:

grep -R . /sys/module/zswap/parameters

I normally configure swappiness which prevents kswapd kicking in and slowing everything down:

sudo nano /etc/sysctl.conf
vm.swappiness = 1

I got the latest WordPress and installed it in the usual place:

wget https://wordpress.org/latest.tar.gz
tar zxvf latest.tar.gz
sudo mkdir -p /var/www/html
sudo mv wordpress/ /var/www/html/
sudo chown -R www-data: /var/www/html/wordpress/

I then installed mysql and php:

sudo apt install mariadb-server php-apcu php-fpm php-mysql php-curl php-xml php-imagick php-mbstring php-zip php-gd php-intl

I then configured mysql:

sudo mysql_secure_installation
sudo mysql

CREATE USER "USER"@"localhost" IDENTIFIED BY "PASSWORD";
CREATE DATABASE xyze;
GRANT ALL PRIVILEGES ON xyze.* TO "USER"@"localhost";
FLUSH PRIVILEGES;

This says that character set and collation are defaults so no need to specify explicitly: https://dev.mysql.com/doc/refman/8.0/en/charset-server.html

Caddy comes with a sample config: /etc/caddy/Caddyfile which will display a sample webpage when tested in the browser. Searching Google for a config brought up: https://caddy.community/t/setting-up-wordpress-with-caddy-on-ubuntu/18448 which I’ve altered slightly:

xyze.co.uk {
	redir https://www.xyze.co.uk
}

www.xyze.co.uk {
    # good practice to signal on behalf of who 
    # are the certs getting issue
	tls james@xyze.co.uk

    # logs are optional
	log {
		output file /var/log/caddy/xyze.co.uk
		format console
	}

	root * /var/www/html/wordpress
	encode zstd gzip
	file_server
	php_fastcgi unix//run/php/php-fpm.sock

	@disallowed {
		path /xmlrpc.php
		path *.sql
		path /wp-content/uploads/*.php
	}

	rewrite @disallowed '/index.php'
}

Letsencrypt is configured by Caddy automatically so no Certbot configuration is necessary. I then set up WordPress via the browser using the mysql config from earlier.

I then dumped the previous db and loaded it into the new one:

mysqldump -uUSER -pPASSWORD xyze > /var/www/html/wordpress/sql/wp-backup.sql
mysql -uUSER -pPASSWORD xyze < /var/www/html/wordpress/sql/wp-backup.sql

Logging into WordPress needed a db update and from there I configured the caching.
I’ve found that apcu is faster than Redis so have installed this one which needs the php-apcu I installed earlier:
https://wordpress.org/plugins/atec-cache-apcu/ This plugin gives opcache recommended settings: https://wordpress.org/plugins/atec-cache-info/ which are configured in:

sudo nano /etc/php/8.3/fpm/php.ini
sudo systemctl restart php8.3-fpm caddy

Unfortunately their page cache seems buggy and they’ve superseded it in one of their other plugins, so I’m using https://wordpress.org/plugins/powered-cache/ which I’ve used successfully before. The other plugin needed is the Cloudflare one which will update Cloudflare when a new post is made: https://wordpress.org/plugins/cloudflare/

I like to backup the db every night to make it ready for the external rsync backup:

sudo crontab -l

# m h  dom mon dow   command
@reboot echo zstd > /sys/module/zswap/parameters/compressor
35 0 * * * /usr/bin/mysqldump -uUSER -pPASSWORD xyze > /var/www/html/wordpress/sql/wp-backup.sql

sudo apt install python3-pip python3-venv
python3 -m venv env
source env/bin/activate
pip3 install ansible

The first thing I created was a hosts.ini file:

cat hosts.ini

[xyze]
backup.xyze       ansible_ssh_extra_args="-Jxyze@bastion.xyze"
git.xyze          ansible_ssh_extra_args="-Jxyze@bastion.xyze"
icinga.xyze       ansible_ssh_extra_args="-Jxyze@bastion.xyze"
tickets.xyze      ansible_ssh_extra_args="-Jxyze@bastion.xyze"
bastion.xyze

[client1]
website.client1

[client2]
website.client2
.
.
.

Our DNS is on AWS’s Route 53 and I’ve created private zones for each of our clients and us, so when we’re logged into our VPN the addresses above can be resolved. Our machines can only be accessed via an intermediate bastion host so the jump host is included in the hosts file.

Then I created some apt playbooks:

cat apt-safe-upgrade.yml 
---
- hosts: xyze, client1, client2
  remote_user: xyze
  become: true
  become_method: sudo
  gather_facts: false

  tasks:
    - name: apt-get dist-upgrade
      apt:
        update_cache: true
        upgrade: safe

cat apt-reboot.yml 
---
- hosts: xyze, client1, client2
  remote_user: xyze
  become: yes
  become_method: sudo
  gather_facts: no

  tasks:
  - name: Check if a reboot is required
    register: reboot_required_file
    stat: path=/var/run/reboot-required get_md5=no

  - name: Reboot box if kernel/libs updated and requested by the system
    shell: sleep 10 && /sbin/shutdown -r now 'Rebooting box to update system libs/kernel as needed'
    args:
        removes: /var/run/reboot-required
    async: 300
    poll: 0
    ignore_errors: true
    when: reboot_required_file.stat.exists == true

cat apt-autoremove.yml 
---
- hosts: xyze, client1, client2
  remote_user: xyze
  become: true
  become_method: sudo
  gather_facts: false

  tasks:
    - name: apt autoremove
      apt:
        autoremove: true

Unfortunately some of my playbooks had errors which were found by the Ansible Lint tool. This can be installed via pip and installs yamllint which can be run first:

pip3 install ansible-lint

Playbooks are executed in the following manner:

ansible-playbook -i hosts.ini -l xyze,client1,client2 apt-safe-upgrade.yml -f10

Running ‘ansible-playbook’ on its own will give a list of all the options or passing ‘-h’ or’–help’ to it will display the same thing.

In our example above we are using hosts.ini as the inventory, and limiting the hosts it runs on to a subset. This can be a single host within the hosts.ini file or a group of hosts like xyze,client1 or it’ll run on all the hosts if you pass the limiting option ‘all’. Normally it’ll fork into 5 parallel processes but I’ve doubled that to 10 in this example.

A check or test run can be performed and this is useful for testing if a machine needs a reboot after patching. A machine or group of machines need rebooting if it shows ‘ok=2’:

ansible-playbook --check -i hosts.ini -l xyze apt-reboot.yml

To perform the actual reboots the playbook is run again without the ‘–check’.

I’ve been meaning to try Lighttpd for ages as I wanted to downgrade my server to save money and it takes the least amount of memory compared with apache or nginx.

Installed with the help of:
https://www.how2shout.com/linux/install-wordpress-on-lighttpd-web-server-ubuntu/
https://www.howtoforge.com/how-to-install-lighttpd-with-php-and-mariadb-on-debian-10/
But had to make changes too.

james@instance-1:~$ sudo mysql
CREATE DATABASE mydatabase CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
create user 'demoh2s'@'localhost' identified by 'password';
GRANT ALL PRIVILEGES ON `mydatabase`.* to `demoh2s`@localhost;
exit

james@instance-1:~$ sudo apt install php php-cgi php-cli php-fpm php-curl php-gd php-mysql php-mbstring zip unzip php-zip php-xml php-intl php-imagick apache2-

I’m installing on Ubuntu 22.04 Minimal https://wiki.ubuntu.com/Minimal on Google’s cloud platform. I try to keep original config files intact so have only altered the server.document-root and rewrite rules in lighttpd.conf

There is a tool similar to the one in apache to install the symlinks in /etc/lighttpd/conf-enabled

james@instance-1:~$ sudo lighty-enable-mod fastcgi fastcgi-php auth deflate rewrite accesslog

Rewriting

At first all the permalinks were showing ‘index.php’ which apache wasn’t doing before so I gathered together some rewriting rules from various places via Google. Through trial and error I ascertained that the order is significant, and I’ve included the rules I found but commented out. These still aren’t definitive but seem to be a work in progress and WordPress only seems to need the 4 uncommented ones at present. You can Google them to see where they came from:

lighttpd "^/(wp-admin.+).*/?" => "$0"

The Lighttpd wiki is very useful too: https://redmine.lighttpd.net/projects/lighttpd/wiki/Docs

james@instance-1:/etc/lighttpd$ cat lighttpd.conf
server.modules = (
	"mod_indexfile",
	"mod_access",
	"mod_alias",
 	"mod_redirect",
)

#url.rewrite = (
#"^/?$" => "/index.php",
# Exclude common directories
#"^/(wp-.+)$" => "$0",
# Exclude letsencrypt certbot
#"^/(.well-.+)$" => "$0",
#"^/xmlrpc.php" => "$0",
#"^/sitemap.xml" => "$0",
# Handle permalinks and feeds
#"^/(.+)/?$" => "/index.php/$1"
#)

url.rewrite-if-not-file = (
 "^/(wp-admin.+).*/?" => "$0",
# Exclude letsencrypt certbot
 "^/(.well-.+)$" => "$0",
# REST API for block editor & Jetpack 
 "(?:\?(.*))?$" => "/index.php?$1",
# "(\/styles\/|\/images\/|\/scripts\/)" => "$0",
 "^/(wp-.+).*/?" => "$0",
# "^/images/.*/?" => "$0",
# "^/temp/.*/?" => "$0",
# "^/keyword/([A-Za-z_0-9\-]+)/?$" => "/index.php?keyword=$1",
# "^/.*?(\?.*)?$" => "/index.php$1"
)

server.document-root        = "/var/www/html/xyze.co.uk"
.
.
.

You need to set the permalinks in wp-admin as well under Settings, Permalinks:

Custom Structure https://www.xyze.co.uk/index.php /%postname%/

WordPress (this is not needed if using Let’s Encrypt)

The way I’ve installed WordPress before is to just use http and then use the ‘Flexible’ shared Cloudflare certificate that encrypts traffic from the user to Cloudflare but then its served to Cloudflare using http. This has worked fine with apache but both the classic editor and block editor weren’t working in Lighttpd and searching with the error from the block editor I stumbled across this:
https://wordpress.org/support/topic/publishing-failed-you-are-probably-offline/

james@instance-1:~$ cat /var/www/html/xyze.co.uk/wp-config.php

/* Add any custom values between this line and the "stop editing" line. */

define( 'WP_SITEURL', 'http://xyze.co.uk' );
define( 'WP_HOME', 'http://xyze.co.uk' );

/* That's all, stop editing! Happy publishing. */

This allowed me to use http if I wanted to add or edit a post and Cloudflare’s Flexible https worked too.

Let’s Encrypt

However now we have Let’s Encrypt with certbot, its better to have proper https.

james@instance-1:~$ sudo apt install certbot
james@instance-1:~$ sudo certbot certonly --webroot -w /var/www/html/xyze.co.uk/ -d www.xyze.co.uk

https://eff-certbot.readthedocs.io/en/stable/using.html
Says that “For historical reasons, the containing directories are created with permissions of 0700 meaning that certificates are accessible only to servers that run as the root user. If you will never downgrade to an older version of Certbot, then you can safely fix this using chmod 0755 /etc/letsencrypt/{live,archive}”
Which I did.

This time I copied /etc/lighttpd/conf-available/10-ssl.conf to 10-ssl-xyze.conf and then created the symlink manually.

james@instance-1:/etc/lighttpd/conf-enabled$ sudo ln -s ../conf-available/10-ssl-xyze.conf
james@instance-1:/etc/lighttpd/conf-enabled$ cat 10-ssl-xyze.conf
# /usr/share/doc/lighttpd/ssl.txt
# -*- conflicts: mbedtls, gnutls, nss, wolfssl -*-

server.modules += ( "mod_openssl" )

# ssl.* in global scope gets inherited by
#   $SERVER["socket"] == "..." { ssl.engine = "enable" }
#ssl.pemfile = "/etc/lighttpd/server.pem"
ssl.pemfile = "/etc/letsencrypt/live/www.xyze.co.uk/fullchain.pem" # Combined Certificate
ssl.privkey = "/etc/letsencrypt/live/www.xyze.co.uk/privkey.pem"

ssl.cipher-list = "HIGH"

$SERVER["socket"] == ":443" {
ssl.engine = "enable"
}

$HTTP["scheme"] == "http" {
$HTTP["host"] == "www.xyze.co.uk" { # HTTP URL
url.redirect = ("/.*" => "https://www.xyze.co.uk$0") # Redirection HTTPS URL
}
}

include_shell "/usr/share/lighttpd/use-ipv6.pl 443"

At first I copied the example from the howtoforge link above but lighttpd kept failing to restart producing an error like this: ssl.pemfile has to be set in same $SERVER[“socket”] scope as other ssl.* directives

Eventually I worked out that the pem files had to be in the global file or in all of the socket, scheme, and host directives, so I moved it out of the socket directive so they’re now global and inherited by all of them.

Finally I changed the wp-config.php to https – or you could comment them out as long as they’re set to https in Settings, General on the WordPress Dashboard. And Cloudflare can now use ‘Full’ encryption and ‘Always Use HTTPS’.

Cloudflare

I added some Firewall rules using the dash console:

WordPress Spam Filter
(http.request.uri.path contains "wp-comments-post.php") or (http.request.uri.path contains "wp-login.php")
Managed Challenge

Protect the wp-admin Area
(http.request.uri.path contains "/wp-admin/" and not http.request.uri.path contains "/wp-admin/admin-ajax.php" and not http.request.uri.path contains "/wp-admin/theme-editor.php")
Managed Challenge

Block xmlrpc.php Attacks
(http.request.uri.path contains "/xmlrpc.php")
Block

Block wp-login
(http.request.uri.path contains "wp-login.php" and ip.geoip.country ne "GB")
Block

Postscript

I followed a guide to install Redis here except I installed the php client as it seemed to work faster. This uses a WordPress plugin from here. Php will need reloading but I rebooted as I’d also installed some kernel updates.

james@instance-1:~$ sudo apt install php-redis

I also installed a Cloudflare cache plugin from here which needs your api key from Cloudflare and works really well. There is also the W3 Total Cache in place of the redis cache plugin.

I wanted to start an instance from my terminal rather than going onto the AWS console. Firstly choose the profile, then search for the instance:

export AWS_PROFILE=xyze
aws ec2 describe-instances --output table | grep -B150 backup | grep InstanceId

–output table presents the output as a nice table rather than json. Grep searches for the name I gave the instance, and B150 prints the lines before we get to the name. I’m grepping this to find the InstanceId so I can launch the machine.

Its a good idea to check you’ve got the right one:

aws ec2 describe-instances --output table --instance-ids i-044c47167ba728a23

And then you can start it:

aws ec2 start-instances --instance-ids i-044c47167ba728a23

Amazon’s guide to AWS Backup is here: https://docs.aws.amazon.com/aws-backup/latest/devguide/whatisbackup.html

Our Backup plan is called xyze-prod-backup and can be viewed by clicking on ‘Manage Backup plans’ from the AWS Backup dashboard.

I’ve set up a backup rule called ‘DailyBackups’ which are kept for 7 days in EBS as most recent backups are likely to be the ones that may be required to restore a server. I’ve written a boto3 script using their API to copy weekly backups to their ‘Deep Archive’ tape backups. Keeping the weekly backups on the Deep Archive for 12 months will save about $140 per month from our AWS bill and can be restored in less than 12 hours if needed. The bucket lifecycle rules are set to: keep for a year and then delete, and the backups will transition from standard to deep storage right away.

They’re based on snapshots which are also stored as AMI’s which allow for easy restores instead of messing with snapshots and volumes like before.

What to backup (aka Resource assignments) hasn’t changed and uses the ‘Daily’ tags. To add an instance to the backup plan all we do is tag an instance as ‘Daily’ under the Backup tag.

My boto3 script requires the latest python so the easiest thing to do is to run it in a virtual environment which can be set up as shown. In the end I had added it to Lambda running once a week which meant we didn’t have to run an additional backup server like before. Unfortunately Lambda only lets your function operate for 15 mins at a time which wasn’t enough for the whole script to run as I’d introduced a wait between each one as there was a maximum gigabytes of concurrent copying that you could use at a time. I may modify the script but for now I’ve created a new instance which, like the previous one, is called backup.cloud.xyze. The script runs once a week as a cronjob on the ubuntu user. Apparently lifecycle transitions are queued before midnight UTC so 10pm was chosen so standard storage will cost very little. This failed once so the time was put back to 7pm. I’ve set it to power down after the backup completes and sends an email. The minutes sleep in the crontab is because Postfix needs some time to send us the email. I’ve created a small Lambda function called boot-backup-instance and created a cronjob in EventBridge which starts backup.cloud.xyze at 6.45pm every Friday which should be enough time for it to initialise before the backup starts using the cronjob below:

# m h  dom mon dow   command
0 19 * * 5 cd /home/ubuntu/newscripts && source env/bin/activate && python3 ./copy-ami.py 2>&1 | /usr/bin/mailx -s "Weekly backup to glacier. Please keep in public-support-internal" support@xyze.co.uk && /usr/bin/sleep 60 && /usr/bin/sudo /usr/sbin/poweroff

At first the script failed as ‘source’ is only in bash not dash so I had to reconfigure using ‘sudo dpkg-reconfigure dash’ (which is preferable to alternatives according to the internet) and then it failed again as the backup user needed the additional IAM permissions detailed here:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebsapi-permissions.html
I was using my permissions originally and Lambda was internal so that’s why its different. The backup user has EC2 & S3 full access too.

sudo apt-get install python3-venv
python3 -m venv env
source env/bin/activate
pip3 install boto3
export AWS_PROFILE=xyze

To run the copy-ami.py script below we need to export our AWS credentials.

#!/usr/bin/env python
#James Holland September 2021

import boto3
import datetime
import json

#Uncomment for Lambda
#def lambda_handler(event, context):

# A counter was added for readability when cron sends email
count = 0
s3 = boto3.client('s3')
client = boto3.client('ec2')

#Get yesterdays date as today's backups might not have been done yet
date_filter = (datetime.datetime.now() - datetime.timedelta(days=1))
#This is only needed if archiving legacy backups as was done initially and is always one less day than the above date
date_filter_minus = (datetime.datetime.now() - datetime.timedelta(days=0))

#Documented here: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#EC2.Client.describe_images
response = client.describe_images(
Filters=[
       {
           'Name': 'tag:Backup',
           'Values': [
               'Daily',
           ]
       },
   ],
)

for ami in response['Images']:

   ami_creation_date_str = ami['CreationDate']
   ami_creation_date = datetime.datetime.strptime(ami_creation_date_str, "%Y-%m-%dT%H:%M:%S.%fZ")
   ami_image_id = eval(json.dumps(ami['ImageId']))
   ami_image_name = ami['Name']
   ami_image_id_bin = eval(json.dumps(ami['ImageId'])) + ".bin"
   name = [tag['Value'] for tag in ami['Tags'] if tag['Key'] == 'Name'][0]
   if datetime.datetime.timestamp(ami_creation_date) > datetime.datetime.timestamp(date_filter) and datetime.datetime.timestamp(ami_creation_date) < datetime.datetime.timestamp(date_filter_minus):
#Print the result to standard output to maybe send by email - but not implemented yet
       count = count + 1
       print(count, ami_image_id_bin, ami_creation_date, name)

#Documented here: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#EC2.Client.create_store_image_task
       copyami = client.create_store_image_task(
       ImageId=ami_image_id,
       Bucket='xyze-backups',
       S3ObjectTags=[
       {
               'Key': 'Name',
               'Value': name
           },
           {
               'Key': 'Backup-Date',
               'Value': ami_creation_date_str
           },
       ],
       DryRun=False
       )

#A waiter was added because doing them all at once exceeded the limit imposed by Amazon
#Now the bucket is polled every 30 seconds to see if the backup file is there before doing the next one
#I've included an hour's worth of checking because sometimes the copying pauses for minutes at a time
#Documented here: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html?highlight=waiter#S3.Waiter.ObjectExists
       waiter = s3.get_waiter('object_exists')
       waiter.wait(
       Bucket='xyze-backups',
       WaiterConfig={
          'Delay': 30,
          'MaxAttempts': 120
       },
       Key=ami_image_id_bin
       )

Restore from backup

Hopefully the backup you need will be from the last 7 days. All you need to do here is to go to the AMI images on the EC2 dashboard on the AWS console, sort for the date you want, choose the image and launch it as a new instance. You can check what the instance family is from the current instance.

Restoring from cold storage is a bit more involved particularly as this method is new and Amazon hasn’t developed it much yet. Lets say you want to restore a previous copy of our backup box. This xargs cludge will search through all the backups and grep the results to your terminal as shown:

aws s3 ls s3://xyze-backups | awk '{print $4}' | xargs -n1 --verbose -I {} aws s3api get-object-tagging --bucket xyze-backups --key {} | grep -B5 backup.cloud.xyze

aws s3api get-object-tagging --bucket xyze-backups --key ami-00323c135ac547f5b.bin
aws s3api get-object-tagging --bucket xyze-backups --key ami-006955f1847ecac96.bin
aws s3api get-object-tagging --bucket xyze-backups --key ami-018110dd5440eeb60.bin
aws s3api get-object-tagging --bucket xyze-backups --key ami-01c721b7baa2f09d9.bin
aws s3api get-object-tagging --bucket xyze-backups --key ami-037206e2c482b8c07.bin
aws s3api get-object-tagging --bucket xyze-backups --key ami-0423091a04eed102e.bin
aws s3api get-object-tagging --bucket xyze-backups --key ami-054d2d2434c6acd87.bin
           "Key": "Backup-Date",
           "Value": "2021-09-01T07:07:54.000Z"
       },
       {
           "Key": "Name",
           "Value": "backup.cloud.xyze"
aws s3api get-object-tagging --bucket xyze-backups --key ami-05ce0d7981dfa3afb.bin
aws s3api get-object-tagging --bucket xyze-backups --key ami-0615d33a9898f5a41.bin
aws s3api get-object-tagging --bucket xyze-backups --key ami-06eed532a0d19e117.bin
aws s3api get-object-tagging --bucket xyze-backups --key ami-07f0a939f8d408d3f.bin

Choose the date you’re after and the ami key is shown above the grepped date. In our example we then use the aws cli to restore the ami image. Again we’re utilising the s3api. I’ve set the days the restored object will expire to 3 days when I am next on shift. A restored object is charged at the standard rate so don’t set it any more than necessary. If you click on the object in the s3 dashboard you will see a ‘Restoration in progress’ dialog. The restoration is normally complete within 12 hours but in practice can be shorter. You can check on its progress using ‘s3api head-object’ as shown.

When its complete you can run the ‘create-restore-image-task’ and the object will soon appear in the AMI images on the EC2 dashboard and as before you can now launch a new instance from the backup.

aws s3api restore-object --bucket xyze-backups --key ami-054d2d2434c6acd87.bin --restore-request '{"Days":3,"GlacierJobParameters":{"Tier":"Standard"}}'

aws s3api head-object --bucket xyze-backups --key ami-054d2d2434c6acd87.bin

aws ec2 create-restore-image-task --bucket xyze-backups --name ami-backup.cloud.xyze --object-key ami-054d2d2434c6acd87.bin

I use Amazon’s Daily Backup service which creates Amazon Machine Images from snapshots which I keep for a year in S3 Deep Archive and then automatically delete using a Lifecycle Rule (Daily backups.) However I’ve upgraded the Git server and I want to keep a copy of the old one in a bucket of old instances. I want to try using the cli again rather than the AWS console. Firstly choose the profile, then search for the ami:

export AWS_PROFILE=xyze
aws ec2 describe-images --owners 000000000000 --output table | grep -B50 git | grep ImageId

This uses an option ‘owners’ which is your AWS account. Help with the command can be viewed using the help:

aws ec2 describe-images help

As previously (AWS CLI start instance) its useful to check we’ve got the right one:

aws ec2 describe-images --output table --image-ids ami-0905c88505b225019

This command stores the image in s3:

aws ec2 create-store-image-task --s3-object-tags Key=Name,Value=old-git --image-id ami-0905c88505b225019 --bucket old-instances

And this renames the file and moves it into Glacier Deep Archive:

aws s3 mv s3://old-instances/ami-0905c88505b225019.bin s3://old-instances/old-git.bin --storage-class DEEP_ARCHIVE

I’m using pip to install the latest awscli as the version in apt won’t work for the new ‘Deep Archive’ however awscli v2 is now available and should be used instead of the old one: https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html

pip3 install awscli

Now we need to configure the cli by installing authentication from AWS. Log into the AWS account and go to IAM. Click on Users, then your name, and then Create access key as shown below. Save the file as you won’t be able to get it again and will have to create a new one.

Now add the keys using the default region: eu-west-1

aws configure --profile xyze

To use the xyze profile for the whole of your current terminal session:

export AWS_PROFILE=xyze