Oliver Helm

Tag: scripting automation

Backing up to Amazon S3

There’s a great (and free) command line tool set called s3cmd that makes it simple to push and pull files to AWS S3 buckets, and is easy to script with.  We use it for cheep, reliable, offsite backups of media and database files.

The tool set can be downloaded form the GitHub repo here.  There’s a simple howto guide at the bottom.  One slight bump we did run into is that some of the older versions struggle with larger file sizes so make sure you get version 1.5.0-alpha3 at the minimum.

To install the tool simply download the repo onto your server / laptop and cd into the directory to run s3cmd --configure. You’ll need to have generated iAM credentials through the AWS control panel first.  Once you’ve got it configured you can push files to a bucket with the following command:
s3cmd put /local/file/path s3://bucket-name

Below is a super simple (and fairly crude) bash script we call by cron every night that backups all db’s on a server and sends the backups to s3:

echo "------- Starting " $(date) " -------"
rm -rf /backups/*.out
cd /backups/
mysqldump --all-databases -uroot -ppassword > $(date +%m%d%Y).out
cd /root/s3cmd-1.5.0-alpha3
./s3cmd put /backups/$(date +%m%d%Y).out s3://backups
echo "------- Finished " $(date) " -------"

It’s also good for backing up crucial log files in environments where a dedicate syslog server isn’t really justifiable or is perhaps a little too pricy.

Side note – you can also use this to push data into s3 that is to be served through cloudfront, making scripting media into a CDN simple.

Updating AWS DNS records from the CLI

One of the most useful features of AWS is the ability to do pretty much everything from the provided CLI tools.  Even more usefully they are actually pretty easy to use!

For a number of reasons (including automating deployments, updating records based on dynamic IP addresses and creating internal hostnames for instance deployments) I wanted to be able to push updates to DNS zones hosted on AWS Route53, and I wanted to be able to script the process.  Below is an example of how acheive these updates from the CLI (in this instance updating an existing host record).

Assumptions:  You have installed and configured the AWS cli tools & the credentials you are using have the permissions necessary to make updates.  If you need any pointers with this you can find AWS’s documentation here.

Step 1 – Get the hosted zone ID

When you push a DNS update to Route53 you need to pass in the ID of the hosted zone (a hosted zone normally being the domain name you wish to update).  This command will list all of the zones / domains currently hosted under your account:

aws route53 list-hosted-zones

returning an output along the lines of:

    "HostedZones": [
            "ResourceRecordSetCount": 4,
            "CallerReference": "C510CAC3-D5D9-XXXX-B039-1DFA2XXXXXXX",
            "Config": {},
            "Id": "/hostedzone/Z1W9BXXXXXXXLB",
            "Name": "oliverhelm.me."
    "IsTruncated": false,
    "MaxItems": "100"

“Id”: “/hostedzone/Z1W9BXXXXXXXLB” is the bit you’re looking for with everthing after ‘/hostedzone/’ being the ID (in this instance Z1W9BXXXXXXXLB).

Step 2 – Building the change file

The changes are requested by building out a JSON file which is then sent to AWS. The format of this file varies a little based on the type of record you wish to update (details of this can be found here). In this instance i’m updating the A record homerouter.oliverhelm.me with a new IP address.  Create a file (i’ve called it: change-resource-record-sets.json) and insert the below.

    "Comment": "Update record to reflect new IP address of home router",
    "Changes": [
            "Action": "UPSERT",
            "ResourceRecordSet": {
                "Name": "homerouter.cunniffehelm.co.uk.",
                "Type": "A",
                "TTL": 300,
                "ResourceRecords": [
                        "Value": ""

The action ‘UPSERT’ will update the record for homerouter.cunniffehelm.co.uk if it exists and it not will create it.

Hint: Using http://jsonlint.com/ to check the format of your JSON file saves a fair bit of faffing around.

Step 3 – Pushing the update to AWS

The update is then pushed to AWS with the following command, where ‘change-resource-record-sets.json’ is the name of the JSON file you saved above and ‘–hosted-zone-id’ is the ID you found in step one:

aws route53 change-resource-record-sets --hosted-zone-id Z1W9BXXXXXXXLB --change-batch file:///root/change-resource-record-sets.json

A JSON responce will be returned (making it easy to script the interaction) and should look something like the below.  In theory it might take a short while for the update to take effect but in my experience it seems to be pretty much instant.  The status should be ‘PENDING’ upon submission and will change to ‘INSYNC’ after the change has been applied.  Don’t forget though that you may not see the change reflected in DNS queries until the TTL has been reached.

    "ChangeInfo": {
       "Status": "PENDING",
       "Comment": "Update home IP Address",
       "SubmittedAt": "2015-08-16T11:54:24.907Z",
       "Id": "/change/C2JAIG0XXXXXXX"

You can check the status of any submitted updates with the command:

aws route53 get-change --id C2JAIG0XXXXXXX

where ‘–id’ is the ID returned after the submission.  Further details on this here.

Copyright © 2019 Oliver Helm

Theme by Anders NorenUp ↑