Backups Moved To Amazon S3

Previously, I’ve been backing up all the irreplaceable household data (movies, photos, etc…) to my colo’d server using Scott Ludwig’s most excellent Link-Backup script.

Unfortunately I’ve been running out of space on that server and am thinking about decommissioning it, so as the first step of the decommission I need to move my offsite backups somewhere else. The obvious place appears to be Amazon’s S3 service as the price of storage seems very cheap - $0.15 per gigabyte. There are also transfer charges, but in the general case it’s upload only, so I’m ignoring that. I have about 100GB to store, which works out at $15 per month.

Very reasonable.

The next thing to figure out is “how do I perform the backup?” I initially thought that the easiest thing to do would be to continue to use Link-Backup. All I need to do is bring up and EC2 instance every time I need to backup and then use Link-Backup as I had been doing previously.

The big problem with this was that bringing up an instance is not trivial. Relatively simple, but not trivial. Also, the EC2 tools require Java 1.5 to run and it’s not available on PPC Debian linux. Sigh.

I then dig a search around and found Jeremy Zawodny’s excellent blog post on the subject which provides a nice roundup of the available solutions. I decided to go with the ruby based s3sync.

Set up was pretty simple. I first created an AWS account, collected the required data and configured s3sync. I then created some scripts as suggested by John Eberly and I was off to the races!

I ran some tests to make sure I could:

1. Backup a small directory that had some hierarchy in it.
2. Restore that directory.
3. Make some small changes to the local directory and make sure that only the changes were synced.
4. Pull back a single file using the included s3cmd script.

It all worked perfectly, so now I have ~70GB of photos syncing up to S3.

Sweet!

Next steps:

1. Add everything else (home movies, etc…) to the backup
2. Rebuild my little webapp using Google’s AppEngine.
3. Move my blogs somewhere else (maybe Media Temple - an opinions? I need to run MovableType and MySql).
4. Decommission server.
5. Use saved cash to buy more toys.

Sounds like a plan.