What is Amazon S3?
Amazon S3 is a vast distributed storage cloud which is accessed via a HTTP REST engine to enable the storage and retrieval of data with automatic redundancy and geographic distribution.
Whilst it is a powerful platform, it's also a very basic implementation which focuses on the lower-level storage and retrieval, providing an API for the development of more "intelligent" engines.
I personally host 100s of gigabytes of data in the S3 cloud, and have invested time into the efficient storage and retrieval of this data. This page documents some of the potential uses.
HOWTO: Mirror your s3backer volume across continents for availability and assurance
Whilst Amazon's s3 service provides Service Level Agreements and in-built redundancy, it is still vital that your data be available whenever possible.
This strategy combines two s3backer volumes, stored in the US and EU datacentres, to be logically bound together as a single block device under Linux, enabling client-side mirroring and failure avoidance through software-level RAID mirroring.
HOWTO: Use Amazon S3 + s3backer + Linux LVM2 for unlimited, flexible remote storage
s3backer is a great way to utilise Amazon S3 storage as a virtual block device on permanently-connected machines. It works by mapping a virtual file on a filesystem (presented using FUSE)
While using s3backer to store data on Amazon S3, I was concerned about the inefficiency of spawning hundreds of s3backer userspace processes once my storage requirements become more complicated. Even more difficult is the management of the storage, which is inflexible for a constantly-changing environment. Read my HOWTO on combining s3backer with LVM2 to enjoy the benefits of both.
[ Read HOWTO here ]
s3backer storage cost calculator
I've written a simple calculator here.
s3rcs - Simple Amazon S3 Revision Control
This set of scripts provides an rcs-like revision control system, using the Amazon S3 storage engine to store revision information for files.
Please Note that this software is in its' infancy, it is important to read the documentation below to understand potential issues. Remember that use of the S3 storage product incurs real fees and an understanding of the system and how this applies is important.
[ View Documentation | Download Version 0.10 ]
