Amazon’s Simple Storage Service (S3) has a lot to like. It’s cheap, can be used for storing a little bit of data or as much as you want, and it can be used for distributing files publicly or just storing your private data. Let’s look at how you can take advantage of Amazon S3 on Linux.<\/p>\n
Amazon S3 isn’t what you’d want to use for storing just a little bit of personal data. For that, you might want to use Dropbox, SpiderOak, ownCloud, or SparkleShare. Which one depends on how much data, your tolerance for non-free software, and which features you prefer. For my work files, I use Dropbox \u2013 in large part because of its LAN sync feature.<\/p>\n
But S3 is really good if you need to make backups of a large amount of data, or smaller amounts but you need an offsite backup. It’s also good if you want to use S3 to host files for public distribution and don’t have a server or need to offload data sharing because of capacity issues. Maybe you just want to\u00a0use it to host a blog, cheaply<\/a>. S3 also has some nifty features for content distribution and data storage from multiple regions, which we’ll get into another time.<\/p>\n You can use S3 in a number of ways on Linux, depending on how you’d like to manage your backups. If you look around, you’ll find a bunch of tools that support S3, including:<\/p>\n S3 Tools and Duplicity are command line utilities that support S3. S3 Tools, as the name implies, focuses on Amazon S3. Duplicity has S3 support, but also supports several other methods of transferring files. Deja Dup is a fairly simple GNOME app for backups, which has S3 support thanks to Duplicity. Dragon Disk is a freeware (but not free software) utility that provides more fine-grained control of backups to S3. It also supports Google Cloud Storage and other cloud storage software.<\/p>\n For the purposes of this article, I’m going to focus on S3 Tools. If you’re a GNOME user, it should take very little effort to set up Deja Dup for S3. We’ll tackle Duplicity and Dragon Disk another time.<\/p>\n You might find S3 Tools in your distribution’s repositories. If not, the S3 Tools folks have\u00a0package repositories<\/a>\u00a0and have support for several versions of Red Hat, CentOS, Fedora, openSUSE, SUSE Linux Enterprise, Debian, and Ubuntu. You’ll also find instructions on adding the tools on the package repositories page.<\/p>\n Once you have S3 Tools installed, you need to configure it with your Amazon S3 credentials. If you haven’t signed up for them yet, hit the\u00a0Sign Up<\/strong>\u00a0button at the top of\u00a0the S3 overview page<\/a>. You’ll also want to look at the\u00a0pricing<\/a>, which starts at $0.125 per GB per month.<\/p>\n The\u00a0pricing calculator<\/a>\u00a0can help you get an idea how much it would cost to store your data in S3. For example, if you’re storing 100GB in S3, it would run about $12.50 per month – before any costs for data transfer\u00a0out<\/em>\u00a0of S3. Transfer\u00a0in<\/em>\u00a0to S3 is free. Amazon also charges for get\/put requests and so forth – so if you’re using S3 to serve up content, then the pricing is going to be higher.<\/p>\n Back to the tools. You need to configure s3cmd (the command line utility from the S3 Tools project) like so:<\/p>\n It will walk you through adding your Amazon credentials and GPG information if you want to encrypt files while stored on S3. Amazon’s storage is supposed to be private, but you should always assume that data stored on remote servers is potentially visible to others. Since I’m storing information that has no real need for privacy (WordPress backups, MP3s, photos that I’d happily publish online anyway) I don’t worry overmuch about encrypting for storage on S3.<\/p>\n There’s another advantage of foregoing GPG encryption, which is that s3cmd can use an rsync-like algorithm for syncing files instead of just re-copying everything.<\/p>\n Now to copy files and use s3cmd sync. You’ll find that the s3cmd syntax mimics standard *nix commands. Want to see what is being stored in your S3 account? Use\u00a0 Want to copy between buckets? Use\u00a0 To put files in a bucket, use\u00a0 But if you want to sync files and save yourself some trouble down the road, there’s the\u00a0 The first time, it will copy up all files. The next time it will only copy up files that don’t already exist on Amazon S3. However, if you want to get rid of files that you have removed locally, use the\u00a0 It’s pretty simple to use\u00a0 <\/p>\n <\/p>\nGetting the Tools<\/h2>\n
\n
S3 Tools<\/h2>\n
s3cmd --configure<\/code><\/p>\ns3cmd ls<\/code>\u00a0to show all buckets. (Amazon calls ’em buckets instead of directories.)<\/p>\ns3cmd cp\u00a0bucket1<\/em>\u00a0bucket2<\/em><\/code>. Note that buckets are specified by the syntax\u00a0s3:\/\/bucketname<\/em>.<\/p>\ns3cmd put\u00a0filename<\/em>\u00a0s3:\/\/bucket<\/em><\/code>. To get files, use\u00a0s3cmd getfilename<\/em>\u00a0local<\/em><\/code>. To upload directories, you need to use the\u00a0--recursive<\/code>\u00a0option.<\/p>\nsync<\/code>\u00a0command. It’s dead simple to use:<\/p>\ns3cmd sync\u00a0directory<\/em>\u00a0s3:\/\/bucket\/<\/em><\/code><\/p>\n--delete-removed<\/code>\u00a0option. Note that you should test this with the\u00a0--dry-run<\/code>\u00a0option first. You can accidentally delete files that way.<\/p>\ns3cmd<\/code>, and you should look at its man page as well. It even has some support for the CloudFront CDN service if you need that. Happy syncing!<\/p>\nDownload<\/a><\/h3>\n