Soapbox Engage Blog

You have a very large video file, pdf or document... or you have a large number of smaller files.  These can become too large to store on your web server, and they can quickly push your beyond your bandwidth allowance. So, what do you do?

One solution worth considering is Amazon's S3 cloud storage solution.  S3 allows you to store large amounts of data at a surprisingly low cost.  Additionally, the cost structure is scalable, so you will only ever pay for what you use.

What is Amazon S3?

S3 stands for Simple Storage Service.  Simply put in Amazon's own words, "Amazon S3 is storage for the Internet. It’s a simple storage service that offers software developers a highly-scalable, reliable, and low-latency data storage infrastructure at very low costs."

All that means is that it is a service that allows you to store data in the "cloud" in a way that is fast, cheap and reliable.

How much does it cost?

The cost of hosting files on S3 is almost negligible.  They charge about $.15/gig a month and transfer costs are about the same.  So, even with a large number of files and a decent amount of traffic, you are usually looking at single-digit dollars a month.

You can calculate the costs yourself, but a brief example would be a large 500M video file, viewed 100 times over a month.  That would be $.07 for storing it, and transfer costs would add up to less than $8.50. So, less than $9.00 total for storing and serving a very large file.

How does it work?

It's actually pretty easy.  You can use one of several "clients" to create your S3 account and upload and manage your data (a "client" is a piece of software that allows you to access a remote server), then link to it directly from your website.

For uploading and managing, we recommend the S3 client called CloudBerry S3 Explorer.  It's easy to use and makes your experience with S3 simple and enjoyable.  It's not free though - it costs $40 after a 30 day free trial period. But in our experience it is worth the $40 as it has features and ease-of-use that make it preferable to the few free solutions that are out there.

Let's go through the steps of setting CloudBerry up and connecting to Amazon S3:

• Download and install CloudBerry S3 Explorer.  You can get it here: http://cloudberrylab.com/?page=s3-explorer-pro
• Install it the same as you would any other new application.

Once you have done that, it's time to set up your S3 account.

Once you have completed the registration process, Amazon will send you a confirmation email.  Check your inbox and open it up.  You will see the following message in the email:

You need Access Identifiers to make valid web service requests. Please visit the Access Identifiers section of your account to obtain your identifier and to learn more: http://aws-portal.amazon.com/gp/aws/developer/account/index.html?action=access-key

Click through that link, then scroll down to the "Access Credentials" section, and look under the "Access Keys" tab.  You are looking for you "Access ID Key" and "Secret Access Key" (It will say "show."  Click "show" to see the key.)

Connecting CloudBerry S3 Explorer

Open CloudBerry S3 Explorer. It should open as a two-panel explorer. The left panel should show your local drive and the right panel will be for you S3 account.

At the top of the right panel, you will see "Source:" and a drop-down menu.  Select "New Amazon S3 Account."

You will be asked for "Display name" (put in whatever you want this account to be referred to,) and the Access Key and Secret Key that you got earlier from your S3 account.  Enter the name and those keys and click "OK."

Creating your first "Bucket."

S3 organizes files into "buckets."  These are sort of like folders on your desktop.  You have to have a bucket before you can upload anything to your S3 storage.

On the S3 (right) half of CloudBerry, look for and click a blue cube.  This is the "create a new  bucket" icon.  You will link to these buckets with a URL, so the bucket name should have something to do with what you will be storing there.  In other words, if you will be putting videos from a conference there, you could call it videosConf2009.  Note that bucket names have to be unique across the entirety of Amazon S3.

Bucket permissions

One thing you have to make sure of... that your bucket's permissions will allow public access.  The default settings set access to "private," so you will probably have to change them.  To assure this, right-click your bucket and select "ACL Settings."  Set each user type to "Read," and Owner should have all of the boxes checked.

Uploading a file

Once you have your bucket set up, it's time to upload your first file.  This is really simple to do.  In the left-hand side of CloudBerry, navigate your local computer's C drive to find the file to upload.  Then, simply drag it to your new bucket.  It will upload, and you are done!

Linking to your file

Ok, now you want to know how to link to your file from the web, don't you?  This is the nice part.  To link to your files, use a url that looks like this: http://s3.amazonaws.com/BucketName/FileName

It's that easy.  Continuing our example from above, we might have a url that looks like this: http://s3.amazonaws.com/videosConf2009/vid001.mp3

Then of course, with HTML, to link to your file it would look like: <a href="http://s3.amazonaws.com/BucketName/FileName">Link Name</a>

And that's it!

By leveraging Amazon S3, you should be able to store and serve very large files, and/or large numbers of files easily and inexpensively.  You can also tell people that your data is stored "In the cloud," which is an impressive buzzword these days - it will identify you as being on the edge of online technology!

After working through this tutorial, you should now be able to:

• Understand what Amazon S3 is, and why it's a good choice for large file storage
• Download and install a client that will allow you to manage your account
• Create an S3 Account
• Create buckets, and properly set their permissions
• Upload files from your local drive to your S3 Account
• Link to the uploaded files via your website.

I hope this has been helpful... have fun and happy S3ing!