Leveraging Aamzon's Servers for your own Website

[40hz]

If you generate a lot of traffic; host large files; or you're running into bandwidth caps, you need to know about Amazon S3 storage services.

Never heard of S3?  From Amazon's FAQ:

Q: What is Amazon S3?

Amazon S3 is storage for the Internet. It's a simple storage service that offers software developers a highly-scalable, reliable, and low-latency data storage infrastructure at very low costs.

Q: What can I do with Amazon S3?

Amazon S3 provides a simple web services interface that you can use to store and retrieve any amount of data, at any time, from anywhere on the web. Using this web service, developers can easily build applications that make use of Internet storage. Since Amazon S3 is highly scalable and you only pay for what you use, developers can start small and grow their application as they wish, with no compromise on performance or reliability. It is designed to be highly flexible: Store any type and amount of data that you want; read the same piece of data a million times or only for emergency disaster recovery; build a simple FTP application, or a sophisticated web application such as the Amazon.com retail web site. Amazon S3 frees developers to focus on innovation, not figuring out how to store their data.

Link: www.amazon.com/gp/browse.html?node=16427261


This article provides a decent how-to to get you started.

http://nerdbusiness....th-machine-amazon-s3

How to Create an Infinite Bandwidth Machine with Amazon S3

April 15, 2008 By Schwabe

3 comments

We have entered a new era of the internet. An era of infinite bandwidth! Now you can take advantage of Amazon's vast network of servers to allow your website to survive even the most extreme spikes in bandwidth. Here's a tutorial on how to to access S3 like an FTP server to host your bandwidth sucking media files (pictures, videos, mp3's, zips or any other type of file).

And in this article, I'll show you how you can integrate your existing website on any typical hosting platform to "tap in" to this infinite bandwidth supply and leverage the power of the Amazon S3 infrastructure.

I've been working with a few people that are using this technology. I was so impressed at how well S3 worked out that I have been meaning to do an article about it for some time. Then I ran into the above link purely by chance and figured it was quicker and easier to share that. 

[Veign]

Didn't S3 have a pretty large outage last week?

This is something I have been meaning to do.  I wanted to create an article on how to create a simple, free, backup system using an online storage service for mom and pop home user.

[40hz]

Didn't S3 have a pretty large outage last week?

This is something I have been meaning to do.  I wanted to create an article on how to create a simple, free, backup system using an online storage service for mom and pop home user.

-Veign (July 22, 2008, 10:38 AM)

Yes they did. About six hours to get everything fully back to normal. Here's what Amazon had to say about it:

Here’s some additional detail about the problem we experienced earlier today.

Early this morning, at 3:30am PST, we started seeing elevated levels of authenticated requests from multiple users in one of our locations.  While we carefully monitor our overall request volumes and these remained within normal ranges, we had not been monitoring the proportion of authenticated requests.  Importantly, these cryptographic requests consume more resources per call than other request types.

Shortly before 4:00am PST, we began to see several other users significantly increase their volume of authenticated calls.  The last of these pushed the authentication service over its maximum capacity before we could complete putting new capacity in place.  In addition to processing authenticated requests, the authentication service also performs account validation on every request Amazon S3 handles.  This caused Amazon S3 to be unable to process any requests in that location, beginning at 4:31am PST.  By 6:48am PST, we had moved enough capacity online to resolve the issue.

As we said earlier today, though we're proud of our uptime track record over the past two years with this service, any amount of downtime is unacceptable.  As part of the post mortem for this event, we have identified a set of short-term actions as well as longer term improvements.  We are taking immediate action on the following:  (a) improving our monitoring of the proportion of authenticated requests; (b) further increasing our authentication service capacity; and (c) adding additional defensive measures around the authenticated calls.  Additionally, we’ve begun work on a service health dashboard, and expect to release that shortly.

Sincerely,
The Amazon Web Services Team

Nothing is perfect, alas! But I thought the candor displayed by Amazon regarding the problem was refreshing to say the least. The first step in preventing future problems is to be honest about the problem you're currently fixing. There's a lesson in there somewhere...

[tomos]

thought this was interesting
from Amazon

We wanted to share a brief note about what we observed during yesterday's event and where we are at this stage. As a distributed system, the different components of Amazon S3 need to be aware of the state of each other. For example, this awareness makes it possible for the system to decide to which redundant physical storage server to route a request. In order to share this state information across the system, we use a gossip protocol. Yesterday, we experienced a problem related to gossiping our internal state information, leaving the system components unable to interact properly and causing customers' requests to Amazon S3 to fail. After exploring several alternatives, we determined that we had to temporarily take the service offline so that we could clear all gossipped state and restart gossip to rebuild the state. These are sophisticated systems and it generally takes a while to get to root cause in such a situation. We're working very hard to do this and will be providing more information here when we've fully investigated the incident. We also wanted to let you know that for this particular event, we'll be waiving our standard SLA process and applying the appropriate service credit to all affected customers for the July billing period. Customers will not need to send us an e-mail to request their credits, as these will be automatically applied. This transaction will be reflected in our customers' August billing statements.

-
via Jungle Disk forums
http://forum.jungled...t=16281&start=25

[kartal]

For some reason I do not find Amazon service to be cheap. If you have couple gb to backup sure it is cheap. For people like me who like to back up more than couple gb it is not that cheap. And their pricing scheme just makes it more confusing. For example I may need to transfer between 1 to 20 gb of data (mainly upload). So not knowing how much I need to pay monthly makes it unusable for me. I personally prefer fixed price.

Maybe someone can explain  better why I should use s3 over others for me

[40hz]

For some reason I do not find Amazon service to be cheap. If you have couple gb to backup sure it is cheap. For people like me who like to back up more than couple gb it is not that cheap. And their pricing scheme just makes it more confusing. For example I may need to transfer between 1 to 20 gb of data (mainly upload). So not knowing how much I need to pay monthly makes it unusable for me. I personally prefer fixed price.

Maybe someone can explain  better why I should use s3 over others for me

-kartal (July 22, 2008, 01:32 PM)

The S3 service is more geared towards storing and serving files rather than acting as backup solution - although it can be used for that as well. If you are hosting large media files (photos, videos, ISO images, etc.) on your webhost account you can run up a lot of byte traffic.

You can also easily exceed your allowed bandwidth if anything on your site gets popular or you're pwned by one of the social sites. Happened to an acquaintance of mine after a shareware product she created got a glowing write up on three ultra-popular blogs.

If you're lucky, you'll get shut down by your ISP for exceeding your bandwidth allowance. If you're unlucky, you'll stay up and then get hit with high surcharges. If you have your own server and T1, you run a very good chance of crashing. S3 lets you run your site like you always do - but it serves selected objects from Amazon's infrastructure instead of your own host.

There's a very good example scenario if you'll take a look at the second link in the original post. I've repeated it here:

http://nerdbusiness....th-machine-amazon-s3

I can't really say much about price since everybody's purse is a different size. From my perspective (i.e. US-based prices + no kids in school  ) their pricing structure seems reasonable. But that's just me.

Hope that helps

[40hz]

This just in: you can now monitor the status of the Amazon cloud from your browser courtesy of Hyperic CloudStatus.

From the website:

CloudStatus Provides Free, Real-Time Performance Information for Amazon Web Services

Velocity Conference—San Francisco, Calif.—June 23, 2008 – Open source web infrastructure management provider Hyperic Inc., (Velocity Booth #6), today launched the beta of Hyperic CloudStatus, the first service to provide an independent view of the health and performance of the most popular cloud on the Internet, Amazon Web Services (AWS). The new service gives businesses that use the cloud the perspective they need to determine the cause of performance changes in their cloud-based web applications. CloudStatus beta is a free service built on the Hyperic HQ management platform and will expand to include additional cloud providers this summer. Read More

http://www.hyperic.c...008-cloudstatus.html

See it in action from this link:

http://cloudstatus.com/

[Renegade]

Amazon makes sense when you start getting into very high bandwidth usage. For most people, it's ok, but certainly not necessary. Their pricing gets better the more you use.