Main Area and Open Discussion > General Software Discussion
Leveraging Aamzon's Servers for your own Website
40hz:
If you generate a lot of traffic; host large files; or you're running into bandwidth caps, you need to know about Amazon S3 storage services.
Never heard of S3? From Amazon's FAQ:
Q: What is Amazon S3?
Amazon S3 is storage for the Internet. It's a simple storage service that offers software developers a highly-scalable, reliable, and low-latency data storage infrastructure at very low costs.
Q: What can I do with Amazon S3?
Amazon S3 provides a simple web services interface that you can use to store and retrieve any amount of data, at any time, from anywhere on the web. Using this web service, developers can easily build applications that make use of Internet storage. Since Amazon S3 is highly scalable and you only pay for what you use, developers can start small and grow their application as they wish, with no compromise on performance or reliability. It is designed to be highly flexible: Store any type and amount of data that you want; read the same piece of data a million times or only for emergency disaster recovery; build a simple FTP application, or a sophisticated web application such as the Amazon.com retail web site. Amazon S3 frees developers to focus on innovation, not figuring out how to store their data.
--- End quote ---
Link: www.amazon.com/gp/browse.html?node=16427261
This article provides a decent how-to to get you started.
http://nerdbusiness.com/blog/how-create-infinite-bandwidth-machine-amazon-s3
How to Create an Infinite Bandwidth Machine with Amazon S3
April 15, 2008 By Schwabe
3 comments
We have entered a new era of the internet. An era of infinite bandwidth! Now you can take advantage of Amazon's vast network of servers to allow your website to survive even the most extreme spikes in bandwidth. Here's a tutorial on how to to access S3 like an FTP server to host your bandwidth sucking media files (pictures, videos, mp3's, zips or any other type of file).
And in this article, I'll show you how you can integrate your existing website on any typical hosting platform to "tap in" to this infinite bandwidth supply and leverage the power of the Amazon S3 infrastructure.
--- End quote ---
I've been working with a few people that are using this technology. I was so impressed at how well S3 worked out that I have been meaning to do an article about it for some time. Then I ran into the above link purely by chance and figured it was quicker and easier to share that. :)
Veign:
Didn't S3 have a pretty large outage last week?
This is something I have been meaning to do. I wanted to create an article on how to create a simple, free, backup system using an online storage service for mom and pop home user.
40hz:
Didn't S3 have a pretty large outage last week?
This is something I have been meaning to do. I wanted to create an article on how to create a simple, free, backup system using an online storage service for mom and pop home user.
-Veign (July 22, 2008, 10:38 AM)
--- End quote ---
Yes they did. About six hours to get everything fully back to normal. Here's what Amazon had to say about it:
Here’s some additional detail about the problem we experienced earlier today.
Early this morning, at 3:30am PST, we started seeing elevated levels of authenticated requests from multiple users in one of our locations. While we carefully monitor our overall request volumes and these remained within normal ranges, we had not been monitoring the proportion of authenticated requests. Importantly, these cryptographic requests consume more resources per call than other request types.
Shortly before 4:00am PST, we began to see several other users significantly increase their volume of authenticated calls. The last of these pushed the authentication service over its maximum capacity before we could complete putting new capacity in place. In addition to processing authenticated requests, the authentication service also performs account validation on every request Amazon S3 handles. This caused Amazon S3 to be unable to process any requests in that location, beginning at 4:31am PST. By 6:48am PST, we had moved enough capacity online to resolve the issue.
As we said earlier today, though we're proud of our uptime track record over the past two years with this service, any amount of downtime is unacceptable. As part of the post mortem for this event, we have identified a set of short-term actions as well as longer term improvements. We are taking immediate action on the following: (a) improving our monitoring of the proportion of authenticated requests; (b) further increasing our authentication service capacity; and (c) adding additional defensive measures around the authenticated calls. Additionally, we’ve begun work on a service health dashboard, and expect to release that shortly.
Sincerely,
The Amazon Web Services Team
--- End quote ---
Nothing is perfect, alas! But I thought the candor displayed by Amazon regarding the problem was refreshing to say the least. The first step in preventing future problems is to be honest about the problem you're currently fixing. There's a lesson in there somewhere... ;)
tomos:
thought this was interesting
from Amazon
We wanted to share a brief note about what we observed during yesterday's event and where we are at this stage. As a distributed system, the different components of Amazon S3 need to be aware of the state of each other. For example, this awareness makes it possible for the system to decide to which redundant physical storage server to route a request. In order to share this state information across the system, we use a gossip protocol. Yesterday, we experienced a problem related to gossiping our internal state information, leaving the system components unable to interact properly and causing customers' requests to Amazon S3 to fail. After exploring several alternatives, we determined that we had to temporarily take the service offline so that we could clear all gossipped state and restart gossip to rebuild the state.
These are sophisticated systems and it generally takes a while to get to root cause in such a situation. We're working very hard to do this and will be providing more information here when we've fully investigated the incident. We also wanted to let you know that for this particular event, we'll be waiving our standard SLA process and applying the appropriate service credit to all affected customers for the July billing period. Customers will not need to send us an e-mail to request their credits, as these will be automatically applied. This transaction will be reflected in our customers' August billing statements.-
via Jungle Disk forums
http://forum.jungledisk.com/viewtopic.php?t=16281&start=25
kartal:
For some reason I do not find Amazon service to be cheap. If you have couple gb to backup sure it is cheap. For people like me who like to back up more than couple gb it is not that cheap. And their pricing scheme just makes it more confusing. For example I may need to transfer between 1 to 20 gb of data (mainly upload). So not knowing how much I need to pay monthly makes it unusable for me. I personally prefer fixed price.
Maybe someone can explain better why I should use s3 over others for me :)
Navigation
[0] Message Index
[#] Next page
Go to full version