Kevin L
Chairman & CEO, Didit - Founder, We-Care.com - Columnist - SEMPO BOD , Author "The Eyes Have It"
Amazon EC2 / SimpleDB or other scalable cloud computing platform
I may be building (or having built) an online directory similar to Alexa (which runs on Amazon's EC2 and SimpleDB). What factors should I use to make a decision and what other highly scalable cloud style options should I consider?
Does EC2 have a high uptime percentage?
Answers (11)
Amazon's EC2 system should have enough uptime for your needs, as you will most likely want to have multiple servers running at the same time. As for SimpleDB, that may not be able to handle Alexa type loads. You will need a *lot* of storage and performance for something of that magnitude.
It seems to me that Database is going to be the biggest issue with a system like this.
I'm also in the process of evaluating "cloud computing" options such as Amazon Web Services. If you're interested in AWS you should take a look at Rightscale and their value-add, and Scalr. Other options include Mosso (from Rackspace), Google's AppEngine, and many more. The viability of any of these will depend on how well your application matches what they're providing, of course. In particular, the big wins seem to be for applications that break down into components which are loosely coupled and can scale independently. For an online directory, I would be surprised if SimpleDB will be up to the task -- you're more likely going to need to set up your own database (you might consider looking at Google for BigTable, or using something like Hypertable). Good luck, these are really interesting possibilities these days.
You may want to look at the following blog to understand how EC2 compares with Google App Engine:
http://blogs.zdnet.com/Hinchcliffe/?p=166
On the data storage part, Google BigTable is certainly much more scalable and dynamic. The catch is you would be "locked in" to App Engine (beta).
If you want to build something from scratch and are looking at scaling the hosting platform, you could consider slicehost.com as well.
You basically get a connected (virtual) machine which you can scale up/down at any time. However, you need to do basic systems administration yourself.
The rest is up to you / your developers: RoR, PHP, if it runs on Linux you're fine.
Some keywords: oss platform, virtual server, linux.
Note: the link below contains my referral code.
Links:
In general this seems like a great tactic in terms of scalability. Regarding uptime, you may want to Google "amazon down june 6" and see the results. Amazon was down for a couple of hours that day which also affected their cloud computing operations. You could take this message in 2 ways: A) it's a sign some issues still need to be worked out, or B) it shows that major outages are so rare (once a year or so) that it's big news.
Dipankar S
Software engineer, Web Consultant, Blogger, Entrepreneur
Best Answers in: Using LinkedIn (1)
I would say that unless they have some persistant storage in EC2, it is not recommended to try it out for something that may need to collect data real-time. They have been beta-testing persistant storage for sometime.
I would say that you should wait for some more time before trying to build something on the cloud computing systems and having downtimes while launching. From my current usage, EC2 does not have a high uptime.
Altan K
Technical Architect
Best Answers in: Enterprise Software (2), Web Development (2), Events Marketing (1), Business Development (1), Public Relations (1), Computers and Software (1), Software Development (1)
Hello!
I think this is a good question and there are several factors to consider. So far many of the cloud computing offerings show great promise in where this concept can go but they are all attempting different means. For example Google allows access to their infrastructure, but has a heavy reliance on Python as the language of choice. Also others such as Amazon allow much greater flexibility. Then there are companies such as the many mentioned in these threads which allow for even a greater diversity of options in terms of machine scalability, service scalability (i.e. data, web, etc.), dynamic resource management, and several others. Another cloud computing entity to keep in mind is Elastra who also supports more familiar data service scaling via MySQL.
Overall the differences in all of these approaches are based on making what works more accessible to others and more manageable. Again both Amazon and Google base their offerings on infrastructure that their business models are running on, and as a result offer quite a great platform of scale and availability. The tradeoff is that one has to be much more willing to accept the leaps that they have made in terms of answering the scaling challenge which may not be that easy to accept. For example SimpleDB and by that token BigTable are very great concepts and implementations, but they are not and should never be considered "traditional" such as a relational database model. Other players such are either extenders of such services or offering new services that may be more in line with what people are used to.
In the end which choice you determine to go with should be based on your needs, your comfort level with the chosen vendor's options, and your own thresholds for implementation. Any of these vendors are good for their own reasons, but whether those reasons are valid for what you need is entirely the crucial point that you will have to make.
Great question and I hope you can share your experiences with others when the time comes.
Cheers!
Links:
Amazon EC2 is a good choice which has a very good uptime as they claim i.e 99.999% . To work with the EC2 you can use the Amazon Web Services(AWS) and use the Amazon S3 (Simple Storage Services) to store the data that you use in EC2. You can manage all these by using web service commands of the console provided by amazon alternatively you can take the help of two firms called RightScale and PersistentFS to do the things. RightScale provides a dashboard and takes care of all the scalability and command operations while PersistentFS provides better storage than the Amazon S3, and both RightScale and PersistentFS can work together for you.
Links:
Cloud computing is so cheap you can use two independent clouds and still have costs 3-5x lower than alternatives like colo or managed hosting. Considerations should include total cost, time to market, security, lock-in, scale. To capitalize on EC2 to achieve lowest TCO, fastest ROI, etc., you will need some pretty seasoned linux gurus, who may get spread thin in this market. There's been some FUD around EC2 on security, lock-in, etc., - the machines are simple Xen instances with a little Amazon wrapper. From the kernel level up, the developer has complete control; if there are security, data portability, or any other issues above that layer, it's the developer's fault.
Also, to achieve fastest ROI, you will need to "know your core and outsource the rest" - there are certain needs that just about every web app has (storage, db, cache, logging, search), and yet these wheels get reinvented over and over. The utility computing part of cloud computing holds great promise, that if you develop to/with a service-oriented architecture leveraging these rock-solid metered services, you can minimize your software development costs, get up and running fast, and stay focused on your core as you scale up.
Links:
Hi Kevin,
The Amazon Webservices can definitely handle the load you are looking for. Just read this quote from the AWS case studies page:
"Alexa stores over 12 million objects in Amazon SimpleDB, and performs over 5 million queries against it daily. In addition to SimpleDB, Alexa is using Amazon S3, EC2, and SQS as the scalable foundation to power their own high volume web services delivering web search and web information."
I have included a few more links regarding AWS and Alexa for your perusal.
That said, you should give a lot of thought to your infrastructure design in the cloud. One of the keys to being successful on a cloud platform is planning for and anticipating failures, and designing resiliency around that.
Especially for someone whose experience is primarily in the software layer, you will definitely want to take a look at platform management tools for the cloud such as RightScale, Enomaly and Scalr among others.
While AWS has been around longer than some others and is more 'visible', there are other cloud/grid/utility computing providers as well that you may want to consider such as GoGrid, MediaTemple (gs), 3Tera AppLogic and others.
This is a popular and growing space so more options will appear at lower prices in the near future. Good luck with your endeavors!
Best regards,
- Phil
---------
Links:
- http://aws.typepad.com/aws/2008/05/amazon-simpledb.html
- http://aws.typepad.com/aws/2007/12/a-place-for-eve.html
- http://www.amazon.com/Success-Stories-AWS-home-page/b?ie=UTF8&node=182241011
- http://www.rightscale.com
- http://www.enomaly.com
- http://www.scalr.net
- http://gogrid.com
- http://www.mediatemple.net/webhosting/gs/
- http://www.3tera.com/Cloud-computing/
Links:
Go with the Amazon EC2 solution. At the end of the year they will let you mount permanent storage in which you can install a Master-Master MySQL solution for fault tolerant writes to which you can attach several Slaves for speed up the readings.
Using the Amazon Web Services, you can add / remove MySQL slaves in real time according to the site needs.
You can use EUCALIPTUS a tool to simulate your enviroment before posting it to EC2.
You can use PHP for programming. There are a lot of human resources in the market for PHP.
There are companies that can help you setup that kind of scenario in EC2, so you don't have to invest 6 months learning that.
Good luck with your projects!