Two years ago, I blogged about ChessJam, a chess application that I helped develop where people from all over the world play live chess with each other. Since launching ChessJam, it’s been up and running 24/7 with very little downtime and has experienced some good growth.
To give you a sense of the activity levels, here are a few ChessJam stats:
- At any time, day or night, there is an average of 100 users logged in playing chess. We’ve seen it as high as 180 concurrent on weekends.
- Our average load is about 2 or 3 chess moves per second.
- During the month of March, 2012, more than 4,500 users played 128,920 games (5.8 million moves in one month)
Not bad for no marketing budget I think. It’s not a huge amount of activity compared to a lot of applications, but the performance demands are very high. Serious chess players don’t tolerate delays in response time, so we have to make sure we have plenty of CPU and bandwidth to spare.
Late 2009/Early 2010 — development and early production – hosting from my home office
During development and the first few months of production, ChessJam was hosted from a computer in my home office. The cost was around $70/month for FiOS fixed IP address plus whatever power my server was drawing. This was fine until the Florida summer storms took out my power twice in a single month. That’s when we realized the difference between a real data center and my personal data center! It was also a bit stressful when I traveled because it left my wife in charge of late-night server repairs.
Late 2010 — moved to rack mounted servers in a real data center
We acquired two used (circa 2008) rack-mounted IBM servers and rented some rack space at a local data center. Reliable power and Internet were no longer a problem, but the hardware was ours to manage and eventually repair. It was better, but it wasn’t a great situation. Once we settled in, our cost was around $190/month (rack space rental and bandwidth).
Why didn’t we move to Amazon EC2 instead of the data center? We did consider moving from my home office to Amazon EC2 instead of the data center, but at the time, Amazon didn’t have an instance type that suited us. The small instance type was affordable, but it only offers 1.7GB of memory, which just won’t cut it for our server app (we tried). The next step up was the large instance type with 7.5GB of memory, but it was too expensive and overkill.
Two months ago, Amazon filled the huge gap between the small and large instance type by announcing the new “medium” instance type. It provides 3.75GB of memory as well as a decent amount of CPU for a good price… so we decided it was time to make the move.
Our EC2 configuration:
- Medium EBS instance running Ubuntu Linux. EBS is “Elastic Block Store”, which basically means that the storage is persistent. If you haven’t looked at EC2 in the past two years, you might not know that persistent storage on server instances is an now an option. With EBS storage, you can shut down your server and all of the data is saved (just like a physical server).
- 200GB EBS disk volume – for our MySQL database and fast growing log files.
AWS and bill anxiety
Most people exploring Amazon EC2 ask the same question I asked… “What is this really going to cost me?”. When you dig into how EC2 is priced, you’ll find that there are multiple variables, which creates a bit of uncertainty. Basically, it boils down to the server instance type, the disk space used, and the amount of data moved in and out of the instance. There are other small charges for number of I/Os and any unused IP addresses, but these are typically only a few cents and don’t have much impact on the final bill (based on my experience).
- The server instance — this one is easy and predictable. Amazon charges by the hour. For example, our medium instance is $0.16 per hour. You could fire up a medium server instance and spend a full 8 hour work day playing with it and only pay about $1.30. Not bad! Our app runs 24/7 so 24 hours per day * 30 days = 720 hours per month. So, our monthly cost can be calculated with $0.16 * 720 = $115/month. (later in this article, I explain how reserved instances work and how they can lower this price by nearly half)
- Disk volume — this one is also easy and predictable as well. It cost $0.10 per GB per month. We have a 200GB volume, so that’s 200*$0.10 = $20/month.
- Disk I/Os — The EC2 pricing page also shows a $0.10 charge per million I/Os. We do about 300,000 I/Os per day (based on my last bill) so that comes to about 9 million I/Os per month resulting in less than $1/month… so we’ll just call that a rounding error.
- Data transfer charges — this is the part that scared us. Amazon charges $0.12 per GB of data transferred out of our EC2 instance. They do not charge for inbound traffic. This method of charging for data is very different than how our data center charges us. The data center uses “95th percentile peak billing” which means we are charged based on the average rate that we move data, not the volume of data moved. I’ll spare you the details, but it doesn’t translate to Amazon’s model. However, our data center provider does have a report that shows the total amount of data transferred each month and we used that number to estimate. Now that we’ve been up and running a few days, I know that we transfer about 250MB of data every hour or 6GB of data per day. So, 6GB * 30 days * $0.12 = $21/month.
Summary: $115.20/month for our server instance, $20/month for our disk volume and $21/month for data transfer which comes to about $156.20/month. That’s already about $30 less than we were paying the data center. That price is already lower than what we were paying… but with reserved instances, we can lower it considerably.
Saving money with reserved instances
A lot of companies use Amazon EC2 in a much more “elastic” fashion than ChessJam does. They set up load balancers and auto-scaling to dynamically increase and decrease instances based on the load. However, our app is not nearly that sophisticated (although eventually our usage may justify re-architecting our back end!). Our app uses a consistent amount of EC2 resources. Thankfully, Amazon offers special pricing for this type of use-case called reserved instances. Reserved Instances give you the option to make a one-time payment for each instance you want to reserve and in turn, receive a significant discount on the hourly charge for that instance. Basically, Amazon is rewarding you for being predictable. There are three different options, light utilization, medium utilization, and heavy utilization. Our server is up and running 24/7, so our best deal is the heavy utilization model.
I reserved a medium instance for one year by paying an upfront fee of $390. Now, instead of paying $0.16 per hour, we pay $0.032 per hour (just over 3 cents!). Some simple math shows the savings:
- With a normal “demand instance”, we paid $0.16 per hour * 24 hours per day * 365 days per year resulting in a yearly cost of $1,401.60 (plus disk, I/O, transfer)
- With the reserved instance, we paid $390 up front and then $0.032 per hour * 24 hours per day * 365 days per year resulting in a yearly cost of $670.32 (plus disk, I/O, transfer)
Bottom line cost including disk, data, etc.
- We paid around $2,280 a year to the data center.
- We now pay $1,160 a year to Amazon EC2 – saving us over $1,000 per year (not counting any hardware costs we could have incurred).
There are two smaller instance types – micro and small. You might can get by far cheaper for your app/site. With reserved instance pricing, you can get a micro instance for about $100 per year! I currently host 6 low-use websites on a single micro instance (local bagel store, my aerial photo site, etc.).
TIP: If you plan to use Amazon’s micro instances, you should be aware of how they throttle CPU. It’s unique to the micro instance type. I blogged about this months ago.
TIP: Amazon’s billing data is updated several times per day, so you can quickly determine what your run-rate is for everything. I was able to project our monthly cost after 24 hours of activity.
We didn’t move to EC2 just to save money
Yes, we’re saving some money – that’s awesome, but to be honest, it’s not the reason we moved. The real reason we wanted to move to a cloud solution like EC2 is for peace of mind and to prevent us from dealing with a hardware failures. We had a backup server in the data center, but if there was a failure in the primary server, it would likely result in a 30 minute drive to the data center and a few hours in a cold room moving hard drives from server to server and hoping it all goes well. It was a precarious situation! Now, if there is a hardware failure (rare, but they happen on EC2), I simply launch a new instance with a few clicks of the mouse, mount our disk volume to it, tweak a few things, and get us back online. All of this while watching Big Bang Theory in my PJs!
I know that a few of you IT-minded folks are seeing some holes in my final architecture, and yes, you’re correct. The above scheme doesn’t address a disk volume crash. Disk volume crashes are very rare on EC2, but I do plan to make a few more enhancements to the architecture. I could start up a new instance and use it as a slave server to replicate data from the primary server’s database (MySQL). I could start this backup instance in a completely different zone, which basically means that it’s in a different physical facility. This would allow us to quickly recover if Amazon has a massive failure like the one a year ago that brought down several popular sites for a few hours. However, I might go the cheap route and create a 2nd disk volume and use a file system that allows me to mirror the database between the two disk volumes. I think this will be adequate until someone buys us for a $1 billion.
Another option to mention is Amazon RDS (MySQL in the cloud). If we used RDS for our database, it would give us more redundancy and scalability, but for now, we’re going to avoid that cost and stay on the single instance until we grow more.
It may not be for all applications
Our experience with EC2 was very positive, especially with this application. However, it’s not perfect for all applications. The server is still “virtual” so the performance is not perfectly
consistent. Things like disk I/O speed, network bandwidth, etc., is what it is. You can’t tell Amazon that you need a SSD for your swap drive and 10,000RPM drives for your databases. In other words, you don’t have as many ways to tweak performance as you do by managing your own real hardware. However, if your app runs well on EC2, why bother with that stuff?! I personally feel that I’ve served my time in computer rooms during my career.
The rest of the Amazon Web Services world
Everything we’ve talked about in this article has been specific to the low-end instance types of Amazon EC2. There are many other instance types and other related features including load balancers, auto-scaling, monitoring, automatic alerts, etc. Also, EC2 is only one of many services offered by Amazon. Check them all out at http://aws.amazon.com/.
Check it out for free
Amazon recently started offering a “free tier” for EC2 newbies. You basically get a year free of a micro instance along with some disk space and bandwidth. See http://aws.amazon.com/free for more details. Remember – you can resize instances anytime. It’s very common to use a micro instance during development, and then upgrade as needed when you go live.