Amazon EC2 Micro Instance and Stolen CPU

February 7, 2011 | By

Last week, I moved my blog to Amazon EC2 in order to have more control of it (blogged here). I chose a Micro instance running Ubuntu Linux because Micro is the least expensive (2 cents per hour) and from my previous experience hosting various websites and services on Micro instances, I felt it was more than adequate to support my blog.

To my surprise, the results were disappointing. I noticed that when accessing my blog, it would sometimes pause for several seconds before serving the requested page. I spent several hours trying to figure out what was going on. When I looked in the Apache logs, I realized that the CPU spiked when my blog was crawled by one of several search engine bots. When a full crawl wasn’t happening, my blog was lightning fast but when one of the search engine bots crawled my site, my blog was nearly unresponsive until the bot finished. Obviously this was not acceptable.

While running top, I quickly discovered that when my site was being crawled, the CPU would spike, but the spike was not the typical user or system CPU. It was showing up as 97%st “CPU steal time”. Here’s a screenshot:

During the first 5 to 10 seconds of the bot crawling my site, everything was super fast, but then things would come to a near halt.

Steal time is a metric that only has meaning in a virtualized computing environment. It represents the amount of CPU that is taken away from a virtual machine to serve other purposes.

Amazon’s website describes Micro instances as follows: “Instances of this family provide a small amount of consistent CPU resources and allow you to burst CPU capacity when additional cycles are available. They are well suited for lower throughput applications and web sites that consume significant compute cycles periodically“.

What I’ve discovered is that when you tax the CPU more than a few seconds, you get throttled back almost to a standstill. I put together a brief video where I demonstrate this happening. In this video, I run a simple script that makes repeated calls to sysbench to tax the CPU for 2 seconds. After it runs for a few times, you’ll see an immediate and severe reduction in performance and you’ll see the steal time go to over 97% leaving very little left to run my job.


Advance to the 1:20 mark if you want to skip to the benchmark

I still think Micro instances are awesome, especially considering the cost, but I’ve now learned that they are not right for all applications. A simple 10-page website with only a few thousand hits a day will do just fine since it will only need CPU in short bursts. These burst are executed very fast, so it is a good fit for this type of application. However, if your application needs sustained good CPU performance, you’ll need to upgrade to a Small instance or higher.

The nice side effect of this whole experience is that my blog is very optimized. Pre-cached pages, minified HTML/JS/CSS and a well tuned DB makes a speedy blog!

Editorial: I think the extreme CPU throttling really limits the usefulness of Micro instances. The next step up in the Amazon EC2 world is a small instance but the price jump to a small instance is substantial. Originally I saw micro instances as a “foot in the door” to Amazon EC2. It was a way for someone to try the model and then grow into larger instances as the demand grows. It seems to me that having a model that basically shuts down the CPU after a few seconds of heavy use gives a lot of new users a bad experience, especially considering that Amazon does not clearly explain the model. I think Amazon needs to either modify the micro instance to not throttle CPU as severely, or offer a “tiny” instance that provides a good entry into the EC2 world.


SHAMELESS PLUG – 10/27/2011

As I continued to work with Amazon EC2, S3, Beanstalk, RDS, etc., I quickly saw a need for a tool to help visualize the various AWS assets. I teamed up with James Ward and Jon Rose and we’ve created, “Greg’s Toolkit” for desktop, iPad and Android tablets. I encourage you to check it out at http://www.gregstoolkit.com.

Greg's Toolkit for AWS EC2 S3 Beanstalk RDS and more

Filed in: Amazon EC2, Linux | Tags: , , , , ,

About the Author (Author Profile)

Greg is a developer advocate for the Google Cloud Platform based in San Francisco, California

Comments (39)

  1. Geoff

    I played for a (short) while with a windows server micro instance. It took hours to install sql server express – jammed at 100% cpu all the time – and eventually I gave up. I didn’t realize it was throttled so badly!

    • I had the same experience while installing ColdFusion – got about 50% through the install…then hung for a few minutes. At the time, I didn’t know what was going on…but now I do :)

      Greg

  2. johans

    Greg – have you looked at Rackspace CloudServer? How do you think the smallest (256MB) would compare with EC2 micro instance? Its about $11/month and my understanding is you do not have the CPU capping as with EC2 – fee comparisons here:

    http://www.rackspace.com/cloud/cloud_hosting_products/servers/compare/

    Be interested in your thoughts as I’m a novice with these cloud hosting options.

    • Hey Johans – I’ve been playing with Rackspace CloudServer the past couple of days. The CPU for the low-end instance is far more consistent from my I can see so far. I’m going to be blogging soon comparing RackSpace CloudServer, EC2, Linode and others.

      Thanks!

      Greg

  3. johans

    Look forward to your post.

    I signed up for EC2 but after 4 days, despite verification via phone/PIN I am still stuck in a loop of Account Pending -> Click here to verify -> you are already signed up click here to use -> Account Pending.

    So I went looking and that’s when I saw Rackspace CloudServers.

  4. 2 seconds it a lot of time for a WP blog…

    Q. How does a Micro instance compare in compute power to a Standard Small instance?
    At steady state, Micro instances receive a fraction of the compute resources that Small instances do. Therefore, if your application has compute-intensive or steady state needs we recommend using a Small instance (or larger, depending on your needs). However, Micro instances can periodically burst up to 2 ECUs (for short periods of time). This is double the number of ECUs available from a Standard Small instance. Therefore, if you have a relatively low throughput application or web site with an occasional need to consume significant compute cycles, we recommend using Micro instances.

    • Yes, small instances are slower than the burst speed of a micro instance, but they handle a sustained load far better.

      The burst speed is quite amazing – I just with that when it throttled down, it would leave enough CPU left over to serve up basic HTML pages from Apache. Throttling is severe.

      Thanks!

      Greg

    • C’mon you cannot actually recommend micro instances other than to a beginner who has no idea about cloud computing. “if you have… an occasional need to consume significant compute cycles” – a micro instance will literally come to a halt if you need to compute anything.

  5. johans

    Just to let others know – my EC2 account setup was eventually completed. Seems it can take a few days – my expectation was it was instant as I already had an active account for S3.

    I setup/installed Node – it took about 20 minutes to compile due to the limited CPU. So one has to be patient.

    Your instructions/video really helped a lot. A few things I came across:

    After setting up the Elastic IP I needed to use the IP in place of the public DNS when connecting.

    When connecting via ssh I also had a permission problem with my key pair file – “WARNING: UNPROTECTED PRIVATE KEY FILE!” I solved that by setting permission using chmod 700 keyFileName.pem

    @Nabil – thanks that information is interesting.

    • Johans – that’s weird that it took so long to get your account active. It took about 6 hours on mine.

      One nice thing about EC2 is that you can change the instance type of a given instance. You could start it up as a small instance to get everything installed, and then downsize it to a micro instance. I’ve used micro instances for development and then upsized to small or larger when it’s production time. The flexibility is great.

      I had the same issues on the pem file permissions.

      Thanks!

      Greg

  6. I think there is a misconception that a “small blog” running on WordPress can be easily hosted on the smallest virtual server offerings. In my experience WordPress demands quite a lot more.

    • I’ve seen the same…however… with the wp super cache, you can basically cache your entire site as static HTML pages…so when people access pages on my blog (including this page), they are actually hitting static content…so no PHP or MySQL is being used other than logging.

      I think WP should ship with caching built in…seems like an obvious requirements because without caching, it is a bit of a resource consumer ;)

      Greg

    • Agreed. Just trying to configure a WordPress site (that has NO traffic) very frequently causes throttling on my micro instance. Sigh. Guess I’ll have to investigate Rackspace.

  7. Hi Greg!

    Very interesting. This was new information to me — and makes me reconsider hosting even small WordPress projects on micro instances. I’m looking forward to your comparative tests!

    Thanks,

    Kasper

  8. Btw — what’s the blog running on now?

    • It’s currently on a c1.medium (high CPU) but it is WAY WAY overkill for my blog. I’m consolidating some other sites on this new instance to save money across the board.

      The c1.medium is very fast. If I wasn’t combining with other stuff, I’d go with m1.small. If you pay the reserve price, it gets affordable.

      The awesome thing is that you can change instance types with a simple command line command.

      I’m also wanting to try two micro instances with a load balancer for fun.

      Greg

  9. Thanks, glad I read this. I’m currently on a 256mb rackspace cloud instance. I have 8 low traffic sites, two of them wordpress. I’ve done quite a bit to reduce the amount of memory used, but I’m often bumping the limit so I was considering moving to a Amazon EC2 Micro instance since for only a little more per month I would get more than double the memory, but from what you said, it doesn’t sound like a good move. I’ll think I’ll stick with Rackspace for now.

    • It really depends on whether a search bot scan will trigger the throttling. If all request are under 4 or 5 seconds, it will be dramatically faster than rackspace because the “burst cpu” is awesome. However, if you hit throttling, it will go to near zero.

      What’s taking so much memory?

  10. will

    Great research. This will save people many hours of confusion.

  11. Thanks for the article. I was actually thinking on how cheap micro instance was specially with 3yr plans. I’m currently using fibervolt.com
    They are the cheapest vps I found. Its been reliable so far.

  12. Hi Greg:

    You can overcome Amazon’s CPU throttling by throttling your applications on your own, with a tool such as cpulimit (aptitude install cpulimit). That way you can prevent processes from ever reaching the “cap” in which Amazon’s throttling kicks in, resulting in a severely incapacitated server.

    At least that’s what I did for my Micro-instance, and it has proven to work out great :)

    Kind regards
    Poul Wittig

    • Hey Poul… This is a great idea. I’ll try it and do some testing. Maybe a good “formula” can be found to make the micro instances more useful.

      Thanks!

      Greg

  13. Although keep in mind it is more of a hack and is not a fail-free solution, since you obviously can’t limit every single process on your server (in my case it was only teamspeak3, mysql, & apache processes I chose to limit).

    As an example, I came across my server still weekly entering that incapacitated state, despite having limited the processes which I thought were causing the Amazon CPU throttle. It turns out a default Ubuntu has a weekly cron by the name man-db, which will would (afaik) re-index available packages in the sources which spikes the CPU for a long amount of time. This job caused my server to enter that “incapacitated” state, and I’ve had to force a reboot on the instance since even after 10 hours it didn’t recover. My solution was to “sudo aptitude purge apt-xapian-index”, and have since then not experienced any Amazon-throttling occurring.

  14. Oh! Forgot to mention CloudFlare! I started using it recently, and it has saved my server many GB’s of traffic each month, by filtering out botnet’s and other bad stuff, but also by providing some additional cache (even CDN’ing it to speed my blog up for visitors far away from US-east) . A nice addition to remove some strain from your server (and it’s actually free).

    Haha not affiliated with them btw, just really impresssed so far :)

    • CloudFlare looks very interesting. I do block some things with .htaccess rules (mod rewrite) but there’s looks more comprehensive.

  15. Alex Athanasopoulos

    I ran into this problem when I typed “sudo apt-get install ec2-api-tools” on an ubuntu micro instance. Ironic, isn’t it? I wanted to try out the EC2 tools but the installer hangs on the micro instance. I ended up killing (stopping) the instance twice. In order to go past the install, I converted the instance to Small, did the install, and then converted back to Micro It was really fast. You can change the instance type from the AWS console and simply restart the instance.

    • Yeah – good plan. I had to do a full ColdFusion install on a micro instance so I converted it to a c1.medium (high cpu), did the install then went back to micro. Worked great and maybe cost me 3 cents ;-)

      Greg

  16. Good idea about the CPU throttling. You could also try playing with the settings in your apache2.conf file. For example, you could try limiting the number of client threads that are spawned. That would reduce the number of concurrent clients you can serve but it may also reduce your overall CPU consumption during peaks. In addition, you could also try hosting at least your static files (.css, .js, images, video, audio, static .html files, etc) in S3. There is no hourly charge for S3; you only pay for the storage and bandwidth costs, which should both be negligible. You could even set up CloudFront to serve the static content even faster.

  17. Ben

    Greg, thanks a million for sharing this. I found your video very easy to understand. You probably saved me 37.8 hours of work!

  18. Greg, this was a great article for me. I had been considering moving one of my sites to EC2, but now I’ll try it with a throw away domain name that I can measure and monitor it with.

    Again, thank you so much for the post.

    By the way, where are you hosting this site? Still on EC2? It’s pretty fast.

    Jon

  19. Hayden

    Ahh! Maybe this is why when I use apachebench it slows down A LOT after the first few seconds of requests. Did you upgrade to small instance? or found a work around?

  20. Hayden

    Was listening to video while I posted. lol Just noticed at the end of vid you said you were able to upgrade to small instance with just one command! How is this done? Cant find blog post. Thanks

  21. Adam Burr

    Hi Greg,

    Two thoughts from me…

    1. An alternative solution: The file called robots.txt on your website allows you to throttle back search engines. You can add commands to instruct search engines to pause between requests and not hit your site too hard. Most search engines will obey these instructions.

    2. Do you know if you can upgrade a reserved micro instance to small? What happens about the fact that you have paid an advance reserved fee?

    Regards,

    Adam

  22. Greg,

    Your post helped me to solve a mystery server problem that has been plaguing me for some time. It’s documented on Server Fault (http://serverfault.com/questions/304252/ubuntu-11-04-server-hanging-due-to-landscape-sysinfo-cpu-overconsumption/331757#331757). Thanks for the great detective work, and for taking the time to write up your findings so thoroughly. I wouldn’t have been able to solve the problem without your help.

    Also, boo on Amazon for not documenting this behavior :/

    Chris

  23. Chris

    I just started an t1.micro instance with Windows 2008 R2 64-bit, IIS and a few WordPress 3.3 sites. It’s working fairly well, actually! I don’t have a lot of traffic but generally the CPU is fairly low utilization, but there are occasional spike (probably from me doing something via RDP).

    I have not found anyone readily on the net using this kind of instance with Windows 2008 R2, 64-bit! I’d love to learn tips on how to optimize and secure this setup! Comments?

  24. bluehost does the same thing (limits your cpu to like nothing if you spike). Recommend linode or digital ocean or maybe others over it :)