Benchmarking Tools Evaluation for Non Cloud Based Web Services

Introduction

This article is about my evaluation of various famous web services bench marking tools. In this post I’ve mainly focused on free or open source bench marking tools for Non Cloud (NC) based web systems. Here I present my evaluation of mainly these 6 famous tools – Apache Benchmark, Apache JMeter, Smart Meter, Httperf, Tsung, Http Load and a little bit about other 23 proprietary or paid tools. I will focus on bench marking tools for Cloud Based (CB) web systems in my next post. Some of those tools will be PerfKit/Perf Explorer, AutoBench and Yahoo Cloud System Benchmark.

Difference between bench marking tools for CB & NC Web Systems ?

A NC (non-cloud) web system could serve similar to a CB web system functionality wise but the biggest difference lies between the maximum load these systems can bear. For ex, a simple web service deployed on just a couple of web servers hiding behind a single load balancer might handle a few thousand requests per seconds however CB web systems can tolerate a load much more than that ! Due to this basic difference the bench marking tools that are used for NC based web systems have to generate lower loads which a single or two client machines are capable of generating. However to benchmark CB web systems we generally need to generate ten to hundred of thousands of requests that a single client machine is incapable of generating. Since we need many client machines for CB web systems many things like synchronization between the client machines, distribution of responsibility, distribution of resources like visit trace etc. comes into picture. These added functions required for bench marking makes CB web systems trickier and non trivial to benchmark. Hence there tools are more sophisticated. On the other hand bench marking tool for NC web systems do not have to support these functionality and hence they are little less sophisticated and pretty straight forward to use. Please do not be mistaken by my previous statement into thinking that bench marking a NC based web service itself is easy. On the contrary, generating accurate benchmarking results are far from trivial. It requires an in-depth understanding of Networking, Operating Systems, Web Servers etc.

Evaluated Benchmarking Tools

This section deals with the evaluation of the 6 aforementioned benchmarking tools for web services.

1. Apache Benchmark

A very famous command line bench marking tool by Apache. Pretty widely used.

Pros:

  1. Runs on a single thread on the client machine, thereby not bogging the client machine down and skewing the results of bench marking.
  2. Regularly maintained. New and improved releases every now and then.
  3. Very easy to generate requests. Commands are pretty simple to use.
  4. Clear and precise output. Very easy to understand.
  5. From the famous Apache community this is a reliable tool. We all trust Apache in catering quality products.

Cons:

  1. Does not offer a build in graphical visualization tool. Relies on your knowledge of data viz tools like GNU Plot or Excel to generate charts that others can make sense of.
  2. Does not support complex operations like interaction with database as pre or post condition while running the tests.

Sample Request:

ab -n 100 -c 10 -g C:\Users\Sam\Downloads\out.tsv http://google.com/

GNU Generated Plot:

Command: plot “force.dat” using 1:2 title ‘Column’, \”force.dat” using 1:3 title ‘Beam’

AB

Useful Links:

  1. Apache Benchmark and GNU Plot: You are probably doing it wrong
  2. GNU Plot

2. Apache JMeter

Another widely used GUI based bench marking tool by Apache. It is well known for its range of powerful functionality and ease of use.

Pros:

  1. Capable of generating high loads. Since it uses multiple threads on the client machine, this tool is capable of generating pretty high loads.
  2. Provides a GUI which makes it easier to use.
  3. Provides a very good graphical visualization of response set in variety of ways.
  4. A very flexible tool that supports complex requests like HTTP, FTP, DB queries, JMS etc. Kind of like a single ring to rule them all 🙂
  5. Regularly maintained. Frequent new and improved releases.
  6. Relatively easy to generate requests given the flexibility it provides.
  7. Clear and precise output. Very easy to understand.
  8. Since it is from Apache community pretty trust worthy.

Cons:

  1. Does not runs on a single thread on the client machine, there by exposing a user to bogging down his client machine and skewing the results of bench marking. Users should pay specific attention to the CPU and memory usage of their client machine for accurate evaluation.
  2. Could be a little time consuming to learn in the start due to so many functionalities.

Sample Request:

http-request

Response Data Visualization:

jeme

Useful Links:

  1. JMeter – Web Test Plan Tutorial

3. Smart Meter

This tool is an extension of JMeter with many additional features and a support for high concurrency. For example it provides many new thread patterns, samplers & listeners. It has recording functionality for the benchmarking play back as well. This is not a free tool but it can be used up to 500 concurrent threads in its free version. Which can be sometimes enough to test a web application on a single server. It has most of the benefits of Apache JMeter since it is an extension of it but a few additional ones are.

Pros:

  1. In paid version, it is capable of generating higher loads than JMeter.
  2. Provides a new set of sampler like Lazy incremental step sampler that are really good to replicate real life scenarios and perform accurate bench marking.
  3. Provides new set of listeners that can analyze and present the response data in more variety of way than Apache JMeter.
  4. Additional samplers are pretty configurable hence giving powerful request sampling capabilities.
  5. Seems reliable, a lot of firms are using this tool.

Cons:

  1. Can only be used to simulate 500 concurrent users in its free version.
  2. Could be a little time consuming to learn in the start due to so many functionalities.

Sample Request Pattern:

Lazy Step Sampler

smart

Response Data Visualization:

smartoutput

Useful Links:

  1. Download Smart Meter

4. Httperf

This is a command line tool from HP labs. It was pretty frequently used a few years back. It is not maintained anymore. The focus of httperf was on providing a robust, high-performance tool that facilitates the construction of both micro- and macro-level benchmarks.

Pros:

  1. It is a single thread event based program hence doesn’t bog down the client’s machine thereby affecting the bench marking results.
  2. Its commands are very clear and easy to use.

Cons:

  1. Not maintained anymore. So you have to add any additional functionality or fix bugs yourself.
  2. Has no way to store the response data to a file. Hence you  must parse the console to save the data for analysis.
  3. Doesn’t provide a in-built tool to visualize the response data. It relies on the user’s knowledge of data viz tools like GNU Plot or Excel to generate graphs etc.
  4. Configurations are not that easy to use.
  5. Lack of detailed documentation and support.

Sample Request:

httperf –server hostname \ –port 80 –uri /test.html \ –rate 150 –num-conn 27000 \ –num-call 1 –timeout 5

Console Output:

Httperf

Useful Links:

  1. Httperf Download
  2. Httperf Tutorial

5. Tsung

It is an open source multi protocol distributed load testing tool. It can be used to stress HTTP, WebDAV, SOAP, PostgreSQL, MySQL, LDAP and Jabber/XMPP servers

Pros:

  1. Tsung is very fast and effective.
  2. Capable of generating heavy loads since it is written in Erlang, its makes use of the light-weight processes offered by BEAM itself. Doesn’t bog the client machine down.
  3. Provides implicit ways of graphical visualization using Tsung stats perl script that uses GNU plot internally.
  4. Its graphical visualization is impressive in the variety of ways that the response data is analyzed. Additionally an HTML file is created which can be directly saved into a web server for online access from remote location.
  5. It is regularly maintained and updated pretty frequently.
  6. Can perform stress testing of HTTP, WebDAV, SOAP, PostgreSQL, MySQL, LDAP and Jabber/XMPP servers.
  7. Its a pretty reliable tool.

Cons:

  1. Not supported on Windows.
  2. Holds the request load generation configuration in an XML file which can get pretty tricky to write.
  3. Not very easy to generate real life scenarios requests.

Sample Request:

tsung -f basic.xml start

Response Data Visualization:

Useful Links:

  1. Testing Web Applications Using Tsung
  2. Tsung Tutorial

6. Http Load

It is an multiprocessing HTTP command line test client mainly created to test web servers. It reads a list of URL to hit from an input file. It is one of those tools that you can use at any instant for quick bench marking.

Pros:

  1. It is very light weight since it runs on a single thread on the client machine hence doesn’t bog it down.
  2. Very straight forward and easy to use.
  3. Data shown in the console is limited but easy to understand.

Cons:

  1. Supports only fetch operations, thereby limiting its application.
  2. Doesn’t provide an inbuilt data visualization tool that can be used to generate graphical reports etc.
  3. Doesn’t support export of the data to an output file for visualization purpose. You need to parse the console for data retrieval.
  4. Cannot support flexible operations like communicating with database as part of pre or post conditions while running the test.
  5. Not maintained anymore.

Sample Request:

./http_load -rate 5 -seconds 10 urls_file

Response Data:

httpload

Useful Links:

  1. Http Load Download

More Benchmarking Tools

This section deals with many other famous benchmarking tools available in the market. Below is a list of 23 tools I did not evaluate since either they are paid or limited in functionality and support. But many of them scale up brilliantly and are very flexible, thereby being used by a lot of companies.

1. Open STA

Free. User friendly but only supported on Windows OS.

2. WebLOAD

Not free. Very user friendly and highly scalable.

3. HP LoadRunner

Not free. User friendly. Used for testing for cloud. Capable of generating high loads.

4. IBM Rational Performance Tester

Not free.

5. NeoLoad

Not free. Very user friendly. Can be used unlimited for max 50 concurrent users only in the free version.

6. LoadUI / LoadComplete

Not free. Very user friendly. Free version can be used for max 50 concurrent users only.

7. WAPT

Not free. Only supported for Windows OS.

8. Loadster

Not free. Not updated frequently.

9. LoadImpact

Not free. Very user friendly. Good for cloud based web applications. Capable of generating high concurrent load.

10. Testing Anywhere

Not free. Cloud based benchmarking tool.

11. Appvance

Not free. Decently user friendly.

12. LoadStorm

Not free. Capable of generating very high loads up to 35 K concurrent users.

13. Web Performance Analyzer

Not free. Free version only allows maximum 50 concurrent users.

14. OctoPerf

Not free. Very user friendly.

15. Flood.io

Not free. Capable of generating high concurrent loads. Very good for highly scalable web services and cloud applications.

16. Load Tester

Not free. Very good for high concurrency real life scenarios. Capable of supporting 1,000,000 concurrent users !

17. Web Cat

Free. Only supported on Windows OS. Pretty old.

18. Load Complete

Not free. Decent in usage.

19. Blaze Meter

Not free. Doesn’t support a very high concurrency but works decently.

20.Load Focus

Not free.

21. Raw Load Tester

Free but pretty old. Hence it is not updated and maintained anymore.

22. Test Maker

Not free. Limited time trial. Decently user friendly.

23. Ixora RMS

Free. Pretty user friendly. Mainly a tool for monitoring purpose. Requires configuration changes at server side.

Useful Links

1.http://www.softwaretestinghelp.com/performance-testing-tools-load-testing-tools/

2. http://www.opensourcetesting.org/performance.php

I hope this information helps out people evaluating bench marking tools for Non Cloud based Web Systems. In the next post I’ll present my evaluation for bench marking tools that support performance testing of Cloud Based Web Systems like PerfKit / Perf Explorer, AutoBench and Yahoo Cloud System Benchmark etc. Thanks for reading !!

Advertisements

One thought on “Benchmarking Tools Evaluation for Non Cloud Based Web Services

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s