Docker v/s Rkt Benchmarking: Community & Documentation Support

Introduction

In this article we will talk about the Community support, development, and bug resolving pace and documentation support for Docker v/s Rkt.

Community & Documentation Support

Over the course of three months we got a decent idea about Docker and rkt. This section is dedicated to our experience of using these containers. Firstly, when it comes to the developers community support, Docker wins over rkt. Docker has 1366 number of contributors [21] whereas rkt has 130 [22]. This is due to the fact that Docker has been in the market for over 2 years now, whereas rkt only had its first release in Feb 2016. Secondly, since rkt is fairly new in the market, it is comparably less stable than Docker. In last three months we have faced a few major bugs with the latest rkt version. However Docker is fairly stable and no such bugs surfaced in our use case. Thirdly, from documentation support, Docker is fairly strong with both, detailed official [23][24] and numerous unofficial documents (blogs, articles, guides & tutorials). Hence analysing & resolving issues in Docker takes much less time than it does with rkt.

Docker v/s Rkt Benchmarking: Clustering

Introduction

In this article we will talk about Native cloud platform for running containers on Google Cloud, Amazon, Azure & Kubernetes tool for Docker v/s Rkt.

Clustering

Native Solutions

Without any major surprise Docker takes the lead when it comes to the of-the shelf containerization solutions. Compared to its less popular counterpart Docker as a company puts a lot emphasis on advertising its solution as a full-fledged platform (rather than mere runtime) – essentially striving to become an “AWS of application container world”. Thus, Docker offers its custom, out-of-the-box solutions for simple container provisioning (Docker Machine [13]), automatic multi-container environment deployment (Docker Compose [14]), container cluster orchestration and management (Docker Swarm [15]) and even unified web interface (hosted service) for provisioning and management of resources across several popular cloud service providers (Docker Cloud [16]).

Rkt on the other hand seems to be focusing its attention elsewhere. In fact at the time of this report CoreOs as a company shadowing the project doesn’t envision rkt as anything more than a simple, customizable runtime. As such there exist no native rkt counterparts to above-mentioned Docker solutions.

Turn-key Cloud Providers

Much like in case of native solutions it is only Docker that benefits from 3rd-party cloud integrations. Unsurprisingly, 3 largest and most popular AC clustering solutions – ECS, Docker Swarm, Kubernetes – are offered respectively by 3 largest and most popular cloud providers – Amazon AWS, Microsoft Azure and Google Cloud Platform.

AWS EC2 Container Service is an Amazon service that handles container deployment, management, scheduling and cluster communication. Behind the scenes ECS really only simplifies provisioning and management of EC2 instances on which Docker containers run – combining several other AWS services such as EC2, CloudFormation, EBS, Auto Scaling, ELB and CloudWatch under one hood – one UI. As ECS really is just an abstraction effectively wrapping other AWS services users only pay for the actual underlying resources (typically only EC2).

Azure Container Service, Microsoft Azure primary clustering platform, is the youngest of all 3 services (until recently only preview release version). Unlike AWS, Azure didn’t implement its own clustering system but rather integrated native clustering platform from Docker – Docker Swarm. No longer a preview, ACS now provides full support for automatic setup of Docker Swarm (as well as Apache Mesos) via Azure Resource Manager UI, no longer requiring configuration templates [17].

Google Container Engine, much like its Amazon counterpart, features its own container clustering solution managing application containers on Google Compute Engine VMs. It comes as no surprise that GCE is powered by Kubernetes – an open source container cluster manager developed by Google. GCE provides uniform environment both for image storage (private image registry) and cluster provisioning.

Kubernetes

Kubernetes is an open-source system for automating deployment, operations, and scaling of containerized applications. We believe that Kubernetes is currently the most mature container clustering platform on the market with largest company adoption and developer/user community. Additionally, the platform features numerous integrations with both cloud providers and custom on-premise infrastructures. Moreover,based on our research Kubernetes is currently the only clustering technology that supports rkt as an alternative container runtime.

Kubernetes comes as a set of scripts that leverage cloud-specific command-line tools to provision resources and register services. By default cluster kube-up script points to Google Cloud Engine but can easily be customized via environmental variables to use AWS, Azure, Rackspace or others instead. After the cluster gets spawned the majority of orchestration happens through kubectl – command line tool responsible for deployment creation, pod management, scaling+load balancing and monitoring.

With the introduction of Docker to the market Google adapted their clustering platform to use it as a default runtime. This decision came at the cost of additional level of abstraction bridging gap between Kubernetes CLI and discovering services and Docker daemon. However even without direct access through Docker CLI or container definition via Docker Compose (Kubernetes pods require different configuration YAML templates) Kubernetes still manages to make things as simple as possible (essentially 3 commands to create a cluster and provision containerized service). Rkt seemed like a reasonable runtime alternative especially because of its simplicity and essentially the same type of resource isolation (Kubernetes pods = rkt pods) however current state of things still leaves a lot to be desired. For one, Kubernetes master node mainly responsible for service hasn’t been made compatible with rkt yet (only worker nodes are). Additionally, explicit changes to flannel (common networking service) config on every files are required, which obviously negates automatic setup. Finally, at the time of this report rkt can be only used to run Docker images (via docker2aci tool), which has interesting conversion implications [18].

Ultimately, rkt setup turned out to be more complicated than we’d foreseen. Strong dependency on CoreOS operating system prevented us from creating the cluster both on GCE (resources were provisioned but cluster setup failed) and AWS (CoreOs was declared as unsupported OS).

Interestingly, even CoreOs’ custom Kubernetes solution, which was the only reliable way of getting the cluster running, currently hard-codes Docker as its primary runtime. It would appear that at the time of this review rkt integration with Kubernetes isn’t the primary focus for either technology (and documentation is simply outdated) [19]. With that being said, rkt promises full integration with Kubernetes by the end of May [20].

Docker v/s Rkt Benchmarking: Performance Benchmarks

Introduction

This article presents the performance benchmarking like Startup Time, Network Performance, Fedora Benchmark, Wikipedia Benchmark for Docker v/s Rkt.

Performance Benchmarking

StartUp Time

Our first benchmark models simple container startup speed as ACs proud themselves with their speed of deployment (when compared to VMs). To eliminate external influences on the benchmark we chose one common image for both container technologies – busybox:latest. Our benchmark then consists of sequential startup of 20 containers running busybox that print a message to stdout (echo) exiting immediately.

Network Benchmark

The main aim of running network benchmarks is to evaluate any networking overheads imposed by running applications within containers. We used IPerf3 [6], an industry standard networking benchmark to evaluate network traffic throughput. Our setup included containers running IPerf servers on one host machine connected and IPerf clients running on another host machine. These two machines were connected in a private network via a switch. We generated traffic between clients and servers to capture the network throughput. To generate a real world scenario we had multiple containers running IPerf server. The graph below shows network bandwidth per client as we increased the number of containers serving the requested traffic. Both Docker and Rocket performed close to each.

Distributed Processing – Wikipedia Benchmarking

To compare the performance between Docker and rkt over a distributed environment for realistic usage, we performed Wikipedia benchmarking. This involved setting up a Wikipedia Containerized Mirror (Apache Web Server [7], MediaWiki [8] & MySQL [9]) and generating many concurrent HTTP requests to evaluate its performance with metrics like latency, throughput and number of errors received. This had two benefits. Firstly, Wikipedia is a realistic application that is used by millions of people everyday. Hence the benchmarking results are actually applicable for real life usage. Secondly, the load distribution of Wikipedia could be prepared accurately because of the huge availability of Wikipedia statistics. Our benchmarking setup involved two configurations – Single Node Four Containers & Four Node Two Containers, as shown below. In the first configuration we had 4 containerized Wikipedia Servers deployed on a single machine, backed by one common MySQL database container. In the second configuration we had 4 containerized Wikipedia Servers deployed over 4 different machines each having one MySQL container for the Wikipedia data.  In both the setups, the load generator sent HTTP request to an NGinx load balancing server [10] which further distributed these requests to the Wikipedia containers.

Single Node Four Containers	Four Node Two Containers

We used Yahoo Cloud Service Benchmark 4 Webservices [11] to generate HTTP workload, which was prepared by analysing the Wikipedia statistics. Both reads and writes had a Zipf distribution with a Zipf’s constant value of 0.6175 & 0.6391 respectively. To perform a fair benchmarking, we increased the workload gradually by 25 Requests Per Second (RPS), starting from 25 RPS upto 300 RPS, with 60k operations on each step. We did 3 iterations of these benchmarks for both configurations, with each iteration running upto 3 hours. CPU and RAM were monitored throughout the process. The results for average latency, throughput and number of errors (5xx HTTP code) were captured for performance comparison. The graphs for Average Latency versus RPS are shown below for both the configurations.

The performance of Docker and rkt were pretty comparable in terms of Average Latency, throughput, CPU and RAM usage. Both the containers supported 200 RPS for the first, and 150 RPS in the second configuration. After this point, the number of request exceeding 10 seconds of execution rise rapidly, since the process was CPU bound. We also ran both these configurations with 2 instead of 4 containers, and the results were still comparable. From these numbers we believe that there is no clear winner when it comes to handling the load in realistic distributed server side applications between rkt or Docker. The graphs for CPU & RAM usage, Zipfs constant calculation, benchmarking results with 2 containers are all shown in the Appendix.

Distributed Processing (Fedora Benchmark)

First of our benchmarks for distributed container setup Fedora Benchmark’s main purpose is to evaluate different strategies of work partitioning for various distributed processing workflows. All the benchmark experiments have common denominator – Fedora Repository – popular open-source digital repository platform built on top of Red Hat ModeShape project. The benchmark itself features 3 distinct scenarios modeling 3 different, real-world use cases. Additionally, the benchmark does not generate synthetic data but rather relies on collected sensor data obtained from sensor- and accelerometer-outfitted Goodwin Hall. This raw data is stored in 180 HDF5 files of roughly equal size – 50MB.

We ran Fedora benchmark on Chameleon Cloud service – an experimental environment for large-scale cloud research [12]. Chameleon allows researchers to either provision VMs using its hosted OpenStack KVM or to lease a bare metal high-compute node. For the purposes of the benchmark we leveraged both options – provisioning separate OpenStack VM to run RabbitMQ server, separate VM to run Fedora machine and leasing 4 bare metal machines to host our container workers. For detailed overview of resource setup as well as environment specifications please consult Figure 1. in Appendix.

Although the full benchmark constitutes of 3 separate workflows emulating 3 different types of data processing – Ingestion (bulk load of HDF5 files into Fedora), Fixity checking (SHA1 checksum comparison) and FFT (highly parallelizable heavy computation) – we decided to only include our results from the last experiment (FFT) as we find it to be the most representative scenario of distributed Mapreduce-like processing. FFT leverages the fact that our stored Fedora objects contain sensor (i.e. signal) data. FFT experiment thus applies fast Fourier forward algorithm (O(nlogn)) to convert said signal data from its temporal domain to its frequency domain.

The execution itself is single-threaded promising predictable linear scaling.

We ran Fedora benchmark on both AC runtimes while incrementing the number of containers (and machines) participating in the FFT experiment. Above results correspond to the following allocation of compute resources: 1 machine/1 container, 2 machines/1 container each, 2 machines/2 containers each, 4 machine/2 containers each, 4 machines/4 containers each, 4 machines/8 containers each, 4 machines/16 containers each and 4 machines/16 containers each. Our observed results indeed imply linear performance scaling with doubling of resources between consecutive runs with breaking point between 32 and 64 containers. Both Docker and rkt appear to have comparable performance. To address the breaking point we make use of our per-file average results (figure below).

Our average-per-file result graph plots average duration spent by 1 container while processing 1 HDF5 record. Download value corresponds to object extraction while processing refers to actual FFT execution. Average times don’t appear to be varying until the limit of 64 containers is reached (performance deteriorates), which, based on our resource monitoring, is caused by excessive CPU context switching (worker nodes only possess 48 cores). Judging from the results, performance-wise rkt seems to be more susceptible to this phenomenon.

Docker v/s Rkt Benchmarking: Feature Support

Introduction

This article compares the feature support like Network Types, Volume Mounting, Image Building, Registry Availability, Dynamic Image Creation, Image Distribution, Interactive, Native daemon service. Stop/Restart for Docker v/s rkt.

Feature Support

#	Feature	Docker	rkt
1.	Runs Docker images	Yes	Yes
2.	Runs App Container images	No	Yes
3.	Architecture	Client-Server	Standalone
4.	Image Distribution via	Centralized Docker Hub	Decentralized over HTTP
5.	Supporting Operating Systems	Linux (with package manager), Mac (via Docker Machine), Windows (only 2016 Server)	Linux (without package manager)
6.	Default enforced Image Signing	No	Yes
7.	Pluggable Isolation	Not supported	Supports pluggable isolation due to 3 staged startup. Fly is a Stage 1 ACI example.
8.	Image Building	Using Dockerfile	With manifest files using actool, goaci, acibuild and docker2aci (for docker images)
9.	In Place Updates	No. Updates requires containers restart.	Yes. Even with Containers

Table 1. Docker v/s rkt feature comparison

Docker v/s Rkt Benchmarking: Lifecycle

Introduction

In this article we talk about the Startup & the Lifecycle of Docker v/s Rkt.

Lifecycle

There are a few differences from Life Cycle point of view between rkt and Docker. Firstly, rkt has linear execution with 4 phases of life cycle [5] – Prepared, Run, Exited Garbage and Garbage. This means that once a container goes to Garbage it can’t be restarted. Docker has a cyclic life cycle which means that a stopped container can be restarted. Secondly, a container in rkt can’t be explicitly stopped but a Docker container can be. This is a conscious decision by the Core OS team, since they believe that containers are a small lightweight execution unit which should be instantiated everytime before use. Thirdly, Docker can perform automatic garbage collection, which means that onces a container is exited, it will be automatically be garbage collected and disk space will be freed. However to free disk space taken by a rkt container, we must explicitly issue a gc command.

Docker v/s Rkt Benchmarking: Architecture

Introduction

In this article we talk about terminology and architecture for Docker v/s Rkt.

Architecture

From architectural point of view there are a few basic differences between Docker and rkt. Firstly, Docker runs as a centralized daemon with a Client-Server model. Hence all the application containers rely on its continuous support throughout their runtime. Rkt on the other hand follows a standalone model, where all application containers once initiated are self reliant and do not communicate with any background rkt service. Secondly, in Rkt the basic unit of execution are Pods, which are a collection of application containers. All containers in a Pod share the same context in terms of the networking & volume configurations. In Docker the basic unit of execution are Application Containers itself and each container must be initiated by explicitly stating their configurations even if they share their configurations with other containers.

Docker v/s Rkt Benchmarking: Ideology & Motivation

Introduction

In this article we talk about the ideology and motivations like specification standardization and security concerns for Docker v/s Rkt.

Ideology & Motivation

Docker is an application container execution provider and a platform, since it provides numerous utility services compiled into one monolithic binary running primarily as root on a server. Rkt is an implementation of the App Container spec [3], a new set of simple and open specifications for a portable container format, by the development team at CoreOS. There are two major motivational differences behind Docker & rkt. Firstly, rkt follows the Unix philosophy and aims to device the idea of a “standard container” with a simple component, a composable unit, that could be used in a variety of systems. Hence with rkt supports the idea of easy multiple implementations of the App Specification. Docker on the other hand has tightly coupled components without any standard interface thereby making it difficult to customize and have multiple implementations. Secondly, in the Docker process model everything runs through a central daemon with root privileges. Rkt developers believe that this is a fundamental security flaw. Hence to enhance security, every instance of a running container in rkt is given a unique identity, coupled with a lightweight HSM-like service for signing. No existing VM or container environment has a concept like this.

Docker v/s Rkt Benchmarking: To sail or to fly?

Introduction

Most of the cloud computing setup nowadays revolves around provisioning of virtual machines, setting up network traffic rules, storage allocation and access filtering (security). All these steps present a large overhead for the user especially in the SOA type of applications. Application Containers (AC) are among alternatives to standard virtualized instances based on the ideas of ultra-fast deployment, flexible portability and superior ease of use. Currently, Docker [1] is the predominant application container with hundreds of companies making use of it and thousands of developers actively participating in its development. However, Docker is no longer the only player on the market. In 2014 CoreOs announced rkt (Rocket) [2], its own implementation of AC, with main focus on the improvement of security (a major issue in the container world) and image distribution, and promotion of container format and runtime standardization.

Motivation

Although there are few tech blogs and news articles stating similarities and differences between Docker and Rocket, these articles tend to neglect evaluation from several important perspectives. Firstly, they do not clearly state a comprehensive list of functionalities supported in these technologies. Secondly, they do not present any quantitative evaluation of performance for both ACs. We feel that a comparative study that provides a quantitative and qualitative assessment of supported functionality, performance and container clustering would be beneficial both to the software developers and DevOps. To the best of our knowledge, we are not aware of any such study.

Questions

What are the performance implications on Network, CPU and IO when running Docker versus rkt? How do these ACs behave in distributed big data processing? How do distributed web applications perform when containerized?

Which of the two ACs should be preferred for software development, for a given set of requirements? To avoid a vendor lock-in, which of the two ACs must be chosen and why? Performing customizations and implementing additional support is easier in which one of the two ACs?

How do these technologies fare in distributed environments? What is the overall support by popular cloud providers? What solutions are available for container clustering and scheduling?

Evaluation Metrics

To perform a fair evaluation of the two ACs, we have decided to create an evaluation metrics shown below. From a list of long metrics that we initially drafted, we have included only a few important and the most relevant metric parameters to our best knowledge.

#	Criteria	Description	Links
1.	Ideology & Motivation	Specification standardization and security concerns.	Ideology & Motivation
2.	Architecture	Terminology and architecture.	Architecture
3.	Lifecycle	Startup & Lifecycle of a container.	Lifecycle
4.	Feature Support	Network Types, Volume Mounting, Image Building, Registry Availability, Dynamic Image Creation, Image Distribution, Interactive, Native daemon service. Stop/Restart	Feature Support
5.	Performance Benchmarks	Startup Time, Network Performance, Fedora Benchmark, Wikipedia Benchmark	Performance Benchmarks
6.	Clustering	Native cloud platform for running containers on Google Cloud, Amazon, Azure & Kubernetes tool.	Clustering
7.	Community & Documentation Support	Community support, development, and bug resolving pace and documentation support.	Community & Documentation Support

Conclusion

Based on our experience with both ACs, evaluation of their features, benchmarking results and 3rd party support Docker presents itself as a more mature, stable and efficient solution out of both technologies – both for individual developers interested in quick environment setup, and devops focusing on resilient clustering infrastructure. We believe that the major benefit of Docker is its user friendliness. The company keeps investing a lot of effort into gradual elimination of learning curve by maintaining vast and concise documentation, automating setups and configuration and abstracting as much of the underlying logic as possible. Based on our understanding Docker essentially wishes to make containerization technology easily available to anyone (regardless of knowledge) and anywhere (by building a dedicated platform around the product + integrating with external cloud providers). At the time of this report rkt is still a relatively young product (its first version was released in February 2016) yet we think that it is bound to find wider application with the upcoming releases. Although less straightforward, rkt’s design promotes simplicity and clear separation of concerns, and also offers a higher degree of customization and control over execution stages. Additionally, long-term, its architecture (pod) makes rkt an ideal candidate to replace Docker as a main execution environment and greatly simplify Kubernetes. All-in-all, rkt definitely deserves its attention from AC user community.

References

1. Docker: https://www.docker.com/what-docker
2. Rocket: https://coreos.com/blog/rocket/
3. https://github.com/appc/spec/tree/master/spec
4. https://docs.docker.com/engine/understanding-docker/
5. https://coreos.com/rkt/docs/latest/devel/pod-lifecycle.html
6. iPerf: https://sourceforge.net/projects/iperf/
7. Apache Web Server: https://httpd.apache.org/
8. https://www.mediawiki.org/wiki/MediaWiki
9. https://www.mysql.com/
10. https://www.nginx.com/resources/admin-guide/load-balancer/
11. https://github.com/shivam-maharshi/YCSB4WebServices
12. https://www.chameleoncloud.org/
13. https://docs.docker.com/machine/overview/
14. https://docs.docker.com/compose/overview/
15. https://docs.docker.com/swarm/overview/
16. https://docs.docker.com/docker-cloud/getting-started/intro_cloud/
17. https://github.com/Azure/azure-quickstart-templates/tree/master/101-acs-swarm
18. http://kubernetes.io/docs/getting-started-guides/rkt/notes/
19. https://groups.google.com/forum/#!topic/coreos-user/g2hKE921BJw
20. https://github.com/coreos/rkt/blob/master/ROADMAP.md
21. https://github.com/docker/docker/graphs/contributors
22. https://github.com/coreos/rkt/graphs/contributors
23. https://docs.docker.com/
24. https://coreos.com/rkt/docs/latest/

Credits

The research done for this evaluation was conducted by me in collaboration with my friend Andrej Galad.

Benchmarking Tools Evaluation for Non Cloud Based Web Services

Introduction

This article is about my evaluation of various famous web services bench marking tools. In this post I’ve mainly focused on free or open source bench marking tools for Non Cloud (NC) based web systems. Here I present my evaluation of mainly these 6 famous tools – Apache Benchmark, Apache JMeter, Smart Meter, Httperf, Tsung, Http Load and a little bit about other 23 proprietary or paid tools. I will focus on bench marking tools for Cloud Based (CB) web systems in my next post. Some of those tools will be PerfKit/Perf Explorer, AutoBench and Yahoo Cloud System Benchmark.

Difference between bench marking tools for CB & NC Web Systems ?

A NC (non-cloud) web system could serve similar to a CB web system functionality wise but the biggest difference lies between the maximum load these systems can bear. For ex, a simple web service deployed on just a couple of web servers hiding behind a single load balancer might handle a few thousand requests per seconds however CB web systems can tolerate a load much more than that ! Due to this basic difference the bench marking tools that are used for NC based web systems have to generate lower loads which a single or two client machines are capable of generating. However to benchmark CB web systems we generally need to generate ten to hundred of thousands of requests that a single client machine is incapable of generating. Since we need many client machines for CB web systems many things like synchronization between the client machines, distribution of responsibility, distribution of resources like visit trace etc. comes into picture. These added functions required for bench marking makes CB web systems trickier and non trivial to benchmark. Hence there tools are more sophisticated. On the other hand bench marking tool for NC web systems do not have to support these functionality and hence they are little less sophisticated and pretty straight forward to use. Please do not be mistaken by my previous statement into thinking that bench marking a NC based web service itself is easy. On the contrary, generating accurate benchmarking results are far from trivial. It requires an in-depth understanding of Networking, Operating Systems, Web Servers etc.

Evaluated Benchmarking Tools

This section deals with the evaluation of the 6 aforementioned benchmarking tools for web services.

1. Apache Benchmark

A very famous command line bench marking tool by Apache. Pretty widely used.

Pros:

1. Runs on a single thread on the client machine, thereby not bogging the client machine down and skewing the results of bench marking.
2. Regularly maintained. New and improved releases every now and then.
3. Very easy to generate requests. Commands are pretty simple to use.
4. Clear and precise output. Very easy to understand.
5. From the famous Apache community this is a reliable tool. We all trust Apache in catering quality products.

Cons:

1. Does not offer a build in graphical visualization tool. Relies on your knowledge of data viz tools like GNU Plot or Excel to generate charts that others can make sense of.
2. Does not support complex operations like interaction with database as pre or post condition while running the tests.

Sample Request:

ab -n 100 -c 10 -g C:\Users\Sam\Downloads\out.tsv http://google.com/

GNU Generated Plot:

Command: plot “force.dat” using 1:2 title ‘Column’, \”force.dat” using 1:3 title ‘Beam’

Useful Links:

1. Apache Benchmark and GNU Plot: You are probably doing it wrong
2. GNU Plot

2. Apache JMeter

Another widely used GUI based bench marking tool by Apache. It is well known for its range of powerful functionality and ease of use.

Pros:

1. Capable of generating high loads. Since it uses multiple threads on the client machine, this tool is capable of generating pretty high loads.
2. Provides a GUI which makes it easier to use.
3. Provides a very good graphical visualization of response set in variety of ways.
4. A very flexible tool that supports complex requests like HTTP, FTP, DB queries, JMS etc. Kind of like a single ring to rule them all 🙂
5. Regularly maintained. Frequent new and improved releases.
6. Relatively easy to generate requests given the flexibility it provides.
7. Clear and precise output. Very easy to understand.
8. Since it is from Apache community pretty trust worthy.

Cons:

1. Does not runs on a single thread on the client machine, there by exposing a user to bogging down his client machine and skewing the results of bench marking. Users should pay specific attention to the CPU and memory usage of their client machine for accurate evaluation.
2. Could be a little time consuming to learn in the start due to so many functionalities.

Sample Request:

Response Data Visualization:

Useful Links:

1. JMeter – Web Test Plan Tutorial

3. Smart Meter

This tool is an extension of JMeter with many additional features and a support for high concurrency. For example it provides many new thread patterns, samplers & listeners. It has recording functionality for the benchmarking play back as well. This is not a free tool but it can be used up to 500 concurrent threads in its free version. Which can be sometimes enough to test a web application on a single server. It has most of the benefits of Apache JMeter since it is an extension of it but a few additional ones are.

Pros:

1. In paid version, it is capable of generating higher loads than JMeter.
2. Provides a new set of sampler like Lazy incremental step sampler that are really good to replicate real life scenarios and perform accurate bench marking.
3. Provides new set of listeners that can analyze and present the response data in more variety of way than Apache JMeter.
4. Additional samplers are pretty configurable hence giving powerful request sampling capabilities.
5. Seems reliable, a lot of firms are using this tool.

Cons:

1. Can only be used to simulate 500 concurrent users in its free version.
2. Could be a little time consuming to learn in the start due to so many functionalities.

Sample Request Pattern:

Lazy Step Sampler

Response Data Visualization:

Useful Links:

1. Download Smart Meter

4. Httperf

This is a command line tool from HP labs. It was pretty frequently used a few years back. It is not maintained anymore. The focus of httperf was on providing a robust, high-performance tool that facilitates the construction of both micro- and macro-level benchmarks.

Pros:

1. It is a single thread event based program hence doesn’t bog down the client’s machine thereby affecting the bench marking results.
2. Its commands are very clear and easy to use.

Cons:

1. Not maintained anymore. So you have to add any additional functionality or fix bugs yourself.
2. Has no way to store the response data to a file. Hence you  must parse the console to save the data for analysis.
3. Doesn’t provide a in-built tool to visualize the response data. It relies on the user’s knowledge of data viz tools like GNU Plot or Excel to generate graphs etc.
4. Configurations are not that easy to use.
5. Lack of detailed documentation and support.

Sample Request:

httperf –server hostname \ –port 80 –uri /test.html \ –rate 150 –num-conn 27000 \ –num-call 1 –timeout 5

Console Output:

Useful Links:

1. Httperf Download
2. Httperf Tutorial

5. Tsung

It is an open source multi protocol distributed load testing tool. It can be used to stress HTTP, WebDAV, SOAP, PostgreSQL, MySQL, LDAP and Jabber/XMPP servers

Pros:

1. Tsung is very fast and effective.
2. Capable of generating heavy loads since it is written in Erlang, its makes use of the light-weight processes offered by BEAM itself. Doesn’t bog the client machine down.
3. Provides implicit ways of graphical visualization using Tsung stats perl script that uses GNU plot internally.
4. Its graphical visualization is impressive in the variety of ways that the response data is analyzed. Additionally an HTML file is created which can be directly saved into a web server for online access from remote location.
5. It is regularly maintained and updated pretty frequently.
6. Can perform stress testing of HTTP, WebDAV, SOAP, PostgreSQL, MySQL, LDAP and Jabber/XMPP servers.
7. Its a pretty reliable tool.

Cons:

1. Not supported on Windows.
2. Holds the request load generation configuration in an XML file which can get pretty tricky to write.
3. Not very easy to generate real life scenarios requests.

Sample Request:

tsung -f basic.xml start

Response Data Visualization:

Useful Links:

1. Testing Web Applications Using Tsung
2. Tsung Tutorial

6. Http Load

It is an multiprocessing HTTP command line test client mainly created to test web servers. It reads a list of URL to hit from an input file. It is one of those tools that you can use at any instant for quick bench marking.

Pros:

1. It is very light weight since it runs on a single thread on the client machine hence doesn’t bog it down.
2. Very straight forward and easy to use.
3. Data shown in the console is limited but easy to understand.

Cons:

1. Supports only fetch operations, thereby limiting its application.
2. Doesn’t provide an inbuilt data visualization tool that can be used to generate graphical reports etc.
3. Doesn’t support export of the data to an output file for visualization purpose. You need to parse the console for data retrieval.
4. Cannot support flexible operations like communicating with database as part of pre or post conditions while running the test.
5. Not maintained anymore.

Sample Request:

./http_load -rate 5 -seconds 10 urls_file

Response Data:

Useful Links:

1. Http Load Download

More Benchmarking Tools

This section deals with many other famous benchmarking tools available in the market. Below is a list of 23 tools I did not evaluate since either they are paid or limited in functionality and support. But many of them scale up brilliantly and are very flexible, thereby being used by a lot of companies.

1. Open STA

Free. User friendly but only supported on Windows OS.

2. WebLOAD

Not free. Very user friendly and highly scalable.

3. HP LoadRunner

Not free. User friendly. Used for testing for cloud. Capable of generating high loads.

4. IBM Rational Performance Tester

Not free.

5. NeoLoad

Not free. Very user friendly. Can be used unlimited for max 50 concurrent users only in the free version.

6. LoadUI / LoadComplete

Not free. Very user friendly. Free version can be used for max 50 concurrent users only.

7. WAPT

Not free. Only supported for Windows OS.

8. Loadster

Not free. Not updated frequently.

9. LoadImpact

Not free. Very user friendly. Good for cloud based web applications. Capable of generating high concurrent load.

10. Testing Anywhere

Not free. Cloud based benchmarking tool.

11. Appvance

Not free. Decently user friendly.

12. LoadStorm

Not free. Capable of generating very high loads up to 35 K concurrent users.

13. Web Performance Analyzer

Not free. Free version only allows maximum 50 concurrent users.

14. OctoPerf

Not free. Very user friendly.

15. Flood.io

Not free. Capable of generating high concurrent loads. Very good for highly scalable web services and cloud applications.

16. Load Tester

Not free. Very good for high concurrency real life scenarios. Capable of supporting 1,000,000 concurrent users !

17. Web Cat

Free. Only supported on Windows OS. Pretty old.

18. Load Complete

Not free. Decent in usage.

19. Blaze Meter

Not free. Doesn’t support a very high concurrency but works decently.

20.Load Focus

Not free.

21. Raw Load Tester

Free but pretty old. Hence it is not updated and maintained anymore.

22. Test Maker

Not free. Limited time trial. Decently user friendly.

23. Ixora RMS

Free. Pretty user friendly. Mainly a tool for monitoring purpose. Requires configuration changes at server side.

Useful Links

1.http://www.softwaretestinghelp.com/performance-testing-tools-load-testing-tools/

2. http://www.opensourcetesting.org/performance.php

I hope this information helps out people evaluating bench marking tools for Non Cloud based Web Systems. In the next post I’ll present my evaluation for bench marking tools that support performance testing of Cloud Based Web Systems like PerfKit / Perf Explorer, AutoBench and Yahoo Cloud System Benchmark etc. Thanks for reading !!

Probabilistic Graphical Models – Bayesian Networks using Netica Tool for Java

This article is about my experience in learning Bayesian Networks and its application to real life data via a tutorial. It took me sometime to understand the theory behind Probabilistic Graphical Model and then to utilize that theory for the application to real life data. For beginners this is a perfect tutorial to learn about Bayesian Networks via implementing. Before going ahead I would recommend you to read the basics of Bayesian Networks from the Wikipedia page here: https://en.wikipedia.org/wiki/Bayesian_network

Terminology:

1. Structure: The representation of labelled nodes and the dependency relationship between them, without the exact probability value.
2. Network: A structure with dependency defined for all nodes in terms of Conditional Probabilistic Tables. Here the exact values of conditional probabilities are defined. Structure + CPT = Network.

Cutting right down to chase, below are the Specific Details:

1. Aim: To learn about Bayesian Networks practically and witness the results first hand using real world tools.

2. Approach Follow: The approach I followed consists of five logically incremental steps. Firstly I assume a basic structure of a network. Secondly, I populated the Conditional Probability Tables of the assumed network with some data/beliefs. Thirdly, I ran queries to find out beliefs of some scenarios using inference on our network built in the aforementioned steps. Fourthly, I generate data set using our network. Finally, I use the data generated in the fourth step and populate Conditional Probability Tables of our network, using only its structure. When we compare these new Conditional Probability Tables computed in step five with our previously assumed CPT, they turn out to be very similar. Hence proving our model to be pretty consistent.

3. Tools: We used Netica tool for Java called NeticaJ. Netica is also available in many other languages. the tutorial for Netica is available here – NeticaJ Tutorial.

4. Conclusion: We learn how to create Bayesian Networks and run queries to find beliefs using inference via the existing tools / libraries.

5. Sample Java Code Using NeticaJ:

Bayesian Network

Presented above is the Bayesian Network that we assume initially and intend to study. Below are the glimpse of the Java code I wrote, which is similar to the tutorial, with slight modifications for better terminology and code re-usability. You can view it at GitHub here: https://github.com/shivam-maharshi/tech-tuts/tree/master/bayesian-networks

1. Network Builder: This class has two responsibilities. Firstly, it builds a Bayesian network. Secondly, it persistently stores that network by writing onto a file.

2. Inference Worker:  This class is responsible for calculating beliefs for events from the constructed Bayesian network.

3. Case Simulator: This class is responsible for simulating random cases using the probability distribution given by our Bayesian network.

4. Model Generator:  This class is responsible for generating a model i.e. building Conditional. Probability Tables for a given a network structure using a data set.

At the time of writing this post I have little knowledge of Python and hence Java was used. Since Python has much better libraries and tools for Probabilistic Graphical Models, in future I plan to evaluate a few famous Python tools.

I sincerely hope this consolidated and organized information helps newbies in understanding a very Basic Bayesian Network and its application without having to spend too much time searching for tutorials at various places. Thanks for reading !!