1. Overview

Who this article is for

If you never use JMeter before and want to get an introduction of how to use it.

The following section will show how the process of creating JMeter scripts step by step through a demo case, and some basic necessary configurations of JMeter Demo Case Step by Step.

And the next sections show you how to generate a readable result Generate HTML Report and how to analysis it Result Analysis. We also show how to simulate the big stress byNon-GUI mode and Remote Testing.

Record & Replay is also a shortcut to generate the HTTP requests in JMeter.

We also show a method to merge JMeter into Jenkins JMeter in CI.

At last, we discussing the difference between JMeter and a browser just for your reference Compare to a Real Browser.

Enjoy your testing and good luck!

What is JMeter

The Apache JMeter™ application is open source software, a 100% pure Java application designed to load test functional behavior and measure performance. It was originally designed for testing Web Applications but has since expanded to other test functions.

Compare to other performance tools, JMeter is a lightweight tool with relatively complete support (third party extensions and clear documentation) as an open source software. But it also has weakness such as the not so user-friendly GUI and relatively complex settings. But if you’re googling keyword like “top X performance testing tool” you will always find JMeter in the recommendations.

Anyway, if you’re looking for an “open source”, “lightweight”, “well-supported” and “extensible” performance testing tool,JMeter should be a very important candidate.

Software version:

JMeter:4.0 r1823414

---

2. Demo Case Step by Step

In this section we’ll prepare a JMeter demo script together, hopefully by setup this demo case you have a feeling to the basic JMeter elements and how they work together.

We’ll prepare this case by analysing the HTTP Requests one by one and create them in JMeter, of course you have another chose to record all these requests and then organize them (refer to Record & Replay), depend on your favourite way.

This following JMeter script can be downloaded at:

https://github.com/wzheng011/JMeterDemoCase

Case Details

Test the performance of GitHub website create/delete empty repository, by simulate multiple concurrent browser users.

Test target system:  

GitHub website

Case steps:

        1. Open GitHub website

        2. Input user name and password -> Sign in

        3. Create a Repository

        4. Delete the Repository

        5. Sign out

Case Step1 Open GitHub website

Let’s analysis the Network requests here: open browser’s Developer Tool by clicking F12 and refresh this web page:

We can see after user inputting the URL, the very first request “login” will be called. 

After the first “login” GET, then there are many requests to get the “static resources”, such as CSS, Jscript and font.

Then there are a POST “collect” request, the domain is google-analytic, which I suppose is a traffic statistics. 

And after it, there is another GET request to collector.githubapp.com, which I suppose is the GitHub traffic statistics itself or something else.

The last one is the POST “stats” to api.github.com:

After the above analysis, there are some requests related with “static resources”, and some requests send to other domains like google to do some other thing (e.g. traffic statistics). In the official doc, JMeter suggest you to focus on the requests you really “interested in”. This is reasonable because for example the “static resource” requests, you’ll find most of time these requests are cached and not really send to server side:

For the other domain requests, in real world, it is not controlled by our developer, so we also ignore them: 

Now we can start to prepare the JMeter scripts: Add a “Thread Group” and give it a meaningful name like UI Simulator, left all other settings by default, then add the first HTTP Request Sample “login” according browser Developer Tool request in browser:

To simplify later operations, we’d like to add a HTTP Request Defaults, which can hold request default values. Then we set the values of Protocol, Server name in the HTTP Request Default, and remove these values from the HTTP Request Sampler:

Next step is to add the Listener so we can see the execution result, here I add 2 most common Listeners: View Results Tree and Aggregate Report:

Now we try to execute it to see the result:   

To make the pressure more real, even we don’t add the “static resource” requests explicitly, we still want our script to generate this kind of pressure. JMeter also has the cache mechanism to simulate this. To approach this, firstly we enable the “Retrieve All Embedded Resources” and “Parallel downloads” from HTTP Request Default:

Execute again and you will find the difference: 

If we let one user to execute this request twice, you’ll find these “static resource” requests be loaded twice: 

This is obviously NOT what we want and different with real browser behaviour: the real browser will cache these resource request in local as below:

So we need to let JMeter enable the cache mechanism, by adding a HTTP Cache Manager: 

Now we run again to see the difference:   

Good! Now we only have one HTTP Request Sampler, let’s add a Simple Controller to make the hierarchy more clear, and also we add 2 Response Assertions to verify the result after calling this request:

Now we go back to the Developer Tools to check which one is the next request. As we said before, the “static resource” requests are not recommend to add to JMeter script, unless we simulate by enable the JMeter “Retrieve All Embedded Resources” function and JMeter Cache Manager. You will see there are another three “font” requests which JMeter not send, I’ll suggest to ignore them. (In other hand, If you add these 3 requests to your JMeter script, during testing the JMeter thread will send these 3 request to server side every iteration, this is not consistent with real world, because in real browser, most of time these 3 requests are been cached and not reach to server side)

And then next request is the POST “collect” one, we also ignore it because it is for other domain. So we only let the “page_view” and “stats” requests, we only need to create them in JMeter as the “login” one, remember to add Response Assertion to them:

Case Step2 Input user name and password -> Sign in

Nextis the user input correct GitHub user name and password, then click Sing inbutton. As the last one, we check the requests firstly:

The first one is the POST “session”, you can see it has 5 parameters: “commit” and“utf8” is fixed, “login” and “password” are the values you input in web page;the only question is, where comes the “authenticity_token”?

Hereis the answer, let’s move back to the previous page and try to find it:

That’s it! The token already transferred to browser side when you first open the loginURL, by the GET “login” request.

So the left work is to extract this token from “login” response, because the response is the HTML format so we use CSS/JQuery Extractor, and save the value to variable “token”:

For the GitHub user name and password, we use a csv file “account.csv” to save it,this file is put into JMeter bin folder:

In JMeter, add a CSV Data Set Config to read this csv file as below:

So now we can add the HTTP Request and add these predefined parameters to it:

After success Sign in, another page is loaded

As above steps, we add the missing requests except the static resources requests:

Note: Forthe GET “page_view” Path, we need to parameterize the value of “actor_id” and“actor_login”, which can be get from the GET “github.com” response:

Then these 2 variables can be used in GET “page_view” Path:

Another interesting request is the web socket request:

By default, the JMeter doesn’t support wss, you need to install the JMeter plug-into support this protocol.

Please refer to https://jmeter-plugins.org/install/Install/to download the latest pluginjar, and then put it into lib/ext, after restarting JMeter, you will find the Plugins Manager, in tab “Available Plugins”, we enable the “WebSocket Samplers by Peter Doornbosch” and restart JMeter again.

Where the parameter WSPath comes from? Actually this URL is already in the responseHTML when “Redirect to Initial Page”:

So we add another extractor to get the socket URL, we only need the string start from“live.github.com/”, so after getting the whole URL, we add a BeanShell PostProcessor to analysis it:

Add HTTP Cookie Manager and you can try to pilot execute the script:

Next operation is to create/delete the repository, here we use User Defined Variables to save the repository name.

Before we start, generally we will judge whether the repository exist or not: If there pository not exist, we create it then delete; but if the repository already exist, we delete it then create.

This kind of optimize sometimes can save you lots of time during performance testing: image you already setup the performance script with many concurrent thread running, suddenly the system crash or get error, you need to re-testagain. But it is hard to tell which thread already created the repository or already delete it, in this scenario, if your script doesn’t judge the precondition,you have to recover then environment for every thread to remove the created repository. If your script has this kind of judgement, you just re-launch it!

In JMeter we use XPath Extractor to try to find the repository, if we find it,means the repository already exist:

Then the follow-up BeanShell PostProcessor will set 2 variables value according toXPath Extractor result, the 2 variables will be used later:

We add an “If Controller” to setup execution branch in JMeter as below:

Case Step3&4 Create/Delete a Repository

Now let’s speedup! The later steps of Create Repository and Delete Repository are basically the same with above steps, I’ll not get into more details, below is the finally result:

Pay attention to the “Check Repository Name” request, it send a multipart form data, then we added a HTTP Header Manager and in this request, we checked the“Use multipart/form-data for POST” and “Browser-compatible headers”:

Case Step5 Sign out

Finally we create the Sign out steps:

Add Think Time

Add think time, we assume user will have 1s think time among each request, toachieve this, just need to add a Constant Timer:

Monitor Server/Client resource

There are many ways to monitor the server resource, at least the operating system already provide the basic tool to monitor the system healthy, like Performance Monitor in Windows and many shell scripts in Linux. Here we just try the JMeter plugin which can be get from JMeter Plugin Manager easily:

Just need to enable this plugin and restart JMeter, you will find the new listenerjp@gc – PerfMon Metrics Collector

We also need to download and start a ServerAgent in target machine, the target canbe your server OS or the client OS where JMeter started.

You can find the ServerAgent here:

https://github.com/undera/perfmon-agent/blob/master/README.md

Now for the demonstration, we start the agent in the same JMeter Windows OS:

Once you start the JMeter script, you will find the OS resource info is collected to this listener:

Adjust stress

Depend on your requirement, JMeter can set different pressure to the target system:

    --- Sometimes we need to fix the total execution iterations, like we need 10 concurrent users, ramp up in 10s, each user execute the case 5 times:

   ---- Sometimes we need to fix the test duration,like we need 10 concurrent users, ramp up in 10s, and continue to testing for 5 min:

Note:In our demo, make sure the “Number of Threads” >= values in account.csv

---

3. Generate HTML Report

Even we have different Listeners to show the runtime statistical data, most of time we hope to save the result to file and we also hope to have a better display,below is how to do this:

Specify the file name in one listener then the crude data will be saved, you can also set which kind of info be saved:

The saved data is hard to read, we need to make it easy to read by the following JMeter command:

JMeter.bat–g “dataFilePath” –o “HTMLFilePath”

Then we get a HTML report:

---

4. Non-GUI mode

You may already noticed when you first start JMeter: there is a tip message said you should NOT use JMeter GUI model to start the performance testing, because the runtime static info update also cost many local OS resources, in case of this happen, JMeter suggest to use non-GUI model to kick off the testing:

When running under non-GUI model, you can also check the runtime result if you haveadded an Aggregate Report listener:

The non-GUI model should be the correct way when you start large stress performance testing in case of running out of local resources.

---

5. Record & Replay

As many other performance/automation testing tool, JMeter also support recording HTTP requests, but JMeter also like other software: the recorded scripts are not so good to use directly, you may need to do the analysis, filter,parameterize, add logic and add response assertion… by yourself manually.

Belowshow how to record by JMeter:

Add a HTTP(s) Test Script Recorder, you can see the default Port is 8888 and there isa selection about how to group recorded requests:

We use the default value, and only change is we add a “URL Pattern to Exclude”, to exclude the default “static resource” requests:

Click the Start button and JMeter will generate a certificate to record HTTPS protocol:

Next you need to import this CA certificate to your local OS “Trusted Root Certification Authorities”:

Then we configure the browser to connect to JMeter:

Now we can operate from your browser, you will find the requests be recorded in JMeter side as you configured:

The above screen is the record result on GitHub Sign in and create repository, I believe you’re very clear about each recorded request because we already created them manually in our demo case ;-) As you see, you still need to check them one by one and do some organization work to let these scripts really can be used in later testing.

Note:After you finished the record, remember to stop JMeter HTTP(S) Test Script Record and remove the proxy setting in your browser.

---

6. Some Common Elements

It is highly recommended to check the official website if you just want to check the usage of some specified JMeter elements, here I just try to list the elements which I think is more comments according to my project experience:

    --- Thread Group

    --- Sampler

                --- HTTP Request Sampler

                --- Java Request Sampler(you can easily implement your customized request by Java, just                         need to extend the AbstractJavaSamplerClient class and implement some interfaces)

    --- Login Controller

                --- If Controller (see Case Step2 Input user name and password -> Sign in)

                --- Loop Controller

    --- Pre Processors

                --- JSR223 PreProcessor(support many languages and you can call external method in                         extend library)

                --- BeanShell PreProcessor(A language similar to Java and you can call external method in                         your own jar file! See Case Step2 Input user name and password -> Sign in)

        --- Post Processors

                --- CSS/JQuery Extractor

                --- JSON Extractor

                --- XPath Extractor

                --- BeanShell PostProcessor

      --- Assertions

                --- Response Assertion

     --- Timer

                --- Constant Timer

     --- Config Element

                --- CSV Data Set Config

                --- HTTP Header Manager

                --- HTTP Cookie Manager

                --- HTTP Cache Manager

                --- HTTP Request Defaults

    --- Listener

                --- View Results Tree

                --- Aggregate Report

---

7. Result Analysis

If we say one system satisfies the performance requirement, it means under some situation/pressure, the system can provide the “in time” and “correct” response after user send requests.

Through JMeter script, we already can simulate the stress by changing the amount of concurrent users and loop count, after execution, we need to judge whether the target system is healthy or not.

CriteriaPerformance Indicators:

        --- Response Time

        --- Error Rate

Both of these 2 indicators can be found in JMeter HTML report:

Another interesting diagram is the “Hit Per Second”, which can show you whether the JMeter add expected pressure or not. (In some case, JMeter cannot add enough pressure due to lack to local CPU/memory):

It is always a good idea to check the Hit Per Second after you finish a round of testing, to make sure JMeter generate correct stress to server.

Inmost of case, you need to increase or decrease the JMeter stress to find out the target system’s capacity, pay attention to the Response Time and Error Rate, any of them exceed your threshold, and you find the capacity.

Thenthrough check the outstanding request and target system resource usage, you can find out the bottlenecks.

---

8. Remote Testing

Sometimes you need to add huge to target server, even you use non-GUI model to kick off,you still find JMeter cannot add enough pressure to target (this can be recognized by checking the “Hit Per Second” diagram after testing, or you can just “feel” directly during the testing: the JMeter OS response very slow, the CPU/Memory is reaching to the maximum but target server works fine).

Inthe event that your JMeter client machine is unable, performance-wise, to simulate enough users to stress your server or is limited at network level, an option exists to control multiple, remote JMeter engines from a single JMeter client. By running JMeter remotely, you can replicate a test across many low-end computers and thus simulate a larger load on the server. One instanceof the JMeter client can control any number of remote JMeter instances, and collect all the data from them. This offers the following features:

        --- Saving of test samples to the local machine

        --- Management of multiple JMeter Engines from a single machine

        --- No need to copy the test plan to each server - the client sends it to all the servers

Pleaserefer to below official link to setup the remote JMeter server:

https://jmeter.apache.org/usermanual/remote-test.html

If you set up the environment correctly, you can launch the remote JMeter server from client:

And the JMeter server will execute the specified jmx and send back result:

---

9. JMeter in CI

Sometimes we need to include performance testing into CI, JMeter has a Maven plugin and Jenkins action to achieve this.

First we need to prepare the JMeter test scripts as above.

Then we create a Maven project, which has dependency “jmeter-maven-plugin”, thepom.xml configured as below:

As above xml configured, we attached the JMeter performance testing as “verify”phase.

The related jmx file name is specified in “testFilesIncluded”.

And we also can specify parameters in “propertiesUser” part, as the above example,we specify the parameters “threadCount”, “rampUpPeriodInSec” and“ExecDurationInSec”. These parameters default value can be set in pom.xml Properties part:

Meanwhile,we need to adjust JMeter script to accept these parameters. Actually these parameters will be write into JMeter configuration file “user.properties”, to use the parameters defined there, the JMeter script variable shall as below:

Then we put the prepared JMeter jmx file into this project, and if you’re using Eclipse, just specified the Goals as “verify”, the performance test will be kicked off:

If you can successful execute the script in your local, you can upload the project into your source code management tool, like GitHub.

Note: The JMeter Maven plugin is already a completed JMeter, actually it will generate a separate JMeter folder to execute your script (at YourProject\target\jmeter).

At Jenkins side:

Configure the job parameters:

Build Configuration:

Finally post the performance test result:

After trigger the Jenkins job, you will find the result in “Performance Trend”:

---

10. Compare to a Real Browser

Belowis the statement from JMeter official site:

“JMeter is not a browser, it works at protocol level. As far as web-services and remoteservices are concerned, JMeter looks like a browser (or rather, multiple browsers); however JMeter does not perform all the actions supported by browsers. In particular, JMeter does not execute the Javascript found in HTML pages. Nor does it render the HTML pages as a browser does (it's possible to view the response as HTML etc., but the timings are not included in any samples, and only one sample in one thread is ever displayed at a time).”

Let’s see the Chrome recorded duration firstly.

For example we want to measure the duration: from user clicking Sign in button,till the new page display.

There is a “Performance” tab in Develop Tool, at the Sign in page start to record:

Here is the result:

We zoom in to find the “click event” action, where indicators we click Sign in button:

Then we find the “read line”, which means the page document object loaded finish and user can see the page from browser:

This is the real “exact” duration from we press down mouse, until the page display in browser.

From the bottom cycle, you can see the different time range categories, like“Loading”, “Scripting”, “Rendering”, “Painting”, “Idle”…

        --- The “other” refers to an activity not yet instrumented by DevTools.Let’s ignore it.

        --- The “Idle” part is browser wait for the CPU or GPU to do some processing. The processing is             different for JMeter and real browser.

        --- The “Loading” part including HTTP Requests sending and receiving. If we don’t jump into details,             JMeter also do this part.

        --- The “Rendering” and “Painting” parts regarding to the UI display,obviously JMeter will not do this.

All the above analysis is just want to show you the difference between JMeter and areal browser, as you can see the difference is mainly on client (that is your browser) side, for the server side, it cannot recognize the differences.

So we cannot say from JMeter, the request(s) cost 2s, then in the real browser, user need 2s to see the page…Not only for the different mechanism, but also the difference of network situation, client machine hardware… But don’t worry about this because in most of cases, the purpose of performance testing is to find out the bottleneck or prove the capacity of target system but not focus on each client side.