How to Monitor Mobile App Traffi

转自 https://stanfy.com/blog/monitor-mobile-app-traffic-with-sniffers/

by Oleg Nikiforov (@ddr3ams), QA Engineer at Stanfy • 13 min read

Are your QA engineers into monitoring apps’ traffic? Really, are they? The answer I usually hear is “why would I be?” Of course, this is irrelevant if your application works without access to a server and is purely offline. On the other hand, if your mobile app has network interactions, it is essential to make sure everything works smoothly there. Unfortunately, this part of testing is often used in web development but rarely used during mobile app development.
When I talk about analyzing app traffic I usually mean working with sniffers (that come in a form of proxy). These are special software that is intended to help you see network interactions in the form of HTTP or HTTPS requests and responses. While monitoring insecure HTTP traffic is relatively easy, working with HTTPS requires some additional actions (I’ll tell you about this in detail later).

How to Monitor Mobile App Traffic With Sniffers
Table of contents:
[Approaches to monitoring mobile application traffic](https://stanfy.com/blog/monitor-mobile-app-traffic-with-sniffers/#Approaches to analyzing mobile app traffic)
[Setting up your workflow](https://stanfy.com/blog/monitor-mobile-app-traffic-with-sniffers/#Setting up your workflow)
[How-to install a certificate on an Android](https://stanfy.com/blog/monitor-mobile-app-traffic-with-sniffers/#How-to install a certificate on an Android)
[How-to install a certificate on an iOS](https://stanfy.com/blog/monitor-mobile-app-traffic-with-sniffers/#How-to install a certificate on an iOS)
[Working with proxy](https://stanfy.com/blog/monitor-mobile-app-traffic-with-sniffers/#Working with proxy)
[Optimizing your workflow with traffic sniffer](https://stanfy.com/blog/monitor-mobile-app-traffic-with-sniffers/#Optimizing your workflow with traffic sniffer)
[When one proxy is not enough](https://stanfy.com/blog/monitor-mobile-app-traffic-with-sniffers/#When one proxy is not enough)
Summary

Approaches to analyzing mobile app traffic
Ask your mobile app developers what monitoring tool they use when developing the network interactions. The most common answer is “IDE console output”. Yes, it provides all the necessary information for you, and can even be used by your QAs (assuming they know how to build apps locally with the help of IDE :)), though it’s output can look really chaotic when server responses are large:

Android StudioAndroid Studio output

XcodeXcode output

And even when requests and responses are displayed on the console, they still may miss some important info, such as custom headers.
We’ve also made a PDF-version of this article. Download
While this approach can be used with applications you develop, you usually won’t be able to monitor other apps’ traffic. Indeed, there are cases when you have to analyze other applications (for example: when building your own API, analyzing other applications can help you to save time). This is where sniffers come in handy.
Let’s define the first advantage of using sniffers:
You get traffic analysis in a convenient and readable format

Burp SuiteBurp Suite

In addition to simply reading network data, there are plenty of situations when you need to manipulate it. I assume you can find a way to inject needed data right into an application while debugging it if you know how to code. However, using sniffers for data manipulation is a relatively easier approach and you can change data without code knowledge. This brings us to the second advantage of using sniffers:
Easy data manipulation.

Setting up your workflow
There are several well known applications for sniffing HTTP/HTTPS traffic: Fiddler,Charles Proxy, TCP Catcher, Burp Suite. In my everyday work I use Burp Suite and TCP Catcher. It’s up to you what sniffer software to use, but further examples will be provided using Burp Suite, so to begin, I’ll tell you how to configure it. Setting-up a mobile sniffer for HTTP traffic is pretty easy:
Download Burp Suite – it’s a Java application, so you will need to install Java on your computer, in case you haven’t installed it already;
Run the app;
Open “Proxy” > “Options”: you will see a list of Proxy Listeners, one is already set by default;
Select it and click “Edit” > and you will see the next screen:

Edit proxy listenerEdit proxy listener

4.1 Select a port to use with proxy (I use port 8200 but you can leave port 8080)4.2 Change the address to your IP4.3 Click “OK” and you’ll see something like this:

Proxy listenersProxy listeners

5. Setup a proxy on your device.
   Let’s take a deeper look at how to setup a proxy on iOS and Android devices.
   How to set up a proxy on Android:

6. Setting up a proxy for your Wi-Fi network: Go to “Settings” > “Wi-Fi” > long press on connected network > choose “Modify network” > change “Proxy” to “Manual” and enter your computer’s IP address into the “Proxy hostname” field and the port where your mobile traffic sniffer is running into “Proxy port”:
   

   Setting up proxy on Android deviceSetting up proxy on Android device

7. Click “Save“.
   How to set up a proxy on iOS:

8. To set up a proxy on a device: Go to “Settings” > ”Wi-Fi” > open network > Scroll to “HTTP PROXY” section , change it to “Manual” and enter your computer’s IP into “Server” and your port where Burp is running into “Port”:
   

   Setting up proxy on iOS deviceSetting up proxy on iOS device

In Burp Suite, open “Proxy” > “Intercept” and turn interception off:

Turning off interception in Burp SuiteTurning off interception in Burp Suite

Next open the browser on your device and go to http://www.w3.org (or other site that uses HTTP protocol).
Finally, open “HTTP history” in the program and you should see requests there:
HTTP History" width="740" height="188" srcset="https://stanfy.com/wp-content/uploads/2016/04/9.png 1920w, https://stanfy.com/wp-content/uploads/2016/04/9-300x76.png 300w, https://stanfy.com/wp-content/uploads/2016/04/9-1024x260.png 1024w" sizes="(max-width: 740px) 100vw, 740px" style="box-sizing: border-box; position: relative; vertical-align: middle; margin: 0px; padding: 0px; border: 0px none; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; font-size: inherit; line-height: inherit; font-family: inherit; max-width: 98.5%; width: auto; height: auto; background-repeat: no-repeat;">Proxy > HTTP History

Note: If you don’t see anything in this tab – check the “Alerts” tab for any Burp errors (the label becomes red if anything wrong happens). Also try to ping your device and check that your firewall is disabled or allows proxy connection; it is common to be unable to connect via proxy because of firewall settings.
We’ll talk about actual work with this data later, but first I’ll tell you how to set up HTTPS proxy. In order to be able to see and manipulate HTTPS requests you need to perform a MitM attack. While it’s not ethical to apply this to third-party applications, this is necessary for testing your own mobile app.

1. Change Burp Suite certificate settings:
   Enable support invisible proxying Enable support invisible proxying

“Options” > “Proxy Listeners” > “Edit your listener” > “Certificate”" width="740" height="338" srcset="https://stanfy.com/wp-content/uploads/2016/04/11.png 1400w, https://stanfy.com/wp-content/uploads/2016/04/11-300x137.png 300w, https://stanfy.com/wp-content/uploads/2016/04/11-1024x468.png 1024w" sizes="(max-width: 740px) 100vw, 740px" style="box-sizing: border-box; position: relative; vertical-align: middle; margin: 0px; padding: 0px; border: 0px none; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; font-size: inherit; line-height: inherit; font-family: inherit; max-width: 98.5%; width: auto; height: auto; background-repeat: no-repeat;">“Proxy” > “Options” > “Proxy Listeners” > “Edit your listener” > “Certificate”

2. Install CA on the device
   How to install a certificate on an Android device:
   Open the browser on your device and go to http://burp/cert and the certificate will be downloaded.
   Rename the downloaded file (cacert.crt) to cacert.cer (or any name with .cer).
   Go to “Settings” > “Security” > “Install from storage.”
   Open cacert.cer to install it.

How to install a certificate on an iOS device:
Open the browser on your device and go to http://burp/cert > and the certificate will be downloaded and added to “Profiles.”

3. On your device, open the browser and go to www.google.com (or other site that uses HTTPS protocol).
4. Open “HTTP history” in Burp and you should see requests there:
   “HTTP History”" width="740" height="162" srcset="https://stanfy.com/wp-content/uploads/2016/04/12.png 1920w, https://stanfy.com/wp-content/uploads/2016/04/12-300x66.png 300w, https://stanfy.com/wp-content/uploads/2016/04/12-1024x224.png 1024w" sizes="(max-width: 740px) 100vw, 740px" style="box-sizing: border-box; position: relative; vertical-align: middle; margin: 0px; padding: 0px; border: 0px none; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; font-size: inherit; line-height: inherit; font-family: inherit; max-width: 98.5%; width: auto; height: auto; background-repeat: no-repeat;">“Proxy” > “HTTP History”

After this, you still won’t be able to see the traffic of some applications. Mostly this is related to the usage of SSL pinning which prevents an app from trusting your certificate. Though this security check can be bypassed, that action is beyond the scope of this article.
Thus, if your application being tested is well protected, you should make changes in code or ask your developers for help. Below is the list of cases that could prevent you from sniffing traffic:
SSL pinning – can be turned off for debugging (make sure it is turned on for release build).
Apple Transport Layer Security – be aware that on iOS 9 you need to add additional parameters to use proxy: open Info.plist and add App Transport Security Settings > Allow Arbitrary Loads:YES
Okhttp library on Android may bypass device proxy settings – in that case you need to setup the proxy in source code (use OkHttpClient “setProxy” method):

client.setProxy(new Proxy(Proxy.Type.HTTP, new InetSocketAddress("10.1.1.90", 8200)));

Working with proxy
I use version 1.6.32. Burp is updating regularly, but I don’t think this main flow should change in further updates.
This is how you can intercept requests and responses:
In Burp Suite open “Proxy” > “Intercept.”
Turn interception on.
Open Browser on device and go to www.google.com >. You’ll see an intercepted request:

On this step you can "drop" or "forward" this requestOn this step you can “drop” or “forward” this request

You can click on “Drop” to cancel this request or “Forward” to send it to the server.
OR you can intercept the response:

1. Click on “Action” or click on request body with RMB > select “Do intercept” > select “Response to this request”:
   select “Do intercept” > select “Response to this request”" width="740" height="294" srcset="https://stanfy.com/wp-content/uploads/2016/04/14.png 1920w, https://stanfy.com/wp-content/uploads/2016/04/14-300x119.png 300w, https://stanfy.com/wp-content/uploads/2016/04/14-1024x407.png 1024w" sizes="(max-width: 740px) 100vw, 740px" style="box-sizing: border-box; position: relative; vertical-align: middle; margin: 0px; padding: 0px; border: 0px none; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; font-size: inherit; line-height: inherit; font-family: inherit; max-width: 98.5%; width: auto; height: auto; background-repeat: no-repeat;">RMB > select “Do intercept” > select “Response to this request”

2. Click on “Forward.”3. You will see the response and can modify it as you like. After all modifications are done, click on “Forward” to send request to the device:
   

   click on “Forward” to send request to the deviceClick on “Forward” to send request to the device

Note: you should make any changes before you reach the timeout limit or this modified response won’t get to the device. Also check the “Options” > “Misc” > “Hotkeys” section in the app, they can save you plenty of time.
That is a basic workflow for using proxy.
Now I’ll show you some examples of how working with requests may help you in testing applications.

1. Request validation
   Check that values submitted to the server are in the proper format.
2. Response validation
   Check that the response is in the proper format and contains all necessary headers and data in the body.
3. Long strings in fields
   Instead of submitting data to the server to test how long values are displayed you can simply change the value in the server response on the fly.
4. Load more items
   When you have a “load more”, usually it is limited by a certain amount of items and this amount is passed on to the server; intercept the request and change this param to get more items and see how your application deals with a large amount of data.
5. Testing error handling
   Intercept any server response and change its value to simulate error – 401 Unauthorized, 400 Bad Request, 500 Internal Server Error, and so on. Just make sure that responses are created properly; a good approach is to save real error messages for future usage.
6. Test an expired session
   A good way to test an expired session is to intercept a request and change the value of token (or auth header), then the server will respond with a real error message.
7. Test time-outs
   On iOS devices you can use Network Link Conditioner to simulate time-outs; on Android this task is a bit harder, but you can always use a proxy to test this – just intercept a request and wait until the app drops it because of the timeout.
8. Perform actions limited by UI
   If you want to perform an action on some user but cannot do it because of UI limitation (this user isn’t shown to you) – perform this action on any other user, intercept the corresponding request and change the user ID to any value you need.
   Optimizing your workflow with traffic sniffer
   Burp Suite has various options to enhance your work with traffic:
   Some apps use various 3rd party libraries and may send tons of server requests that are not relevant for your tests. In this case you can add all needed URLs to scope and work only with them:

“Scope” > “Include in scope” - this is how you add certain URLs into scope" width="740" height="188" srcset="https://stanfy.com/wp-content/uploads/2016/04/16.png 2560w, https://stanfy.com/wp-content/uploads/2016/04/16-300x76.png 300w, https://stanfy.com/wp-content/uploads/2016/04/16-1024x261.png 1024w" sizes="(max-width: 740px) 100vw, 740px" style="box-sizing: border-box; position: relative; vertical-align: middle; margin: 0px; padding: 0px; border: 0px none; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; font-size: inherit; line-height: inherit; font-family: inherit; max-width: 98.5%; width: auto; height: auto; background-repeat: no-repeat;">“Target” > “Scope” > “Include in scope” – this is how you add certain URLs into scope

“Http history” > tap on white bar to see available filters > toggle “Show only in-scope items” - this is how you apply scope filtering" width="740" height="224" srcset="https://stanfy.com/wp-content/uploads/2016/04/17.png 2560w, https://stanfy.com/wp-content/uploads/2016/04/17-300x91.png 300w, https://stanfy.com/wp-content/uploads/2016/04/17-1024x310.png 1024w" sizes="(max-width: 740px) 100vw, 740px" style="box-sizing: border-box; position: relative; vertical-align: middle; margin: 0px; padding: 0px; border: 0px none; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; font-size: inherit; line-height: inherit; font-family: inherit; max-width: 98.5%; width: auto; height: auto; background-repeat: no-repeat;">“Proxy” > “Http history” > tap on white bar to see available filters > toggle “Show only in-scope items” – this is how you apply scope filtering

Let’s compare how our history looks including all items and only in-scope items;

“All items” (pay attention to Filter)“All items” (pay attention to Filter)

Only items in scopeOnly items in scope

Don’t want to see certain requests occasionally? You can add them to the list of URLs excluded from scope and mark them when necessary:

“Scope” > “Exclude from scope” - this is how you remove certain URLs from scope." width="740" height="192" srcset="https://stanfy.com/wp-content/uploads/2016/04/20.png 2560w, https://stanfy.com/wp-content/uploads/2016/04/20-300x78.png 300w, https://stanfy.com/wp-content/uploads/2016/04/20-1024x266.png 1024w" sizes="(max-width: 740px) 100vw, 740px" style="box-sizing: border-box; position: relative; vertical-align: middle; margin: 0px; padding: 0px; border: 0px none; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; font-size: inherit; line-height: inherit; font-family: inherit; max-width: 98.5%; width: auto; height: auto; background-repeat: no-repeat;">“Target” > “Scope” > “Exclude from scope” – this is how you remove certain URLs from scope.

Now look at our history that is filtered by scope while removing some items from it:

History that is filtered by scope while removing some items from itHistory that is filtered by scope while removing some items from it

You can configure app to intercept requests / responses when they meet certain conditions:

“Options” > “Intercept Client Requests” / “Intercept Server Responses” - while interception is turned on only items that apply to specified rules will be intercepted (be aware that you still have to intercept a request before intercepting a response)" width="740" height="423" srcset="https://stanfy.com/wp-content/uploads/2016/04/22.png 2560w, https://stanfy.com/wp-content/uploads/2016/04/22-300x171.png 300w, https://stanfy.com/wp-content/uploads/2016/04/22-1024x585.png 1024w" sizes="(max-width: 740px) 100vw, 740px" style="box-sizing: border-box; position: relative; vertical-align: middle; margin: 0px; padding: 0px; border: 0px none; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; font-size: inherit; line-height: inherit; font-family: inherit; max-width: 98.5%; width: auto; height: auto; background-repeat: no-repeat;">“Proxy” > “Options” > “Intercept Client Requests” / “Intercept Server Responses” – while interception is turned on only items that apply to specified rules will be intercepted (be aware that you still have to intercept a request before intercepting a response)

You can also auto change any part of a response or request:

“Options” > “Match and Replace” - set rules for auto replacing content." width="740" height="164" srcset="https://stanfy.com/wp-content/uploads/2016/04/23.png 1920w, https://stanfy.com/wp-content/uploads/2016/04/23-300x66.png 300w, https://stanfy.com/wp-content/uploads/2016/04/23-1024x227.png 1024w" sizes="(max-width: 740px) 100vw, 740px" style="box-sizing: border-box; position: relative; vertical-align: middle; margin: 0px; padding: 0px; border: 0px none; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; font-size: inherit; line-height: inherit; font-family: inherit; max-width: 98.5%; width: auto; height: auto; background-repeat: no-repeat;">“Proxy” > “Options” > “Match and Replace” – set rules for auto replacing content.

For example, if your application forces users to upgrade to a newer version that is available, this is usually triggered on the backend. You can change app-version (or any similar) header in all requests to see if the server responds with an error message.
To summarize usage of Burp:

• lets you see necessary requests and responses+ lets you intercept requests and responses and manipulate data on the fly+ very flexible settings– needs to be set up on each device– can prevent some apps or 3rd party libraries from running because of SSL errors
  Be aware that if your app uses some 3rd party libraries, they may not work with Burp Suite. For example Amazon SDK will fail because of SSL errors. You can bypass this issue by adding corresponding URLs to the “SSL Pass Through” list in Burp Suite and turning them on when necessary:
  “Options” > “SSL Pass Through” - add URLs that should be bypassed without monitoring" width="740" height="141" srcset="https://stanfy.com/wp-content/uploads/2016/04/24.png 1920w, https://stanfy.com/wp-content/uploads/2016/04/24-300x57.png 300w, https://stanfy.com/wp-content/uploads/2016/04/24-1024x195.png 1024w" sizes="(max-width: 740px) 100vw, 740px" style="box-sizing: border-box; position: relative; vertical-align: middle; margin: 0px; padding: 0px; border: 0px none; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; font-size: inherit; line-height: inherit; font-family: inherit; max-width: 98.5%; width: auto; height: auto; background-repeat: no-repeat;">“Proxy” > “Options” > “SSL Pass Through” – add URLs that should be bypassed without monitoring

When one proxy is not enough :)
Burp Suite is a great network monitoring tool for testing and debugging – I use it everyday and cannot imagine my current workflow without it. What I lack is the ability to log application traffic for 24 hours a day and on devices I don’t have access to (e.g. clients’ or beta testers’ devices).
To handle this necessity I use another proxy tool called Runscope – this is a cloud-based proxy. It doesn’t allow you to manipulate data but it can be useful for logging.
To set it up you need to change base URL in your mobile app to a string created by Runscope. In this way only your app will work with the proxy and only requests sent to this special URL will go through it. So if your app uses several URLs you will need to change all of them.

Use “URL Helper” to modify your base URLUse “URL Helper” to modify your base URL

By using Runscope you will slightly increase the time taken to perform client-server operations (as you are adding an additional point on the route of your data) but you get the ability to monitor app’s traffic from all devices.
Thus, when a client tells you that something isn’t working you can quickly check logs for any errors. Even when incorrect server responses may not trigger UI changes and be visible for users (for example registering for push notifications fails), by creating a filter for failed requests you will be able to check what is wrong at any point in time:
Search for “status: 400” to see requests that failed with 400 HTTP code:

Search for “status: 400” to see requests that failed with 400 HTTP codeSearch for “status: 400” to see requests that failed with 400 HTTP code

Click on “Create Stream View” to save this filter in the “STREAM VIEWS” section:

Click on “Create Stream View” to save this filter in the “STREAM VIEWS” sectionClick on “Create Stream View” to save this filter in the “STREAM VIEWS” section

Now you will be able to quickly navigate to the list of requests that failed with 400 HTTP code:

Now you will be able to quickly navigate to the list of requests that failed with 400 HTTP codeNow you will be able to quickly navigate to the list of requests that failed with 400 HTTP code

Runscope pros and cons:

• no need to turn on proxy on device+ proxy works for specified URLs– may be hard to setup (depends on which network library you use)– changes in source code are necessary to turn proxy on / off– free package is limited in number of requests (though it’s usually enough for testing purposes)
  Product Design and Development Tips & Infographics

Give it a try. It only takes a click to unsubscribe.

Summary
Analyzing app traffic doesn’t require any high-level skills though it may seem difficult when you begin. As you introduce this practice into your daily workflow, soon you will see how natural it is to use proxy every time you test your application.
I hope that the above examples will help you in mastering your mobile app testing skills and advancing them to the next level. Enjoy ;)
P.S. Although this article is written from the perspective of a QA Engineer, I strongly recommend that all mobile app developers who work on projects with a backend use sniffers for inspecting and manipulating apps’ traffic.
More interesting reads:
How Switching From Dev to QA Automation Changed The Way I Work
Tools and Platforms for Mobile App Analytics. Part I: Performance and Errors