BrowserMob Proxy

BrowserMob Proxy is a simple utility that makes it easy to capture performance data from browsers, typically written using automation toolkits such as Selenium and Watir.

The current stable version of BrowserMob Proxy is 2.0 -- see the 2.0.x README for usage information. Version 2.1 is currently in development, and already contains a number of improvements over 2.0. See the build instructions for instructions on creating a 2.1 beta release.

Features

The proxy is programmatically controlled via a REST interface or by being embedded directly inside Java-based programs and unit tests. It captures performance data the HAR format. It addition it also can actually control HTTP traffic, such as:

• blacklisting and whitelisting certain URL patterns
• simulating various bandwidth and latency
• remapping DNS lookups
• flushing DNS caching
• controlling DNS and request timeouts
• automatic BASIC authorization

REST API

To get started, first start the proxy by running browsermob-proxy or browsermob-proxy.bat in the bin directory:

$ sh browsermob-proxy -port 9090
INFO 05/31 03:12:48 o.b.p.Main           - Starting up...
2011-05-30 20:12:49.517:INFO::jetty-7.3.0.v20110203
2011-05-30 20:12:49.689:INFO::started o.e.j.s.ServletContextHandler{/,null}
2011-05-30 20:12:49.820:INFO::Started [email protected]:9090

Once started, there won't be an actual proxy running until you create a new proxy. You can do this by POSTing to /proxy:

[~]$ curl -X POST http://localhost:9090/proxy
{"port":9091}

or optionally specify your own port:

[~]$ curl -X POST -d 'port=9099' http://localhost:9090/proxy
{"port":9099}

or if running BrowserMob Proxy in a multi-homed environment, specify a desired bind address (default is 0.0.0.0):

[~]$ curl -X POST -d 'bindAddress=192.168.1.222' http://localhost:9090/proxy
{"port":9096}

Once that is done, a new proxy will be available on the port returned. All you have to do is point a browser to that proxy on that port and you should be able to browse the internet. The following additional APIs will then be available:

• GET /proxy - get a list of ports attached to ProxyServer instances managed by ProxyManager
• PUT /proxy/[port]/har - creates a new HAR attached to the proxy and returns the HAR content if there was a previous HAR. Supports the following parameters:
• initialPageRef - the string name of the first page ref that should be used in the HAR. Defaults to "Page 1".
• initialPageTitle - the title of first HAR page. Defaults to initialPageRef.
• captureHeaders - Boolean, capture headers
• captureContent - Boolean, capture content bodies
• captureBinaryContent - Boolean, capture binary content
• PUT /proxy/[port]/har/pageRef - starts a new page on the existing HAR. Supports the following parameters:
• pageRef - the string name of the first page ref that should be used in the HAR. Defaults to "Page N" where N is the next page number.
• pageTitle - the title of new har page. Defaults to pageRef.
• DELETE /proxy/[port] - shuts down the proxy and closes the port
• GET /proxy/[port]/har - returns the JSON/HAR content representing all the HTTP traffic passed through the proxy
• GET /proxy/[port]/whitelist - Displays whitelisted items
• PUT /proxy/[port]/whitelist - Sets a list of URL patterns to whitelist. Takes the following parameters:
• regex - a comma separated list of regular expressions
• status - the HTTP status code to return for URLs that do not match the whitelist
• DELETE /proxy/[port]/whitelist - Clears all URL patterns from the whitelist
• GET /proxy/[port]/blacklist - Displays blacklisted items
• PUT /proxy/[port]/blacklist - Set a URL to blacklist. Takes the following parameters:
• regex - the blacklist regular expression
• status - the HTTP status code to return for URLs that are blacklisted
• method - regular expression for matching method., e.g., POST. Emtpy for matching all method.
• DELETE /proxy/[port]/blacklist - Clears all URL patterns from the blacklist
• PUT /proxy/[port]/limit - Limit the bandwidth through the proxy. Takes the following parameters:
• downstreamKbps - Sets the downstream bandwidth limit in kbps
• upstreamKbps - Sets the upstream bandwidth limit kbps
• downstreamMaxKB - Specifies how many kilobytes in total the client is allowed to download through the proxy.
• upstreamMaxKB - Specifies how many kilobytes in total the client is allowed to upload through the proxy.
• latency - Add the given latency to each HTTP request
• enable - (true/false) a boolean that enable bandwidth limiter. By default the limit is disabled, although setting any of the properties above will implicitly enable throttling
• payloadPercentage - a number ]0, 100] specifying what percentage of data sent is payload. e.g. use this to take into account overhead due to tcp/ip.
• maxBitsPerSecond - The max bits per seconds you want this instance of StreamManager to respect.
• GET /proxy/[port]/limit - Displays the amount of data remaining to be uploaded/downloaded until the limit is reached.
• POST /proxy/[port]/headers - Set and override HTTP Request headers. For example setting a custom User-Agent.
• Payload data should be json encoded set of headers (not url-encoded)
• POST /proxy/[port]/hosts - Overrides normal DNS lookups and remaps the given hosts with the associated IP address
• Payload data should be json encoded set of name/value pairs (ex: {"example.com": "1.2.3.4"})
• POST /proxy/[port]/auth/basic/[domain] - Sets automatic basic authentication for the specified domain
• Payload data should be json encoded username and password name/value pairs (ex: {"username": "myUsername", "password": "myPassword"}
• PUT /proxy/[port]/wait - wait till all request are being made
• quietPeriodInMs - Sets quiet period in milliseconds
• timeoutInMs - Sets timeout in milliseconds
• PUT /proxy/[port]/timeout - Handles different proxy timeouts. Takes the following parameters:
• requestTimeout - request timeout in milliseconds. A timeout value of -1 is interpreted as infinite timeout. It equals -1 by default.
• readTimeout - read timeout in milliseconds. Which is the timeout for waiting for data or, put differently, a maximum period inactivity between two consecutive data packets). A timeout value of zero is interpreted as an infinite timeout. It equals 60000 by default
• connectionTimeout - Determines the timeout in milliseconds until a connection is established. A timeout value of zero is interpreted as an infinite timeout. It eqauls 60000 by default
• dnsCacheTimeout - Sets the maximum length of time that records will be stored in this Cache. A nonpositive value disables this feature (that is, sets no limit). It equals 0 y default
• PUT /proxy/[port]/rewrite - Redirecting URL's
• matchRegex - a matching URL regular expression
• replace - replacement URL
• DELETE /proxy/[port]/rewrite - Removes all URL redirection rules currently in effect
• PUT /proxy/[port]/retry - Setting the retry count
• retrycount - the number of times a method will be retried
• DELETE /proxy/[port]/dns/cache - Empties the Cache.

For example, once you've started the proxy you can create a new HAR to start recording data like so:

[~]$ curl -X PUT -d 'initialPageRef=Foo' http://localhost:8080/proxy/9091/har

Now when traffic goes through port 9091 it will be attached to a page reference named "Foo". Consult the HAR specification for more info on what a "pageRef" is. You can also start a new pageRef like so:

[~]$ curl -X PUT -d 'pageRef=Bar' http://localhost:8080/proxy/9091/har/pageRef

That will ensure no more HTTP requests get attached to the old pageRef (Foo) and start getting attached to the new pageRef (Bar). You can also get the HAR content at any time like so:

[~]$ curl http://localhost:8080/proxy/9091/har

Sometimes you will want to route requests through an upstream proxy server. In this case specify your proxy server by adding the httpProxy parameter to your create proxy request:

[~]$ curl -X POST http://localhost:9090/proxy?httpProxy=yourproxyserver.com:8080
{"port":9091}

Alternatively, you can specify the upstream proxy config for all proxies created using the standard JVM system properties for HTTP proxies. Note that you can still override the default upstream proxy via the POST payload, but if you omit the payload the JVM system properties will be used to specify the upstream proxy.

TODO: Other REST APIs supporting all the BrowserMob Proxy features will be added soon.

Command-line Arguments

• -port <port>
• Port on which the API listens. Default value is 8080.
• -address
• Address to which the API is bound. Default value is 0.0.0.0.
• -proxyPortRange <from>-<to>
• Range of ports reserved for proxies. Only applies if port parameter is not supplied in the POST request. Default values are <port>+1 to <port>+500+1.
• -ttl <seconds>
• Proxy will be automatically deleted after a specified time period. Off by default.

Embedded Mode

New in 2.1: New Embedded Mode module

BrowserMob Proxy 2.1 separates the Embedded Mode and REST API into two modules. If you only need Embedded Mode functionality, add the browsermob-core artifact (or browsermob-core.jar file) as a dependency. The REST API artifact is browsermob-rest.

If you're using Java and Selenium, the easiest way to get started is to embed the project directly in your test. First, you'll need to make sure that all the dependencies are imported in to the project. You can find them in the lib directory. Or, if you're using Maven, you can add this to your pom:

<dependency>
    <groupId>net.lightbody.bmp</groupId>
    <artifactId>browsermob-core</artifactId>
    <version>2.1.0-beta-1-SNAPSHOT</version>
    <scope>test</scope>
</dependency>

Once done, you can start a proxy using net.lightbody.bmp.proxy.ProxyServer:

ProxyServer server = new ProxyServer(9090);
server.start();

This class supports every feature that the proxy supports. In fact, the REST API is a subset of the methods exposed here, so new features will show up here before they show up in the REST API. Consult the Javadocs for the full API.

If your project already defines a Selenium dependency then you may want to exclude the version that browsermob-proxy pulls in, like so:

<dependency>
    <groupId>net.lightbody.bmp</groupId>
    <artifactId>browsermob-core</artifactId>
    <version>2.1.0-beta-1-SNAPSHOT</version>
    <scope>test</scope>
    <exclusions>
        <exclusion>
            <groupId>org.seleniumhq.selenium</groupId>
            <artifactId>selenium-api</artifactId>
        </exclusion>
    </exclusions>
</dependency>

Using With Selenium

You can use the REST API with Selenium however you want. But if you're writing your tests in Java and using Selenium 2, this is the easiest way to use it:

// start the proxy
ProxyServer server = new ProxyServer(4444);
server.start();

// get the Selenium proxy object
Proxy proxy = server.seleniumProxy();

// configure it as a desired capability
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, proxy);

// start the browser up
WebDriver driver = new FirefoxDriver(capabilities);

// create a new HAR with the label "yahoo.com"
server.newHar("yahoo.com");

// open yahoo.com
driver.get("http://yahoo.com");

// get the HAR data
Har har = server.getHar();

HTTP Request Manipulation

You can manipulate the requests like so:

server.addRequestInterceptor(new RequestInterceptor() {
    @Override
    public void process(BrowserMobHttpRequest request, Har har) {
        request.getMethod().removeHeaders("User-Agent");
        request.getMethod().addHeader("User-Agent", "Bananabot/1.0");
    }
});

You can also POST a JavaScript payload to /:port/interceptor/request and /:port/interceptor/response using the REST interface. The functions will have a request/response variable, respectively, and a har variable (which may be null if a HAR isn't set up yet). The JavaScript code will be run by Rhino and have access to the same Java API in the example above:

[~]$ curl -X POST -H 'Content-Type: text/plain' -d 'request.getMethod().removeHeaders("User-Agent");' http://localhost:9090/proxy/9091/interceptor/request

Consult the Java API docs for more info.

SSL Support

While the proxy supports SSL, it requires that a Certificate Authority be installed in to the browser. This allows the browser to trust all the SSL traffic coming from the proxy, which will be proxied using a classic man-in-the-middle technique. IT IS CRITICAL THAT YOU NOT INSTALL THIS CERTIFICATE AUTHORITY ON A BROWSER THAT IS USED FOR ANYTHING OTHER THAN TESTING.

If you're doing testing with Selenium, you'll want to make sure that the browser profile that gets set up by Selenium not only has the proxy configured, but also has the CA installed (Firefox set up by Selenium has installed CA by default). Unfortuantely, there is no API for doing this in Selenium, so you'll have to solve it uniquely for each browser type. We hope to make this easier in upcoming releases.

NodeJS Support

NodeJS bindings for browswermob-proxy are available here. Built-in support for Selenium or use CapserJS-on-PhantomJS or anything else to drive traffic for HAR generation.

Logging

When running in stand-alone mode, the proxy loads the default logging configuration from the conf/bmp-logging.properties file. To increase/decrease the logging level, change the logging entry for net.lightbody.bmp.

New in 2.1: Neither Embedded Mode nor the REST API include an slf4j static binding, so you no longer need to exclude the slf4j-jdk14 dependency when including browsermob-core or browsermob-rest.

Native DNS Resolution

If you are having name resolution issues with the proxy, you can use native Java name resolution as a fallback to browsermob-proxy's default XBill resolution. To enable native DNS fallback, set the JVM property bmp.allowNativeDnsFallback to true.

When running from the command line:

$ JAVA_OPTS="-Dbmp.allowNativeDnsFallback=true" sh browsermob-proxy

or in Windows:

C:\browsermob-proxy\bin> set JAVA_OPTS="-Dbmp.allowNativeDnsFallback=true"
C:\browsermob-proxy\bin> browsermob-proxy.bat

If you are running within a Selenium test, you can set the bmp.allowNativeDnsFallback JVM property when you launch your Selenium tests, or programmatically when creating the Proxy:

System.setProperty("bmp.allowNativeDnsFallback", "true");
ProxyServer proxyServer = new ProxyServer(0);
proxyServer.start();

Creating the batch files from source

You'll need maven (brew install maven if you're on OS X); use the release profile to generate the batch files from this repository. Optionally, proceed at your own risk and append the -DskipTests option if the tests are failing.

[~]$ mvn -P release
[~]$ mvn -DskipTests -P release