Add custom HTTP headers in Selenium Webdriver tests with the help of BrowserMob-Proxy (Java)
Despite adding header to http requests when test user interface is not a sort of common practice, being able to do that would really simplify the range of tasks which a tester might potentially encounter. With the help of headers one can control different aspect of application or service logic. Headers might help to enable some part of application logic that would be disabled in a normal mode (for example some king of "guest" mode), or work around some phases of user interaction with your application which cannot be controlled by WebDriver. One of such phases is Basic authentication, which we’re going to use as an example in this post.
If you’re looking for Python solution, it is worth checking this my post: Four simple steps to add custom HTTP headers in Selenium Webdriver tests in Python
How does Basic authentication work
Basic authentication is the very simple mechanism of user identity detection which is (despite its simplicity and poor protection) quite widely used in practice. Such authentication type is supported by all the modern browsers and often becomes an unpleasant obstacle on the way ot test automation engineers since authentication dialog is something not supported by Selenium WebDriver.
When a client requests a resource (URL) that is hidden behind basic authentication mechanism, a server responds with 401 (Unauthorized) status and at the same time adds WWW-Authenticate header to the response with the assigned value of required authentication type (in our case it is Basic) and optional resource description in a format of realm="Resource description"
A client (which is a browser in our case) enables native mechanism of polling user credentials. Then it encodes (note - it does not encrypts) the credentials in a format of username:password into Base64 encoding and sends another request with added http header Authorization where the assigned values are the authentication type and the encoded credentials.
How to work around credentials dialog
The answer is obvious. In order to work around credentials request you should include Authorization header to your requests so that server wont be sending you 401 code back. But how do we do so? Selenium does not allow to inject any information between your browser and the server under test. Here we can get help from proxies.
Proxy is a web infrastructure component that is making all http traffic (in case of http-proxy) pass through it being in the middle of the client and the server. In day-to-day life such components might be used for security sake, letting all the corporate traffic through itself and either blocking requests to black-listed resources or logging request parameters to a dedicated database. Proxies are also able to modify the requests and responses so that some (or all) the parts (or the whole) are modified. Including request/response headers.
Browsers in turn know how to work with proxies. The same relates to the browsers which are under control of WebDriver.
Using the proxy we can add Authorization header with prepared value evaluated from our credentials and hence bypass credentials dialog phase. Proxy might be a part of persistent infrastructure in your test environment. This is not very convenient since it would require a lot of maintenance effort, people would need to spend time to reconfigure the proxy if the test requires reconfiguration and in general will become a factor that everyone will have to consider when use your test environment. Much more convenient would be running your own proxy for your own test and even more convenient is to make proxy configuration a part of your test so that wherever the test is executed, proxy configuration would be valid for it.
One of the solutions allowing to implement such approach is browsermob-proxy library. Using this library we’re going to look at several ways of how we bypass basic authentication dialog.
Example description
In the example that I have prepared we’re going to implement a test that would visit https://webelement.click/stand/basic?lang=en page that is hidden behind basic authentication. Proxy start up and configuration we’ll be executing in test preparation part. We will see three ways of how the proxy can be configured. In the first example we will be adding authorization headers to all requests with no condition or exception. In the second example we’re going to add headers only to the requests which meet certain conditions (for example we could add or not add the header depending ot which resource is being requested). The last third example will not address headers management problem (despite the topic is actually about it) but show how to address authorization issue with help of browsermob-proxy authorization toolset.
Prepare for developing the test
We’re going to develop our test in Java language. In my example I’ll be using FireFox browser and Linux OS. Also except of Selenium I will be using Maven and JUnit. If you are not yet aware of these two frameworks it is not a big problem. They are not impacting the example concept but rather serve to simplify the logic of running our tests. So, in order to let your future code access to all required libraries, make sure your pom.xml is looking like this:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>webelement-click</groupId>
<artifactId>selenium-http-headers-example</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<java.version>1.8</java.version>
<maven.compiler.source>${java.version}</maven.compiler.source>
<maven.compiler.target>${java.version}</maven.compiler.target>
</properties>
<dependencies>
<!-- https://mvnrepository.com/artifact/org.seleniumhq.selenium/selenium-java -->
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>3.141.59</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.junit.jupiter/junit-jupiter-api -->
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-api</artifactId>
<version>5.5.2</version>
<scope>test</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/net.lightbody.bmp/browsermob-proxy -->
<dependency>
<groupId>net.lightbody.bmp</groupId>
<artifactId>browsermob-core</artifactId>
<version>2.1.5</version>
<scope>test</scope>
</dependency>
</dependencies>
</project>
In our project dependencies we list three libraries. Selenium itself, JUnit for managing test execution and asserting test outcome and the library that we are focusing on - browsermob-core.
Create test fish-bone
The test it self is very straightforward. After a bit of manipulation with proxy settings (which I take out to the separate method for the sake of demonstration) the test opens https://webelement.click/stand/basic?lang=en that would cause authentication dialog showing up if we would not run the test throughout our preconfigured proxy. You can check manually how this page behaves in order to make sure that the dialog is indeed rendered and that incorrect credentials do not allow you to proceed to the hidden page. The fish-bone code looks like this:
package click.webelement.basicauth;
import io.netty.handler.codec.http.HttpRequest;
import io.netty.handler.codec.http.HttpResponse;
import net.lightbody.bmp.BrowserMobProxy;
import net.lightbody.bmp.BrowserMobProxyServer;
import net.lightbody.bmp.client.ClientUtil;
import net.lightbody.bmp.filters.RequestFilter;
import net.lightbody.bmp.proxy.auth.AuthType;
import net.lightbody.bmp.util.HttpMessageContents;
import net.lightbody.bmp.util.HttpMessageInfo;
import org.junit.jupiter.api.*;
import org.openqa.selenium.By;
import org.openqa.selenium.NoSuchElementException;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.firefox.FirefoxOptions;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.FluentWait;
import org.openqa.selenium.support.ui.Wait;
import java.io.UnsupportedEncodingException;
import java.time.Duration;
import java.util.Base64;
public class BasicAuthenticationTest {
WebDriver driver;
BrowserMobProxy proxy;
@BeforeAll
public static void globalSetup(){
System.setProperty("webdriver.gecko.driver", "/home/alexey/Dev/webdrivers/geckodriver");
}
@BeforeEach
public void setUp(){
setUpProxy();
FirefoxOptions ffOptions = new FirefoxOptions();
ffOptions.setProxy(ClientUtil.createSeleniumProxy(proxy));
driver = new FirefoxDriver(ffOptions);
}
@Test
public void testBasicAuth(){
driver.get("https://webelement.click/stand/basic?lang=en");
Wait<WebDriver> waiter = new FluentWait(driver).withTimeout(Duration.ofSeconds(30)).ignoring(NoSuchElementException.class);
String greetings = waiter.until(ExpectedConditions.visibilityOfElementLocated(By.xpath("//div[@class='post-body']/div/h2"))).getText();
Assertions.assertEquals("You have authorized successfully!", greetings, "Greeting message is displayed.");
}
@AfterEach
public void tearDown(){
if(driver != null){
driver.quit();
}
if(proxy != null){
proxy.stop();
}
}
private void setUpProxy(){
}
}
If you copy this code to your IDE "as is", then your IDE will likely highlight some imports as unused ones. The fact is that all the imports shown here are the union of the imports required by all three examples we’re going to look at. We’ll be moving from one example to another sequentially but not to waste the space I decided to put all the imports of all three examples to this fish-bone code. It should help you to see which package is to be imported exactly if you have several classes with the same name in different packages.
In the code above we can see that each test will be beginning with setting up proxy server in setUpProxy(); method. The body of that method I will demonstrate in three variants. In remaining lines of setUp() method we cast BrowserMobProxy object to the class that is accepted by WebDriver. We do that with the help of utility method ClientUtil.createSeleniumProxy() that is supplied by prowsermob-proxy library.
Test method knows nothing about authentication on the page. It just opens the URL and expects to see some greetings there. Obviously if the authentication will not be successful, the test will fail.
Add http headers to all outgoing requests
This approach will fit you if all the requests which your page generates are to incorporate some header. To reach this we need to invoke proxy method called addHeader().
Such approach might not fit you in some cases. The fact is that it will add the header to really all the requests which your test page generates. When you open a page the request that is going from your browser will be likely not a single GET request and might be sent not only to the server that is under your control. The page might incorporate many resources such as media elements (e.g. images), javascript files, CSS files, frames, etc. Moreover the page might execute asynchronous script that might request data from third-party servers. All such resources are requested with separate http-requests and not all of them do really need the header to be included. Even for some of them it would make the returned data not relevant to what you expect.
If adding a header to all requests with no exception is your conscious choice, then our setUpProxy() method would look like this:
public void setUpProxy(){
proxy = new BrowserMobProxyServer();
try {
String authHeader = "Basic " + Base64.getEncoder().encodeToString("webelement:click".getBytes("utf-8"));
proxy.addHeader("Authorization", authHeader);
} catch (UnsupportedEncodingException e) {
System.err.println("Couldn't add authorization header..");
e.printStackTrace();
}
proxy.start(0);
}
Here in the line
String authHeader = "Basic " + Base64.getEncoder().encodeToString("webelement:click".getBytes("utf-8"));
we are forming the header that is to be added to the requests. Then we add it to all the requests which are passed through our proxy in proxy.addHeader("Authorization", authHeader);, and finally start our proxy setting 0 as start parameter which means that the proxy will start on the port that is given by execution environment (OS).
Add http headers to only chosen requests
If the approach shown above is not what you can accept in some reason, then you can make use of request filters that are supplied by the library. When we use such filters we can define the logic and the conditions for handling particular requests. By the "handling" we also can understand adding headers. Lets see how our configuration would look like in this case:
public void setUpProxy(){
proxy = new BrowserMobProxyServer();
proxy.addRequestFilter(new RequestFilter() {
@Override
public HttpResponse filterRequest(HttpRequest httpRequest, HttpMessageContents httpMessageContents, HttpMessageInfo httpMessageInfo) {
try {
if(httpRequest.getUri().toLowerCase().endsWith("css")){
System.out.println("Skip adding authorization header for: " + httpRequest.getUri());
}else{
String authHeader = "Basic " + Base64.getEncoder().encodeToString("webelement:click".getBytes("utf-8"));
System.out.println("Adding header for: " + httpRequest.getUri());
httpRequest.headers().add("Authorization", authHeader);
}
} catch (UnsupportedEncodingException e) {
System.err.println("Couldn't add authorization header..");
e.printStackTrace();
}
return null;
}
});
proxy.start(0);
}
In this example we use 'addRequestFilter()' method of our proxy object. This method accepts objects which implement RequestFilter interface as the parameter. That interface defines the only single method to be implemented - filterRequest. This is why we create anonymous object right within the method call where we implement filtering logic.
Developers of browsermob-proxy library took care of passing required objects to the method we’re implementing. The most valuable for us is the object of HttpRequest class. This object holds the request that is passing through our proxy from the web client.
In the example we’re adding header to only the requests whose URLs do not end with css. The example probably does not carry lot of sense, but nevertheless it shows how we do take decisions on whether we add headers to request or not using the properties of those requests themselves. In the real life one could restrict header altering only by the requests which are sent to the servers under their control.
Pass through Basic authentication without explicit header management
Despite our today topic is tightly coupled with http headers it would be not fair not to tell you about the native mechanism of browsermob-proxy that addresses authentication problems (this native method operates with request headers under the hood though). Lets take a look at the third example.
public void setUpProxy(){
proxy = new BrowserMobProxyServer();
proxy.autoAuthorization("webelement.click", "webelement", "click", AuthType.BASIC);
proxy.start(0);
}
I have configured everything but nothing is still working
If you can see that something is going wrong in your case and the application does not respond to your header manipulation in a proper way, I recommend to check that:
-
You are not trying to reach the host by a loopback address (ip range
127.0.0.1-127.255.255.255). Such requests might bypass proxy and go directly to the server. This means that no headers would be added. -
Firewall rules do not block interaction with proxy
-
You have incoming request logging enabled in your application under test so that you can troubleshoot issues more effectively.
If you’re looking for Selenium Grid solution it might be worth checking how using BrowserMob-Proxy in Selenium Grid is different from standalone case.
If you still have the questions please send them to me using this form. I will amend the article according to your feedback.